IPSec Server on the EdgeRouter Lite

Bradley Heilbrun

4 min readMar 30, 2016

Preface

In this guide I’ll describe setting up IPSec VPN server on a Ubiquiti EdgeRouter Lite.

EdgeMax OS Version: 1.7 and 1.8

I also have a guide for OpenVPN.

If you’re undecided between OpenVPN and IPSec, I’ve outlined my preferences here.

SSL Key and Cert Setup

These steps are nearly identical to the OpenVPN guide. Just note there’s a difference regarding the private key format.

First, generate a Certificate Authority certificate. This will be used to sign both the server certs AND the client certs. This is how the VPN server and client authenticate each other.

# All commands will be run on the EdgeRouter itself, unless noted.# Switch to root. You’ll need the permissions for writing to certain directories.sudo su
cd /usr/lib/ssl/misc/
./CA.sh -newca

You’ll have created a new directory called /usr/lib/ssl/misc/demoCA.

Now we’ll generate the keys and cert for the server (our ERL in this case) and critically, sign the server cert with our new CA.

# Create server keys and certificate.
./CA.sh -newreq# Sign the server cert with our new CA cert.
./CA.sh -sign

Let’s save our work onto the config partition.

# Save the CA key and cert.
cp demoCA/cacert.pem demoCA/private/cakey.pem /config/auth/# Save our server cert.
mv newcert.pem /config/auth/server.pem

IPSec does not support the PEM/pkcs8 format for the server key, so we’ll decrypt and reformat it to the raw DER format.

# Reformat key for IPSec and move to the config partition.
openssl pkcs8 -in newkey.pem -outform DER -out server.key
mv server.key /config/auth/server.key

Generating the Diffie-Hellman parameters can take 10 minutes on the ERL. Might I suggest a coffee break?

# Generate dhp params.
openssl dhparam -out /config/auth/dhp.pem -2 1024
cp dhp.pem /config/auth

Now we can generate the client certs. We do this on the ERL directly because we’ll need to sign them with the CA cert. Alternatively you could generate this on the client and copy the cert around for signing.

I would recommend setting the CN (Common Name) to something unique for each client. The CN is used by the VPN software to uniquely identify clients. This will help you see who is connected to the ERL. Also, by default, OpenVPN won’t allow a single CN to connect twice.

# Generate client key and cert.
./CA.sh -newreq
./CA.sh -sign
mv newcert.pem client1-cert.pem
mv newkey.pem client1-key.pem

We’ll need a PKCS12 formatted file for some clients. This format combines the client’s cert, key and the CA cert into one file.

# Combine the clients creds with the CA cert into a single p12 file.
openssl pkcs12 -export  \
  -out client1.p12  \
  -inkey client1-key.pem  \
  -in client1-cert.pem  \
  -certfile demoCA/cacert.pem

Now we’ll need to copy those client credentials onto our client. More on this below, within the respective VPN section.

IPSec Server Setup

On my setup, eth1 is the WAN (external) interface, be sure yours is too. If you’d like to lock down your VPN clients to a specific (eg, local) network, be sure to modify the allowed-network line below.

edit vpn ipsec
set ipsec-interfaces interface eth1
set nat-traversal enable
set nat-networks allowed-network 0.0.0.0/0

Tell the ERL that eth1 uses a dynamic IP provided by your ISP (assuming that’s the case).

top
edit vpn l2tp remote-access
set dhcp-interface eth1

Now define a range of IPs that your VPN clients will utilize. This can be within your existing LAN network, but must not conflict with IPs assigned by your ERL’s DHCP server.

set client-ip-pool start 192.168.1.200
set client-ip-pool stop  192.168.1.210

Tell your clients which DNS servers to use (Google Public DNS in this example).

set dns-servers server-1 8.8.8.8
set dns-servers server-2 8.8.4.4

As a first pass, you could configure simple pre-shared-secret authentication, but I highly recommend certification authentication, so I’ll describe that…

set ipsec-settings authentication mode x509
set ipsec-settings authentication x509 ca-cert-file /config/auth/cacert.pem
set ipsec-settings authentication x509 server-cert-file /config/auth/server.pem
set ipsec-settings authentication x509 server-key-file /config/auth/server.key

The basic L2TP-level user authentication.

set remote-access authentication mode local
set remote-access authentication local-users username client1 password 12345

Might I suggest a different username and password.

Android Client

In order to get an Android client to connect properly, I needed to add the following server configuration. If you’re having trouble with a non-Android client, you may need to do the same, but with slightly different options. Check the server logs for protocol/encryption mismatches to identify the right mix of encryption, hash and dh-group values.

top
edit vpn ipsec esp-group android
set lifetime 3600
set mode tunnel
set proposal 1 encryption aes256
set proposal 1 hash sha1
top
edit vpn ipsec ike-group android
set key-exchange ikev1
set proposal 1 dh-group 2
set proposal 1 encryption aes256
set proposal 1 hash sha256

Lastly…

Be sure to commit and save!

commit
save

Troubleshooting

On the server side, the two most useful commands I’ve found for troubleshooting are,

show vpn ipsec statustail -f /var/log/messages

Client Setup

Client setup is unique to each OS as the support tends to be baked into the OS itself. With the client P12 file described above and the username/password from the L2TP configuration, you have all the necessary ingredients.

Closing Thoughts

I’ll be writing a short post later on adding IPv6 support and will link it here. As mentioned at the top of the article, I personally use and prefer OpenVPN but to each his/her own!