tl;dr: OpenVPN is my preferred VPN server on the EdgeRouter Lite.
If you’d like to setup one or either of these solutions, here are links to my guides, OpenVPN and IPSec (in progress).
IPSec seemingly has the advantage here due to native support baked into most popular operating systems. Eg, MacOS and Android. By contrast, OpenVPN requires a 3rd party client.
That said, I’ve been unable to get the native Mac OS X IPsec+RSA solution to work! More on that can be found in my IPSec guide. OpenVPN’s clients may be 3rd party applications but they’re relatively easy to setup and have been reliable to operate.
I see this as basically a draw, assuming you use key/cert authentication in both cases.
If you’re in the tin-foil-hat camp, the nod appears to go to OpenVPN. It’s simpler (fewer ports and protocols) and it’s believed to have less NSA influence. A bit more about this can be found here and here.
OpenSSL wins here for one reason, the ability to run on single port, specifically 443/tcp. This allows you to connect to your VPN server in the face of most firewalls. I say most, because the Great Firewall of China will still spoil your fun. From what I’ve ready the GF does deep packet inspection, detects OpenVPN handshakes and will respond to your client with spoofed RST packets. Fortunately, most domestic firewalls are not so sophisticated.
OpenVPN is my preference. It’s easier, at least as secure as IPSec and more likely to work.