Making The Easy Things Easy With WireGuard®

Hey there! Welcome to the first engineering blog post of many. I guess we need to introduce ourselves – We’re DevZero, and we’re on a mission to bring FAANG (or is it MANGA -- whatever those big tech guys call themselves) developer tooling to the masses¹. You bring us your Git repos, and we'll dutifully transmogrify them into our bespoke workspace specification (we call them ‘recipes’). From there, we give you a Linux-based workspace (with all of your tools & code in it, ready to go), and a fully-managed Kubernetes cluster that you can test & ship your code in. We do all of this without you ever having to know what the hell a CNI plugin is². 🪄✨ Magic ✨^{(or so it seems)}.

We get out of your way, so that you can ship code faster – hence, making the “easy things easy”.

Connecting two remote machines together

One of the biggest problems that any developer immediately faces when they’re doing development on another machine is - you guessed it, connecting to the machine in the first place. Typical home networking setups place every device on the LAN behind a layer of NAT (network address translation) - meaning that your local devices are assigned private IP addresses, and exposing anything running on those devices to the public internet requires that you do things like ‘port forwarding’ (...or UPnP or NAT-PMP) to get around it. Bleh.

If you’re one of the lucky few that have public IP addresses to spare, this isn’t a problem for you; just relax your firewall rules a little bit to allow connections to your SSH port. But, unless you really like configuring fail2ban³ or are into the masochistic art of hardening OpenSSH⁴ - there’s a fairly good chance that your SSH server will be insecure. It’s hard for regular people to secure an SSH server – not even considering the various exploits that are published regularly (looking at you, regreSSHion). People just don’t keep their software up to date unless it’s absolutely forced onto them – and even then, auto-updating is kind of an awful solution as well!

So, if you’re anything like me, and you think that ‘port forwarding’ is gross, and publicly exposing SSH is a bad idea… and you think that "there has to be a better way to do this", may I present said "better way". Drumroll please! drumroll ...

VPNs! If you’re blissfully unaware of what a VPN is, they basically allow you to seamlessly connect two (or more) computers together over a secure connection. If you’ve ever heard of IPSec, L2TP, IKEv2, OpenVPN, Cisco AnyConnect, etc - these are all examples of VPNs. 

VPNs are a kind of "overlay network", in that they're not quite real - they're somewhat imaginary. What VPNs do allow you to do, however, is reuse the existing infrastructure of the Internet AND be able to access another computer securely. Sorta like if you conjured a magical infinite ethernet cable to connect two computers together. They're super cool - if you're not already using a VPN at home, I strongly recommend you start today.

That being said, though, traditional VPN solutions have an incredibly dubious track record of being secure - see: Check Point Security’s IPsec VPN, Cisco AnyConnect… the list goes on and on, and on. I think it’s fair to say that most VPN solutions are slow, insecure, or overly complicated – some of them being all three. Enterprise software, am I right?

A new challenger approaches!

This all changed in 2018, when Jason Donenfield published a RFC to the Linux Kernel Mailing List (LKML) for a brand-new VPN solution called ‘WireGuard’. WireGuard is if a cryptographer designed a network protocol, as it has top notch cryptography⁵ (elliptic-curve cryptography, Noise, SipHash24), ‘perfect forward secrecy’ and rigorous formal verification that proves all of these guarantees. 

It's like a Ph.D. project that was actually finished, without reinventing the wheel along the way. Instead it’s sleek. Cool. Does everything that it’s advertised to do⁶.

And you’d think that with all of these properties, it’d be some extremely convoluted protocol involving ASN.1⁷ or some bat-shit crazy distributed state machine hackery… but that’s, refreshingly, not the case! WireGuard is incredibly simple to implement, clocking in at around ~3000 lines of code. Which is a shockingly small amount of code, especially compared to OpenVPN & the like. It’s also the fastest kid on the block, easily saturating a 10Gb/s NIC⁸.

When you first hear about WireGuard, your first thought might be "Am I dreaming?". To which, reader, you are not. This is real life. You can thank Jason Donenfeld for his hard work on WireGuard. Maybe even send him some coffee money.

Naturally, WireGuard quickly adopted a cult following amongst developers. It even earned a heartfelt endorsement from Linus Torvalds, which, if you're weird like me and read the Linux Kernel Mailing List, you'll know to be a harsh critic of anything remotely bad. Linus doesn't like poorly designed things. WireGuard, therefore, is not a poorly designed thing. It's quite good - maybe even excellent, one would say.

And so, after two long years, a kernel module for WireGuard was then included in the merge window for Linux 5.6, making WireGuard a permanent first-class citizen within the Linux kernel. I can’t overstate how huge this is – getting code merged into the Linux kernel, nevertheless networking code, is a non-trivial effort. And up to this point, no other VPN solution has ever been integrated into the kernel.

This immediately skyrocketed WireGuard’s popularity. Linux is, without a doubt, the most widely distributed software platform in the world⁹. And now it has first-class support for WireGuard. Mind. Blown.

Correction (09/09/2024): a commenter on Hacker News has dutifully informed me that, actually, IPsec actually beat WireGuard to the punch with being implemented in the kernel first. So, scratch that bit.

🪄✨ Magically ✨ shuttling bits around

This all brings us to today, and the part of this blog post where the bean-counters informed me that we have to have an (obligatory) plug for ourselves. We use WireGuard. And with it, we pair it with some very clever client code¹⁰, and a sprinkle of NAT traversal magic that’s kinda like WebRTC (but not like WebRTC¹¹). With that, we’re able to dynamically build an encrypted mesh network between you, your teammates, and your workspaces. The TL;DR of it all is that we’re able to connect you to both your coworkers & your workspaces from any network, anywhere in the world. 

Let’s say that you want to share your workspace with your coworkers. Simple. Just have them SSH into it directly with ssh devzero@<your workspace>. Or, let’s say that you’re hosting a web server on your workspace, and you want to show your coworkers. Your coworkers can just visit http://<your workspace>:3000 in their browsers, all without you having to expose anything to the public internet – keeping things secure & private. Again, simple.

No ngrok, no fiddling with HTTPS, no exposing stuff to the public internet, no more sending your teammates localhost:3000 links and being surprised when it doesn’t work on their machine – none of that bullshit. We’re frankly sick of it. Our networking stack Just Works™¹², all while being secure by default. We take care of all of that ACME / SSL / networking bullshit¹³, so that you can get back to writing code.

You might’ve seen in our graphic above that we connect everyone together in somewhat of an unconventional way - hence why we call it a mesh network. Instead of a ‘hub-and-spoke’ system, where you have one central VPN server which allows you to connect to everyone else, we directly connect you to your teammates, avoiding an unnecessary extra hop.

We picked our mesh-like approach, wherein you directly connect to one another, specifically to avoid any extra unnecessary latency. This matters a ton, especially when you’re using SSH (which we use to connect you to your workspaces). If you’ve ever been unfortunate enough to have used SSH on a mobile connection, then you’re already well-aware of this – SSH is not tolerant of slow / unreliable connections in the slightest, hence why things like mosh exist.

With that, actually, you might’ve noticed that you’re also actually not just limited to connecting to your workspaces - in fact, you can use this networking stack to connect to just about anything: the Internet, your hoard of “Linux ISOs”¹⁴, RDS databases… a good rule of thumb is if it (or a device connected to it) can run our networking daemon, you can connect it to your team’s network. 

And in fact, we actually have a super secret project going on where we’ve even gotten your browser (via some really cool WebAssembly/service worker magic¹⁵) to connect to your team’s network – that deserves its own separate blog article.

Pulling back the curtains / A Love Letter to Open Source

Our networking stack is super cool, and it’s a major part of the ‘magic’ that goes into our overall product. But it’s important to us that we recognize that we stand on the shoulders of giants, as we make heavy use of some super cool, permissively licensed, open-source technology (... something, something, scales, tails…) and the open-source control plane for said technology, which is, again, using yet more super cool, permissively licensed open-source technology. Without any of these being open-sourced & permissively licensed, there’s not a single chance that we would’ve been able to build any of this.

On a more personal note: I’m incredibly grateful to be living in an age where companies like Scales of Tails and Fe²O³ Computer exist, building their product in the open for all to see. As someone who grew up tinkering & disassembling computers & figuring out how the hell these complicated things all worked together, I feel like a kid in a candy shop, peeling back the curtains and getting to look what’s behind these innovative technologies. I’ve always been fascinated by how complex systems work, and it’s a joy to dig into why / how they work. And I can’t say enough about how I’m genuinely so incredibly grateful that we’re able to build something super cool on top of them - it is genuinely such a privilege.

I’ve built my entire career on open-source technologies, and I can’t say enough about what good it can do for society as a whole. The fact that we managed to convince the bean-counters about open source - ha!

With that in mind, I’ll be attempting to upstream some of our changes wherever possible, and attempting to contribute back to the ecosystem. The tragedy of the commons is very real, and we need to be good stewards of the land & not take great things for granted – especially if we are to continue to build on top of these technologies for free. Call it, “responsible open source”, if you will.

Conclusion

Open source is amazing. Getting to see the products of some of the greatest minds in computer science, and then turning around, and proceeding to build something on top of their life’s work is so unbelievably rewarding. We do exactly this, and have the privilege of being able to use some incredible open-source software to shuttle your bits from point A to point B – all in one large concerted effort to keep your SSH connections snappy, and your life simple. 

Try us out! And do let us know if you run into any bugs or rough edges along the way – we’re still figuring this whole production-quality ‘infrastructure’ thing out¹⁷.

Footnotes

1. If you’ve ever heard of a Devpod, we’re like if a Devpod had a supercharger strapped to it.

2. Shoutout to everyone who’s done Kubernetes the hard way. It shows you how much we take for granted by using EKS / whatnot.

3. Not a diss against fail2ban. It’s a good piece of software. Just a bit annoying to configure.

4. I mean, really. You have to disable password authentication, generate a private / public key pair, set up your authorized_keys file, disable root login over SSH, disable agent forwarding… Easy if you’re a seasoned sysadmin, but try explaining this to a junior engineer, and watch their eyes glaze over.

5. It’s not post-quantum cryptography, which is a little scary for when we inevitably solve integer factorization on quantum computers. Though, Jason Donenfield has said that a v2 of WireGuard would be fairly easy to implement, so that’s reassuring.

6. As someone who’s had to work on one of these Ph.D. led projects in the past… oh lord.

7. ASN.1 is perhaps the craziest thing to exist on this planet, right next to XML. The number of people I trust to write a good parser for that are very few. Whenever I hear the phrase ‘ASN.1’, my fight-or-flight reflexes kick in.

8. Unfortunately, if you need more than 10Gb/s, you’ll need to look elsewhere. Anything faster than that is entering ASIC territory… which, I’m not sure if anyone’s created an ASIC for WireGuard networking. Could be cool!

9. Insert “3 Billion Devices Run Java” here. I think that Java is the only other thing that could even come close to matching Linux’s popularity, since basically every single SIM card / credit card / etc is a JavaCard. Oh, Sun Technologies, how you were so greatly before your time.

10. Admittedly, a little too clever at times. Sometimes we need to turn down the cleverness a notch or two.

11. The Scales of Tails guys call this technology “DERP”, and we run our own globally distributed relays.

12. Except when it doesn’t, and we have to spend a lot of time debugging why it doesn’t work. But, hey, we’re a startup, so our product working at least 50% of the time means that we can sell it. We’re doing better than Juicero, at least!

13. If you have to learn how BGP works, we’ve failed to do our jobs correctly.

14. I know what they actually are. You can’t hide the truth from me.

15. Somewhat inspired via this Notion engineering blog post: https://www.notion.so/blog/how-we-sped-up-notion-in-the-browser-with-wasm-sqlite‍

16. For some, our forks have diverged so significantly that it’s a futile effort to try and reconverge with upstream. But we’ll try!

17. This is, as many of you know, the biggest problem one faces when they build a distributed system. It’s easy to build them. Very easy to build them, I’d say. Hard to operate them. Very hard to operate them. Shoutout to all of the SREs out there.