The Compatibility Struggle Looming Over the Horizon

It's been nearly 20 years since Rob Pike's infamously won the "Best at saying what we're all thinking" prize with his talk about how systems software research is irrelevant. And, while systems software research is doing slightly (though not glamorously) better than in 2000, it's still mostly circling the drain.

That being said, it's not a field that's devoid of challenges. But I think the main challenge of the next 10-15 years is going to be even less glamorous than we like to think.

I think the main challenge of the next 15 years will be to keep existing software and, more importantly, existing programming models, up and running.

The writing is on the wall. Take exo- and unikernels, for example.

Exokernels used to be one of the poster boys of systems software research in the late 1990s. Exokernel, Aegis and Nemesis are some of the more famous ones, and plenty of ink has been spent (I wouldn't say spilled) on them.

This approach -- a minimal, essentially abstraction-less kernel that gives applications nearly direct access to the hardware -- looked good on paper back in the 1990s, too, but the computing technology of the 1990s eventually turned into a dead end. At least for a while.

Cheap, plentiful and easily-managed virtualization has made this approach interesting again, in the form of unikernels. There are too many to make an objective selection that does them justice, so I will just mention the three projects that I am more familiar with, Clive, IncludeOS, and Nanos.

Now, there's some new software being written on top (!?) of unikernels, but a meaningful dent in terms of numbers isn't that easy to attain today. There's lots of software in existence. Any technology needs to have an ennui-inducing number of deployments before you can realistically say it matters.

Many current cloud deployments consist of software that's many years old. And the famous 80% of the applications that haven't been migrated to the cloud but totally will be Real Soon is likely even older than that.

At one point, someone will probably write a Wordpress clone in C++ based on IncludeOS. It's probably going to rock and, if the tools catch up with, it's definitely going to be cheaper to host and easier to manage (oh God please will that day come sooner...)

But until then, hosting a blog will still mostly require managing a LAM? stack, where the "?" will stand for PHP, Python, Ruby, Perl, Go and whatever else you can think of.

Replacing the L will require the brave souls behind it to drag the AMP onto their system, and this, in turn, will eventually boil down to exposing a POSIX-ish interface on top of whatever's underneath.

Unikernels are hardly the only example:

  • Google's Fuchsia is eventually going to have to run Android applications, at least in its early days on the market, because few consumers will be motivated to leave behind a large application base for something as obscure as an operating system. Virtually everyone except for us nerds runs Facebook, Whatsapp or Shazam on their phone, not Android or iOS.
  • Docker is such a hit mostly because it allows us to get all the benefits of 21st century computing while preserving the way we were writing applications in the 1990s. Lawrence Kubner explains it really well (although I don't share his optimism with regard to how easy it is for everyone to just Do The Right Thing).
  • In the increasingly irrelevant space of Linux desktops, Wayland is really cool and has a bunch of great features but the thing that makes it feasible today is XWayland.

I doubt that we're heading for some kind of thermal death of our industry, where people will eventually content themselves with running whatever we're running today and go solve other, more interesting problems.

It's not just that our industry isn't exactly renowned for that kind of realism (plus, if we're being serious, what we're running today is pretty embarrassing).

It's also that there are plenty benefits left to be reaped. For example, if OS-level multitasking will disappear completely, leaving us with hundreds of virtualized unikernels, secure enclaves and encrypted memory will still be relevant (if not more relevant than they are today).

This will leave us with plenty of "interesting" problems to solve and plenty of new ways to improve our computing tools. And that's just one small part of a landscape that encompasses low-power designs, hybrid, reconfigurable and programmable designs and so on.

But the capital, if unglamorous problem that will loom over this will be in the form of "our business runs on ten years' worth of Python code written by my team of Pythonista ninja hackers so can your exoquantum CPU running virtualized nanounikernels run that better than Linux and Docker? We got a ten-year timeframe to migrate everything to femtoservices but we're not quite there yet".

I wouldn't call it a "research" problem, since there's hardly too much research to do. We've known how to write POSIX implementations for non-POSIX operating systems for more than 25 years. But I have a feeling that this, and variations on this theme, are going to be an important component of systems programming in the next 10-15 years or so.

Thankfully, it won't be the only one, and I do think we'll eventually move past this paradigm.

Until then, hey, the good news (for some of us) is that this is going to make all that obscure knowledge about how shm_unlink(3) works really useful once again!