Why systemd is a problem for embedded Linux

The Internet is full of rants about systemd, and I don’t want this post to be another one. Many of the complaints people make about it don’t stand up to much scrutiny, even the technical ones; and many complaints are not even technical. My particular interest in Linux is primarily for embedded applications; and there, I suggest, systemd is creating a potential (technical) problem. In this article I will try to articulate what the problem is; but I have no solution to offer.

Recapping the last ten years

systemd is a set of integrated applications concerned with system management. It replaces not only the traditional init process that brings up long-lived processes, but also much of the other system infrastructure: user session management, device management, logging, timing, and an increasing number of other functions.

The majority of Linux users are uninterested in the pros and cons of systemd. A small number are violently opposed to it, and a small number are violently opposed to those who are opposed to it. Nevertheless, most mainstream Linux distributions have adopted it after a shorter (Fedora) or longer (Debian) period of argument.

I think there’s little argument that the main target for systemd is a general-purpose computer, with a modern, integrated graphical desktop (Gnome, KDE). systemd does well in systems like this because it can handle process initialization on demand and in parallel. This potentially makes boot times faster, and keeps resource usage down, because it isn’t necessary to start a lot of services that are used only occasionally. I don’t think these advantages are the main reasons for systemd’s widespread adoption (see below), but they’re certainly important.

Embedded Linux: the problem

Unfortunately, what makes systemd good for general-purpose desktop applications potentially makes it unsatisfactory for embedded Linux systems. As an illustration, I’ll show some memory figures from the Raspberry Pi 3B that’s currently on my workbench. The board is running the DietPi Linux distribution – probably the best fully-maintained Pi distribution, if you want a minimal system. Although DietPi uses systemd (it has little alternative, as I’ll explain later) it doesn’t necessarily use the full set of components. In fact, a minimal installation of DietPi, for console operation, installs only the systemd service manager (the ‘init’ process), the log daemon, and the udev daemon.

This is the resource usage, as reported by top, for the systemd init process (alone) on the Raspberry Pi 3B.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM
      1 root      20   0  168144  11404   8592 S   0.0   0.3

systemd init is not a particularly greedy user of memory by contemporary standards – its virtual address space is 168Mb, but only ~8Mb is currently mapped to RAM. That’s about 0.3% of the Pi 3’s 4Gb of RAM. But here’s the same figures for SystemV init, on exactly the same hardware:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM
    1 root      20   0    2744   1532   1436 S   0.0   0.2

It’s much smaller. Just to be clear – I got the systemd implementation and the SystemV init implementation from the same Debian ARM binary repository. I haven’t used fiddly compiler optimizations or anything like that, to bias the resource usage figures.

Now let’s look at the systemd logging daemon, systemd-journald.

  PID USER     PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    147 root   20   0   41508   7744   6652 S   0.0   0.2   0:12.05 systemd-jour+

Again, it’s a small, but noticeable, user of the 4Gb RAM. And, for comparison, these are the figures from my own syslogd-lite, which I wrote specifically for embedded applications.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  117 root     20   0    1976     72      0 S   0.0   0.0   0:00.00 syslogd

Note that the memory and CPU usage of this logging daemon are both essentially zero. This is an unfair comparison, in a way, because I wrote syslogd-lite specifically to minimize resource usage, and it has few features. But it shows that it’s plausible to write utilities that
target embedded systems specifically, and sometimes it’s necessary. systemd does not do this, and was never intended to. Running a general-purpose software set like systemd on minimal, embedded hardware can’t be expected to be effective.

With care, though, a minimal installation of systemd does run on a low-resource ARM board like the Pi 3. In fact, it will run on a board with 1Gb RAM, perhaps even lower. But, as the RAM decreases, the proportion of it occupied by systemd increases.

What’s less obvious is the effect on boot time of the use of systemd init. Proponents of systemd argue – correctly, I think – that it decreases boot time in the kinds of system for which it was designed. But on my Pi 3 it increases boot time quite noticeably. That is, the systemd ‘init’ process takes an extra half-second or so to start, compared to SystemV init. Half a second isn’t much, but in an embedded application I actually care about boot time. I couldn’t care less on my desktop computers, because they spend most of their lives suspended. I rarely actually reboot them.

The extra start-up time of the init process is the result, I guess, of the additional size and complexity of the systemd executable. It’s about 200kB in size itself, and dynamically links 26 other libraries. SystemV init, on the same hardware, is 40kB, and links only the standard C library. The additional complexity of systemd is not wasted: it’s needed for the additional functionality that systemd offers, in its intended environment. But this functionality is mostly not needed in an embedded application, so the additional complexity of systemd is a cost without a benefit.

I’ve found that most of the services that systemd replaces have an alternative that is smaller, and faster to start, in an embedded environment. Unfortunately, some of these services don’t have an alternative any more.

So what?

I’m not obliged to run systemd on my Raspberry Pi systems and, in fact, usually I do not. I build my own Linux installation, using binaries that I cherry-pick from the Debian repositories, and code I write myself. Most of the binaries work without systemd. Some complain about not finding systemd, but work anyway. Some things don’t work without systemd: the Gnome display manager, for example, as it is built for the standard Raspberry Pi, won’t work. It can be made to work, but you have to build it from source. How long it will continue to work, even if built from source, is open to question. But I’m not going to be running Gnome on an embedded Linux board, so I don’t see this as a problem for my applications.

The more fundamental problem is that the people who most like systemd are distribution managers. Sure, there are administrators who like it, and defend it vigorously; but most end users and administrators don’t really care. But for the maintainers of mainstream Linux distributions, systemd is like Christmas. systemd works reasonably well for a whole range of usage scenarios and, best of all, it all comes in one bundle. So a distribution manager doesn’t have to maintain, integrate, and support a whole bunch of system utilities from different sources – systemd provides everything in one huge build.

There are just a few Linux distributions that don’t use systemd, and they are not widely used. Maintaining them is difficult, with just a handful of volunteers. All the large, commercial Linux providers have moved to systemd.

Only Gentoo and its derivatives (so far as I know) make systemd optional, with fully-supported alternatives. And even Gentoo can’t be made 100% free of systemd – not in practice.

Is it even practical to avoid systemd any more?

Take, for example, the udev daemon. This service monitors the kernel, and makes configuration changes when devices are added and removed. It’s not an essential part of Linux, but it removes the need for a whole heap of manual configuration.

At the time of writing there is no fully-featured udev implementation outside of systemd. The original udev code was absorbed into the systemd project about ten years ago. The Gentoo alternative eudev is no longer fully maintained. At present, if your Linux distribution requires udev, you’re forced to use the version from systemd. This version can be used without the rest of systemd, but it nevertheless does systemd operations. In particular, it tries to communicate with systemd over DBus. This communication (so far) fails gracefully if systemd is not present, but it’s not clear how long this will continue to be the case.

systemd is a tightly-coupled set of services. I don’t think a design goal of the systemd maintainers is to make its components modular. udev doesn’t need to talk to systemd but, clearly, there’s some benefit to its doing so in a systemd installation. I understand that there was a kind of ‘gentlemen’s agreement’ between the systemd maintainers and the Gentoo maintainers, to keep udev independent of the rest of systemd. I think we can see that this agreement has broken down a little already; I suspect it will break down more, if the systemd folks think that tighter integration will make systemd work better.

Many parts of systemd continue – for now – to have non-systemd alternatives. For example, systemd has a clock synchronizer systemd_systemtimed. The systemd maintainers are perfectly honest that this software lacks features that exist in alternatives like Chrony and OpenNTPD, and it’s less accurate. systemd_timed is included because it’s fast to start, and satisfactory for applications where exact time synchronization is not critical.

At present, Chrony remains widely used, even in some distributions (like Fedora) that use systemd. But with systemd becoming ubiquitous, what motivation will there be to maintain non-systemd alternatives? These alternatives could fall into disrepair, even though some are superior to the systemd utilities – and even the systemd maintainers admit this.

Similarly, systemd has a DHCP client. It isn’t the only DHCP client that is available, but my concern is that one day it might be. In my tests, I’ve found that the systemd components are larger, and slower to start, than the traditional alternatives (where they still exist). Again, this isn’t a criticism of systemd itself – I’m testing them in an environment they were not designed for.

So where does that leave embedded Linux?

I’ve found that many systemd components are less effective in an embedded environment than the traditional alternatives. I’ve shown some illustrative examples in this article, but I really don’t think there’s much controversy here: this simply isn’t the environment that systemd was designed for. But it’s getting increasingly difficult to find a mainstream Linux distribution that doesn’t use systemd – even Raspberry Pi distributions use it. As systemd absorbs more functionality into itself, there’s going to be little motivation to maintain alternatives. After all, if everybody uses systemd, what motivation is there to support anything else? My concern is that we’re moving towards a future where Linux is inconceivable without systemd. That will be a problem for those environments where systemd really doesn’t shine.

I wish I knew the solution to this problem. There’s no point complaining about systemd, because distribution maintainers have grown to like it too much. And, in any event, well-reasoned, technical concerns are drowned out by all the ranting and conspiracy theories. All we can do – if we care – is to continue to use and support Linux distributions that don’t insist on systemd, and stand ready to develop or maintain alternatives to those bits of Linux that it absorbs.