Things I learned about systemd

~ nelsonminar

systemd is the Linux startup system for Ubuntu 16.04, so I figured it’s time I learned about it. I’m very late to the party; systemd started some 6 years ago and became a real thing many Linux distributions used years ago. Ubuntu is the last big distro to include it by default except Gentoo, and even Ubuntu switched last year with 15.

There’s been a years-long debate in the Linux community about systemd. A younger me would have cared about that, maybe participated. These days I don’t care, I just want to use the consensus tech. The bummer is the flamewars dominate the conversation, it’s hard to learn just what systemd is. Here’s my effort to just understand it.

Disclaimer:  I’m not an expert and have undoubtedly made a mistake here somewhere. I reiterate my reminder that I really only write this blog as notes for myself. You’re welcome to comment, but please don’t waste my time with heated opinions.

References

I don’t have more links on actually using systemd in practice because I don’t yet have a system running it myself. That’s how out of date I am. The key pragmatic thing seems to be the unit files which configure how systemd launches services. systemd can also still launch stuff with old SysVInit-style scripts, including parsing the LSB headers.

Stuff I learned

systemd is init, the first user process on a Linux system, PID #1. Once the kernel is done it runs PID 1 whose job it is to then run all the other user stuff that makes a Unix system; mounting filesystems, configuring networking, starting daemons, etc.

The init most of us know is SysVInit, the shell scripts in /etc/rc.d. It hasn’t aged well in a modern era of multiple cores, dynamic devices, etc. Upstart is another Linux init system that Ubuntu has used for a long time, but no one seems to love it. Launchd is the MacOS init and deserves some respect for its speed and power; the systemd authors explicitly acknowledge launchd for inspiration.

systemd is a full rewrite of the idea of init. The primary goal seems to be mostly about efficiency. In the bad old days a Unix system could take 90 seconds to boot. With systemd (and MacOS launchd) < 1 second is possible and < 10 seconds is common. systemd is faster because it can do things in parallel. Also it avoids all the overhead of interpreting shell scripts. And supposedly it’s just modern and new and “better”, which I can’t really evaluate but can believe given the mess of stuff that preceded systemd.

systemd is Linux-only. It uses a bunch of non-POSIX APIs to do things like work with devices, configure security capabilities, etc.

I still haven’t used systemd hands-on so I haven’t learned much else. But my understanding is a whole bunch of Unix config gets consolidated into unit files in one place, which then tells systemd how to configure and launch user space programs.

I’m a little terrified of the Ubuntu 14 to 16 upgrade, it has to replace all the old stuff with the new. Will it work? Particularly reminded of an Ubuntu 12 to 14 upgrade I did where I manually installed something that changed the name of my ethernet device, making the machine unable to reboot with networking. Oops.

Opinions

I really don’t care about the history of the controversy; systemd’s a fait accompli. But it’s helpful to look at the debate to understand better what systemd is.

The big complaint about systemd everyone has is complexity. It does a lot of stuff in one big nest of 60+ interlocking binaries. It’s not just forking daemons. It’s got udev inside it to manage devices. It’s got its own logging system (in a binary format), a cron equivalent, an inetd equivalent, an atd equivalent, virtual terminals, a login daemon, watchdogs, etc. When you look at all the old Unix software that’s replaced it’s sort of astonishing and I can understand why believers in the “lots of little tools” philosophy don’t like it.

OTOH, the way the old components fit together in sysvinit was quite a hairball too. The shell hackery required to mount devices, then filesystems, then get networking up so you could access other filesystems and start other daemons in the right order and blah blah blah. Sure udev and mount might have been separate software, but they could only run correctly in one specific relationship to each other. That complex set of interdependencies had a lot of cost and somewhat made a lie of “lots of little tools”. Also there’s something to be said for unifying init and cron and inetd and login and the like; they’re all about launching programs in response to events, why not share the code?

There’s still a cultural concern that systemd, being a monolithic project, comes from one mindset only. Some of the choices seem odd to me; a binary log format for instance. That’s the part of the debate I can cheerfully withdraw from. For whatever reason most Linux distros have agreed on using systemd, and I trust that they all have the influence to shape it the way they need. That seems like a reasonable path to working software.

 

6 thoughts on “Things I learned about systemd”

  2. >systemd is a full rewrite of the idea of init. The primary goal seems to be mostly about efficiency.

    I don’t think that’s quite accurate, even according to the systemd designers themselves. The goal is a full rethinking/redesign/reimplementation/streamlining/consolidation of the entire userland substrate. I think it’s the very _ambition_ of this, let alone its design and implementation choices that has generated the incredible amount of heat over the years.

    I do find it somewhat interesting, as a similarly uncaring old fart, that you were unable to find substantive technical criticisms worth linking.

  3. The old UNIX init-related programs evolved; systemd was designed. That should tell you almost everything about the situation!

    I will say, though, that for routine kinds of system administration systemd is really very similar to the old way of doing things – at least in my experience on Ubuntu systems. The original log file format may be binary but the text version of the logs are in /var/log. “service” has to be used instead of calling the startup scripts directly, but that has been the case for a while in Ubuntu with upstart. If something breaks in a funny way it is much harder to put in a small hack to fix things, but those sorts of hacks have long been unwise if you didn’t want to break package upgrades.

    Thus the pain you’ll experience will largely be a function of how much tweaking you’ve been doing to your configurations.

    I’ll be interested to hear your experiences once you actually get going with systemd.

  4. pvg: I didn’t spend much time trying to find substantive criticisms worth linking. Perhaps there’s something good that teaches us? Unfortunately many of the criticisms I’ve read are both substantive and flamey, and I deliberately avoided those. I’d be particularly interested in reading technical proposals for alternatives / better ways. I’ve seen some small projects on systemd-lite, trying to retain parts of it without the whole thing, but not sure if any of those will come to anything. Hence my focus on trying to understand the current ubiquitous tech.

    And hi Anil, nice to see you here! I like your insight about evolved/designed and it makes me nostalgic for our student days :-) I’m looking forward to using systemd after what I’ve read. Particularly hoping that it’s easier to write a systemd unit config than an upstart script, I never did get over that hurdle.

  5. I got curious too and gave this another internet search pass and didn’t find much new and notable – if anything, some of the louder, most strident objections now seem to live in secondary archives only.

    One thing that struck me on revisiting is what a terrible starting point for a technical argument ‘the unix philosophy’ is. It’s important and interesting but also multifaceted of meaning and interpretation, suitable for friendly chats between acquaintances or historical reminiscing. As a basis for specific and detailed evaluation of some design or implementation, it seems mostly counterproductive and unhelpful. After all, a lot of the ‘unix philosophy’ flew out the window the moment unix escaped the confines of Bell Labs and PDP 11s.

    I think your last line is a more sensible check for both quality and against the potential pitfalls of our own preconceptions and biases – “is this a reasonable path to working software?”.

  6. Yeah I’m not impressed with “lots of little programs” as a guiding architecture principle. I mean it’s great in some circumstances, I love my `awk | sort | uniq -c | sort` pipelines as much as the next data nerd. But it never made much since for compilers or databases or web servers (CGI, anyone?). And maybe it doesn’t make sense for startup processes either. I keep thinking back to what I read about the relationship of udev and mount. They only fit together one way, maybe they belong together as a single program? I guess we’re exploring that idea now.

Comments are closed.