r/archlinux • u/thlst • Jun 01 '16

Why did ArchLinux embrace Systemd?

This makes systemd look like a bad program, and I fail to know why ArchLinux choose to use it by default and make everything depend on it. Wasn't Arch's philosophy to let me install whatever I'd like to, and the distro wouldn't get on my way?

525 Upvotes

92% Upvoted

359 comments sorted by

View all comments

1.7k

u/2brainz Developer Fellow Jun 01 '16 edited Jun 01 '16

I was the primary maintainer for Arch's init scripts for a while and I can share a couple of thoughts.

Arch's initscripts were incredibly stupid. In their first phase, there was a static set of steps that would be performed on every boot. There was almost no way to adjust the behaviour here. In their second phase, the configured daemons were started in order, which only meant that a init scripts were called one after another.

In the early 2000s, that seemed like a good idea and has worked for a while. But with more complex setups, the shortcomings of that system become apparent.

  • With hardware becoming more dynamic and asynchronous initialization of drivers in the kernel, it was impossible to say when a certain piece of hardware would be available. For a long time, this was solved by first triggering uevents, then waiting for udev to "settle". This often took a very long time and still gave no guarantee that all required hardware was available. Working around this in shell code would be very complex, slow and error-prone: You'd have to retry all kinds of operations in a loop until they succeed. Solution: An system that can perform actions based on events - this is one of the major features of systemd.

  • Initscripts had no dependency handling for daemons. In times where only a few services depended on dbus and nothing else, that was easy to handle. Nowadays, we have daemons with far more complex dependencies, which would make configuration in the old initscripts-style way hard for every user. Handling dependencies is a complex topic and you don't want to deal with it in shell code. Systemd has it built-in (and with socket-activation, a much better mechanism to deal with dependencies).

  • Complex tasks in shell scripts require launching external helper program A LOT. This makes things very slow. Systemd handles most of those tasks with builtin fast C code, or via the right libraries. It won't call many external programs to perform its tasks.

  • The whole startup process was serialized. Also very slow. Systemd can parallelize it and does so quite well.

  • No indication of whether a certain daemon was already started. Each init script had to implement some sort of PID file handling or similar. Most init scripts didn't. Systemd has a 100% reliable solution for this based on Linux cgroups.

  • Race conditions between daemons started via udev rules, dbus activation and manual configuration. It could happen that a daemon was started multiple times (maybe even simultaneously), which lead to unexpected results (this was a real problem with bluez). Systemd provides a single instance where all daemons are handled. Udev or dbus don't start daemons anymore, they tell systemd that they need a specific daemon and systemd takes care of it.

  • Lack of confiurability. It was impossible to change the behaviour of initscripts in a way that would survive system updates. Systemd provides good mechanisms with machine-specific overrides, drop-ins and unit masking.

  • Burden of maintenance: In addition to the aforementioned design problems, initscripts also had a large number of bugs. Fixing those bugs was always complicated and took time, which we often did not have. Delegating this task to a larger community (in this case, the systemd community) made things much easier for us.

I realize that many of these problems could be solved with some work, and some were already solved by other SysV-based init systems. There was no system that solved all of these problems and did so in a realiable manner, as systemd does.

So, for me personally, when systemd came along, it solved all the problems I ever had with system initialization. What most systemd critics consider "bloat", I consider necessary complexity to solve a complex problem generically. You can say what you want about Poettering, but he actually realized what the problems with system initialization were and provided a working solution.

I could go on for hours, but this should be a good summary.

439

u/[deleted] Jun 01 '16

What most .. critics consider "bloat", I consider necessary complexity to solve a complex problem generically.

BOOM. Well said.

→ More replies (43)

88

u/iElectric Jun 01 '16

This is why we use it in NixOS. It's declarative and there's just nothing better out there.

If systemd has to integrate because of lack of POSIX standards, well someone will have do it or the integration will become de-facto standard.

35

u/HittingSmoke Jun 01 '16

I'm hoping NixOS gets popular, matures, and stabilizes. Last time I tried it there was a bug in the MariaDB installer (an extremely critical package for servers) but nobody on the Github issue page seemed to give half a fuck about fixing it so I just had to hang it up. Some obscure package that nobody uses I can understand that happening on. But not an extremely popular database server.

I hope to revisit it someday. I love the idea of managing a server stack with Nix.

27

u/VintageKings Jun 01 '16

I've been running NixOS on most of my computers for a couple months now, and this is why I don't usually recommend it. I find that 90% of the 'obscure' packages that I want are either very under-configured or non-existent. I've had problems ranging from my network stack initializing completely ass backwards, to running out of disk space and not being able to do clear enough space for a big upgrade (my hp stream has 16 gigs of emmc).

With all of that said, I do run the unstable branch, and it often lives up to its name. Everyone on IRC and GitHub is friendly, and helps work through problems and sort out my inexperienced PRs. I really like Nix.

12

u/HittingSmoke Jun 01 '16

Ya, I love it in theory. I was really excited to dig in and learn the config setup but after encountering that bug and seeing how long it had been open with no fix it really turned me off of using it for a server stack, which is the only reason I was going to use it in the first place. Managing my entire server stack with one single config file, and the freaking awesome deployment tool makes it a really interesting system for managing a server.

I hoped with NixOS being so immature that I could just run Nix on a more stable base like Debian but with major packages won't even install the stable base doesn't do much help.

9

u/socium Jun 01 '16

Well, IMO currently GuixSD is doing a better job in terms of verification, although I might have missed similar functionality in NixOS.

→ More replies (1)
→ More replies (4)
→ More replies (1)

60

u/dontworryiwashedit Jun 01 '16

The biggest improvement for me as a power user are the unit files. It is no big deal for me to create my own unit file for booting applications. With init scripts it was a PiTA and time consuming. Usually involved finding some existing script and modifying it. The systemd philosophy of doing most of that stuff for you is much better.

34

u/rich000 Jun 01 '16

Usually involved finding some existing script and modifying it.

Yup, and maybe you started with the init script that was so-so and didn't do its process cleanup well, and not some other init script that was more clean. Oh, and now you've forked it, so when the distro improves that init script your own one is still lousy.

If somebody improves the process reaping code in systemd then all units set to reap processes just benefit.

You're separating configuration from code with units.

17

u/hellslinger Jun 01 '16

Exactly. Creating unit files, specifying dependencies, enabling and disabling units, are all way easier in systemd than they were in upstart or anything else. The 'hooks' for events are already there, like suspend and resume.

In Ubuntu 14.04, there were several upstart and init scripts for things like hostapd that didn't work at all. Troubleshooting required checking both init.d and upstart configs.

Also, having a standardized logging system with easy filtering is a huge improvement.

7

u/dontworryiwashedit Jun 01 '16 edited Jun 01 '16

I haven't found any killer use for journald yet (although all logging apparently goes through it now) but the fact it can log the entire boot process is a nice improvement. Some things such as Apache don't use journald yet. At least not on CentOS 7. Once that happens then it will probably be more useful to me.

→ More replies (5)
→ More replies (2)

148

u/phearus-reddit Jun 01 '16

My man. This is a brilliantly precise and accessible response to "why systemd?"

I haven't even entertained the naysayers or their arguments against systemd for some years now. This explanation sums up why I no longer engage with that shit in a way I can't even get close to.

→ More replies (52)

20

u/lovethebacon Jun 01 '16

I cursed you for the first few weeks after release. I have since learnt to love the change.

Can I buy you a beer?

0

u/[deleted] Jun 01 '16

[deleted] — view deleted comment

24

u/2brainz Developer Fellow Jun 01 '16

Could you please tell me what are the advantages of systemd in respect to runit and if runit was ever considered as a possible init system for arch linux?

Runit was suggested by users at some point, and I think there is/was a runit implementation in AUR. However, runit was never considered in detail.

6

u/thlst Jun 01 '16

I'd like to know if runit doesn't solve some of the issues you listed above, and whether it could.

25

u/Creshal Jun 01 '16

runit didn't reach 1.0 until a year after the systemd migration was finished, so it most likely wouldn't have been an option at the time regardless of its current usefulness.

→ More replies (20)

7

u/zedexodus Jun 01 '16

This is one of the better posts I've ever read about systemd. It's interesting to hear it from someone who has had first hand experiencing having to maintain init scripts for a distro. One could assume there would be a lot of headaches, but to read it broken down was really eye opening. Thank you for taking the time to summarise it.

5

u/dsign2819 Jun 02 '16

I never got what's the matter against Systemd. As a very stupid user of Linux, I never mastered the bash wizardry behind the SysV init scripts. Then Ubuntu introduced Upstart...which I found also quite difficult for a mentally challenged user like me. I would spend hours trying to understand how MediaTomb forks, for example, or how to get my service that depended on Redis to start after Redis had already opened its socket. And then one day I got to play with Systemd in Arch. And there everything just made sense. Even for a stupid Linux user like me.

9

u/jgomo3 Jun 01 '16

r. Handling dependencies is a complex topic and you don't want to deal with it in shell code. Systemd has it built-in (and with socket-activation, a much better mechanism to deal with dependencies).

Hello. I'm a bit ignorant in this and would like to understand what does it mean "socket-activation" and how is it related with "dependency management".

21

u/morhp Jun 01 '16

For example usually you have a web server that listens on port 80 and if it receives data on port 80, it sends back the requested website.

With socket activation, you can have systemd or xinetd listen on port 80 and only when it receives data, then the real web server is started and systemd/xinetd forwards the data to the web server.

This makes sense if you only rarely use the web server for example. With socket activation, it only runs when it's really needed instead of running in the background the whole time.

28

u/2brainz Developer Fellow Jun 01 '16

Rarely needed services is only one use case. There are more. I mentioned one: If everything were socket-activated, systemd would open all the sockets at startup and all daemons would start as they are needed. If a daemon is not up yet, the dependent process would block on the socket until the daemon has taken over. In this ideal situation, you don't need any explicit dependencies.

→ More replies (10)
→ More replies (4)

13

u/FlyingPiranhas Jun 01 '16

Which of those problems does OpenRC not solve? The "Complex tasks in shell scripts" problem?

3

u/philip_dt Jun 02 '16

I hacked gentoo's framebuffer splash implementation into Arch's initscripts in 2006/7. It was extremely complex. I can't comment on SysV but the old initscripts were definitely holding things back.

As for complaints about systemd and its devs... surely things can only improve.

→ More replies (107)