SystemD Debate and Configuration by Example

A "healthy" debate on the virtues, use-cases, and overall disdain and misconceptions of SystemD. Specific use-cases in "less-common" configurations provide a challenge as well as an opportunity to learn about the underlying mechanism of certain aspects of the Operating System. A substantial number of years working in/with Linux doesn't preclude a person from learning how to use new tools and mechanisms even if, or perhaps especially when, the mechanism is low-level software.

Users and Administrators must not get so entrenched in their ways of comfort that they reject advancements in the underlying pieces of the OS on the basis of overestimating the merit of their own existing knowledge, while downplaying the benefits of the advancements and exaggerating their problems an perceived shortcomings on surface value. Nevertheless, even though change is inevitable in our universe, it breeds chaos in the early stages of wide-spread adaptation. It's in this state of chaos where roadmaps get forged, details are laid out, and myths and misconceptions are debunked and corrected. SystemD is a galactic change in the Linux universe; and even though the dust has settled since its volatile creation, debates still arise from staunch critics who don't take the time to TRY to learn the ins and outs of SystemD to the same level of proficiency as that of the legacy tools.
It's in this spirit of spreading knowledge about the seemingly overwhelming, yet highly configurable, software that I decided to highlight the following discussion from Systemd's Google+ Community page.

How does one manage sequencing of shutdown of services. I have a PC in a car, which gets a signal from acpid when the ignition goes off. The OS responds to thiis by shutting down everything in parralel as fast as it can, and that isn't what I want.

I want to run a shutdown script for a data logger, the end of which sends an email. To send this email, I need postfix, bind and pppd all kept running until the email has been sent.

How do I prevent shutdown of ANYTHING until that logging shutdown script has sent its email. Random uses of sleep in various places isn't helping as systemd parrelelises shutdown as well as startup and I lose ppp, bind and postfix before the 'stop logging' script has run.

Is there a way to force serialisation of systemd shutdown, or embed dependencies in the services I need running so that they don't even start to stop until the logging script has sent the email ?

Any examples of where this has been done ?

edit opinion removed.
Just a thought.... You may have better luck with this sort of question if you just ask it, instead of prefacing it with stuff that's not relevant to your actual issue. That said, I'm afraid I don't know how to do what you're asking; hopefully somebody else does.
The question is about managing shutdown dependency ordering.

How does one prevent the shutdown of pppd, named, postfix (and in fact anything else) until the scripts i have (started and stopped from a systemd unit file) has COMPLETED its shutdown activities.

Ideally I want to do this without having to change any of the dependencies of those services unit files (in case the vendor decides to change those services/unit files to fix bugs).

I also asked if it's possible to serialise systemd activities - the behaviour I see now is that it stops everything it can, immediately. Having this stuff parrelelised increases the likelihood of such race conditions.
"I don't believe all linux boxes should behave as a desktop. I run servers for a living and really don't care for whizzy boot up times."

This is a fallacy - systemd is not designed for desktops, nor is it primarily about boot times any more. When Lennart first started conceiving it he was thinking about improving boot times (even then, though, he was thinking about servers as well as desktops), but the project has come a long way since then.

A lot of the subsequent development on systemd has been expressly about server-type use cases, and other non-desktop uses like cloud and container deployment.

You can use dependencies for shutdown ordering, yes. From 'man systemd.unit':

"Note that when two units with an ordering dependency between them are shut down, the inverse of the start-up order is applied. i.e. if a unit is configured with After= on another unit, the former is stopped before the latter if both are shut down."

Read 'man systemd.unit' - the whole thing is worth understanding, really, if you're building systemd units, but especially for this case you're interested in Before= and After= .
The systemd mailinglist is really good for questions like this where you might not know of any good examples. I see lots of people being helped out on there.
Note that this means if you simply have the unit which runs your script specify After= for all the other services it needs, your goal will be achieved - you don't need to do anything to the dependencies for those services themselves.
When stopping a service, all ordering dependencies are reversed. Furthermore, in any transaction systemd will always perform all stop actions before any start actions.

So your service should do nothing at startup (so you should omit ExecStart), and send the email at shutdown, using the ExecStop directive. You will also want to order it after, i.e.




The WantedBy ensures the service is "started" (effectively a no-op) at boot, if it is enabled. The After ensures that it is stopped before all ordinary services, since they're normally
Ok, so to declare the services I need to keep alive before my service has successfully stopped, I declare them all in the "After=" segment in my services unit file.

Does this mean:
postfix.service ?

My ppp interface isn't put there by systemd/networkManager - I am not sure systemd is aware there is a ppp on the host - so - should i be using network.service ?

With regards to the comments about "my service should do nothing at startup" - errr - how would I start the service then ? Two unit files ?
No, you don't want to keep those "alive". You want them to stop on shutdown, as usual. You need not and should not change the configuration of any other unit on your system.

The only thing non-standard about your particular configuration is that you want your service to stop first. As I said, stop dependencies are always the opposite of start dependencies. To ensure that a service stops first, you need to ensure it starts last. That's effectively what that After= directive will do. I chose to make your service, but you could have more fine-grained dependencies (After=foo.service bar.service baz.service) if you wanted.

You service will do nothing at startup because it has no ExecStart directive. It can still be started as normal (i.e. manually by the sysadmin, or by being "wanted" or "required" by another unit)... it just won't do anything when it is started. (Only Type=oneshot service units can be set up in this way, of course.)

When you "enable" the service, systemd will follow the instructions in the [Install] section. In the example I gave, the instructions say "make 'want' this unit". That means that during boot, when is started, your unit will be started as well. Remember, a "wants" dependency is not an ordering dependency; it's merely a "make sure something else gets started some time" dependency. And the only reason we need this unit started at all is so that it can be stopped again during shutdown. The stop action is where all the "send mail" stuff needs to happens.

The whole lot is really quite simple once you understand the various kinds of dependencies systemd makes available (and the not-particularly-well-documented fact that within any one transaction all stop jobs complete before any start jobs are launched).
Having participated in community forums and lists for 15+ years, I'm not yet a fan of helping people who start their requests for helps with flames. Not every person reading the forum is uneducated as to the piece of software they are posting about, completely unable to tell something written primarily for desktop use from something that is equally well-suited to operating on servers. I help people out all the time on forums, and when I read something like this it makes me really want to let the poor soul suffer in silence.

However, I'm a nice guy and like to answer questions, if only so that others reading along might learn something. That and it gives me an opportunity to make you wade through something as unhelpful to you as your original post was to me before getting to the answer.

> How do I prevent shutdown of ANYTHING until that logging shutdown script has sent its email.

I'd focus less on preventing ANYTHING from stopping and just make it clear that your shutdown script has a dependency on those other services. Then systemd can happily shut down other things in parallel, without killing the services your script depends on until you're done with them. That's the whole point of a dependency-based anything.

+Michael Chapman seems to be on the right track here. 
Systemd.path and systemd-inhibit will definitely play a part. Suppose you could send an sms instead?
+Michael Chapman
As I read it, he's just saying he wants to 'keep them alive' until his service is done.

+Gavin Davenport yes, all you need to do is specify After= for each of the services your service requires.
Michael Chapman - many thanks for your clear and illustrative responses.

Carey Riley - Unfortunately I don't think i can use SMS - the payload being sent is too big.

Rich Freeman - I did not 'flame' systemd in my opening statement.
My feelings are around systemd are not yet positive, but it's fairly clear that expressing that feeling isn't welcome here. It looks like I have to just suck it up or go and do it with a non systemd OS.
+Gavin Davenport By all means whatever OS you prefer. Your approach is a bit like walking into a meeting of veterans meeting and saying, "I believe that anybody who joins the military is an unprincipled murderer. Oh, can anybody help me with my car just outside - it seems to be having engine problems." You're welcome to your opinion, but expressing it is going to be counterproductive to getting help from the very folks whose software you just knocked.

In any case, +Michael Chapman did post the solution, and there is a useful example on stackexchange of this as well (though it lacks any dependencies).

I just find your original statement a bit ironic in that the very first system I installed systemd on was a server, and I didn't adopt it on my desktop until probably a year later.
Rich - I think you're overreacting. If your head has been in this space (i.e having to fight systemd to behave the way you want) - great, bully for you. I would like to have a choice to use it (since this is the bazaar, right ?). It looks massive and goes against the grain of unix/linux, of small, simple, specific tools. Don't be surprised when people ask why they HAVE to adopt it when their old stuff works fine. It's not so much about preferring the desktop versus the server, it's the hiding/assimilation of a load of (small, simple, specific) underlying modules that work well - udev/inetd/sysvinit/many more. It looks like systemd wants to take over the world (i.e - manage all OS configuration). Don't be surprised when people challenge why they need it.

Maybe it's the way it's been put into Fedora - but 2 examples of why it doesn't yet work satisfactorily.
1. hostapd and network. hostapd puts layer 2 in place, network puts layer 3 in place. But hostapd won't start up until the interface is instantiated. Race condition. (Ignoring the fact that NetworkManager can't natively configure hostapd)
2. wpa_supplicant and network - same thing. network/networkmanager can't fully bring up network because it needs wpa_supplicant to get layer 2 up. Another race condition.

It's not very intuitive and I would like to be able to serialise things IF I CHOOSE that's what I want.

By parrelelising things - what problem are you aiming to solve ? Please, if it's not 'boot up time', enlighten me.
+Gavin Davenport Nobody HAS to adopt systemd at all. Like you said, their old stuff works fine, and their old stuff will continue to work fine forever. It just won't get any newer.

It sounds like your complaint isn't with starting services in parallel vs serial, but rather with dependency-based ordering instead of just manually specifying an order. The reason for the former is the same as why everybody uses dependency-based PMs now - if the deps are specified correctly you don't need to micromanage your load order on every host. I've been running dependency-based service management for 5-6 years now without issues.
Rich - if i want to use a modern fedora - I have to use it. Yes it's my choice to use fedora and I have to live with it.

Could you answer my question btw ?
"By parrelelising things - what problem are you aiming to solve ?"

I want to buy into it as it looks like it's here to stay. I just want to understand some of the reasons.
+Gavin Davenport "By parrelelising things - what problem are you aiming to solve ? " Turns out that out there, in real life..people do not start just a bunch of units..the biggest reported is around half a million.. then parallel startup matters quite a bit.
Are you saying that there are users who want to start ~500k distinct services ? Are there even that many different software packages in current distros ?

(by distinct I mean different software with varying dependencies on other stuff, not many instances of the same thing)

I don't think this answers my question. What [b]problem[/b] does it solve ?
Back to the subject at hand - is it possible to declare different (asymmetric) dependencies for the startup and the shutdown ?

To bring up - my service needs gpsd and a serial port to the cars ECU (device file discovered and named by udev). It can be started as soon as they are both present. Notably, it doesn't require any networking up.
To shutdown - my service needs pppd, network, postfix and iptables to be available (because I wish to email as part of the shutdown).

Is that possible ?
+Gavin Davenport​​ if you already have dependency-based services, then starting then in parallel is much faster and will never cause a bug vs starting then sequentially.

Again, you're focusing on parallelism when your issue seems to be on dependencies. You can have the latter without the former, and indeed that was what I was doing before I moved to systemd.

Why wouldn't you want your services to start in parallel if you've already gone thorough the trouble to switch from specified order to dependency-based order?

The main benefit comes from the movement to dependency-based services, not from parallelism. Gentoo made the switch ages ago and it worked out well.

As far as choice goes, you can always run last year's Fedora. It will work exactly the same in ten years as it did last year. It will work better if you maintain it. The only downside is that nobody will do it for you unless you're paying. 
+Gavin Davenport for the startup part.. write an udev rule that when the exact device becomes visible to userspace, it tells systemd to start the service. (ENV{SYSTEMD_WANTS}+="yourservice.service" .. to shutdown the unit must be ordered After=gpsd.service ..postfix.. and Wants=gpsd.service 
Rich Freeman: Good shout - it is a change of philosophy to the problem. I guess i (think I would) find it easier to untangle this stuff if I could switch it into a serialised mode to observe where it's getting stuck. With it all going on at once it's hard to see the wood for the trees. As you say, order is not the thing, it's dependency management.

I did notice (but obviously haven't practised enough) the systemd state and dependency tracking/displaying tools. There's quite a lot of new stuff to take on - as I said I find it a bit 'all encompassing' at the moment and I've seen a number of rough edges to it (Fedoras implementation)

It feels reminiscent of when Sun/Oracle adopted SMF into Solaris. Retooling and mindset change was required but what eventually resulted was a better answer.

Cristian Rodriguez: yeah, I saw that approach - nice bridge between udev and systemd. I'm not sure I used it successfully yet (its a shame I can't get to this machine to pull out config)
+Gavin Davenport It probably wouldn't hurt to have a debugging mode where it would serialize unit startups, and randomize their order (while meeting dependencies). This would almost guarantee random breaks a fair bit of the time if there was a dependency issue, which is what you want when you're debugging. Better still dump the RNG seed in the logs and make that a kernel boot option, so that you can get a deterministic startup. That would let you reproduce a failing boot and then confirm you've fixed it, and for those who want a deterministic serial boot they could just run in production with that seed and get a deterministic result every time (at the cost of non-parallel startup).

I've run into dep issues and they can be maddening since they are often not reproducible. They're usually not actually systemd issues, but bugs in units, mainly when you use less-common configurations. For example, if you run a DNS server and use it for resolution on the same host, you probably want to make sure that other services which need DNS at launch run after DNS is up. Since DNS servers aren't super-common on boxes that also run other services that is likely something that most distros don't set up by default. I also had some nfs dependency issues on my distro which I believe we sorted out, and I sometimes have an iptables/iproute save/restore issue on one of my containers.

I've yet to find systemd do anything other that follow the dependencies in the units - it just can be hard for distros to get these right since unlike with dynamic linking you can't just run ldd on a service and find out every other service it consumes.

The real issue isn't the parallel startup so much as non-determinate dependency-based startup. However, parallel startup combined with the fact that systemd is VERY efficient at launching processes tends to cause races when things aren't configured correctly. I didn't see as many issues when I used openrc in parallel startup mode and that is likely because the dependency model was simpler, it was more mature when I started using it, and with all the bash the reality was that services didn't start up nearly as parallel.
It's also important to remember that although systemd works upon dependencies, it really prefers not to have explicit dependencies. For instance, if you have a network service, it's usually preferable to have systemd manage the sockets for that service so that incoming connections are queued, not dropped, until the service has been started (either through socket activation or by other means).

Even if you can't have systemd manage the service's sockets for some reason, having the service handle them more intelligently -- e.g. by binding to INADDR_ANY+IN6ADDR_ANY_INIT or by using the IP_FREEBIND socket option -- can mean they're not actually dependent upon the network being available.


Popular posts from this blog

RHEL 7 and CentOS 7 syslog Rate Limit

Set Focus to Follow Mouse Cursor in GNOME 3

Configure rsyslog Server on Fedora