One of the things I think has generally worked well about “Linux” and the ecosystem on top of it has been the variety of userspace. There’s obviously some pointless things, but also some genuine innovation. It works well when upstream projects are structured in a way that they can be mixed and matched.
For Fedora CoreOS we are combining two technologies; Ignition and rpm-ostree. Previously they were used independently (Ignition with a ChomeOS style A/B updater) and rpm-ostree with the traditional Fedora-and-derivatives setup of Kickstart for bare metal, and cloud-init for clouds.
Putting the two together has been working well so far, but I’ve recently been working on support for root filesystem reprovisioning which is where the two projects intersect strongly. This has meant a lot of time writing code in the initramfs.
And the topic of this blog is “systemd is well designed” because the design for systemd in the initramfs is very flexible and also well documented. We’re continuing to support Ignition independent of OSTree, as well as OSTree independent of Ignition, while also having both of them work together. Further, the projects are written in different languages; Ignition is Go, OSTree is C, we have the usual (unfortunate) mix of shell script in there, and it’s likely we’ll add some Rust soon too.
This is where systemd excels; we can plug these things together in a coordinated fashion by writing unit files with careful dependencies. They can communicate however they want; in practice, writing files in /run is a common pattern.
Also worth noting is we’re using dracut, which is itself independent of systemd, designed to just implement the systemd boot sequence – our units plug into the “custom initrd services” section. And it all Just Worked. The systemd initramfs boot sequence (and the boot sequence in general) was designed long before either OSTree or Ignition were created, but it’s stood the test of time.
Colin Walters 
It’s nice seeing systemd appreciation. The UNIX “philosophers” out there are really tiring when they preach that PID 1 must do only one thing well, neither two things, nor zero things, no, one thing. The damn shell dwellers, enemies of innovation and always acting like they only have 320 KB of RAM!
I love systemd and feel it was one of the best innovations for Linux in a long time. I honestly can’t believe there are people that prefer the old days when there was no actual service manager. At work we’re slowly becoming more container focused and one of the things I always miss when constructing container services is the lack of systemd. I know if I’m constructing a proper microservice there should be no need for initialization ordering or service management, but it still feels like a loss.
Anyways, sounds like a fun project!
Who cares. I don’t want one system controlling my box.
It would help if the article would actually give some definite explanations WHY the author here believes that system is “well” designed. I understand that the author of course writes from a pro IBM Red Hat point of view since that is what he is paid to do – but I can not find within this article a single convincing argument as to how, where or why systemd would be well-designed.
It is massively over-engineered and complicated to use. That’s admittedly one strong selling point for support, but for casual linux users who want to change their system? That’s a dead end at once.
> initramfs is very flexible and also well documented.
Now I don’t agree with this, but here is one thing that fascinates me – IF you write that it is the most awesome documentation ever, why are you too shy to link in to the documentation? I mean, hyperlinks are not impossible in 2019 right? 🙂
https://www.freedesktop.org/software/systemd/man/bootup.html is already linked in the article – I think the problem is my blog’s theme doesn’t make hyperlinks obvious.
I could certainly imagine if the initramfs was just a big shell script it’d be simpler for the simple cases – but the `if nfs {}` and `if ostree {}` and `if ignition {}` etc. type conditionals would grow unmaintainable fast. The point of the blog is that I found the systemd design to work really well for what I was trying to do.
> but I can not find within this article a single convincing argument as to how, where or why systemd would be well-designed.
Articles and documentations exaplaining how systemd is actually the best implementation for system and service management out there are plenty in the internet.
Point is, though, if you really want to listen or keep going stubbornly to say that it’s don’t true and that INIT was better, although it’s not.
Also, saying to people that they have to say that systemd is well designed because they’re paid to do so, it’s just what I mean when I say that you don’t care about other’s point of view.
> It is massively over-engineered and complicated to use.
Ok, so you’re expecting for people to explain why they think systemd is a great piece of software, while you’re not going to explain why it is not. A bit unfair, don’t you think? 🙂
Leaving it aside, systemd IS well engineered because it is a service manager that can actually help system administrators and developers to interact with every part of the operating system in a constent way, while maintaining a CLEAR separation of duties in every component it is built of.
For the first time in its history Linux can have:
– a system that can tracks status of services (especially services that forks themselves), mount points, boot (with systemd-boot), network and much more other;
– a system that can integrate itself in a RESTful way (thanks Red Hat for giving us the ability to export journals in JSON) with more complex monitoring solutions;
– a system that can be configured in a SANE and consistent way: systemd configurations, units and everything related are based on the INI format, that combines the human-readability with a more structured and organized way to store informations;
– a system that can use special kernel features, like namespace and cgroups, to provide resource limiting capabilites to everything it is going to manage;
– a system that can produce rich logs that includes not only the message from the system/applications, but also much more other informations such as the SELinux context or the scope the application is running into. And as an additional feature, you can have indexing, so searching for a specific service in a specific range of time and date is a piece of cake, in constrast to parse that with grep+sed+awk and hope that your regex is absolutely correct and the info you obtained is really all you wanted. And if you combine that with the JSON-exporting capability you can actually integrate all of that with portals and monitoring where you can type a query and obtain info in real-time.
and every of these selling points are BUILT-IN. No wasting time to create huge and unmaintainable shell scripts to try to provide a small-fraction of these features (I’d like to see how you are going to export every log in JSON to integrate them with a REST service while maintaing all the features systemd provides, such as indexing and filtering, with just Bash and write something that don’t suck 🙂 ).
I don’t want to waste my time to debug a stupid script shell. This is NOT what I’m getting paid for. This is what geeks who don’t work in IT thinks is our work. My work is to solve difficult incidents that are damaging the business of my employer, find new ways to improve the infrastructure I work with and reduce downtimes by automating (with Ansible, not shell scripts 😉 ) useless stuff.
Because IT is growing more complex every day, and we don’t have the time we had 20-years ago that we could spend to write obscure regexes that have to parse >5GB of logs. Not anymore. We have to focus to the bigger picture in order to achieve what an always-connected world wants: 24/7 online services. And systemd is part of this big game because it tries to solve a lot of these issues.
Is it perfect? No, it’s not. It has its own shortcomings. Yet it is the only solution out there designed for real workloads that is trying to be useful.
Now, please just explain why systemd is over-engineered, but please stop saying it is complex for casual users.
Causal users don’t know and don’t even care about systemd, cause they are not sysadmins. They don’t even care about the desktop anymore. Casual users just use their smartphones to do what they want.
The only people are complaining about systemd, are people that until it was born don’t even know what INIT was, but now for some reason (like to consider themselves hipsters) it is become the freedom -1 (0 is already taken 🙂 ).
Thanks! How much work needs to be done to add ignition support for debian, ubuntu and other systems?
I think that ignition can’t fully replace cloud-init, cloud-init works for netbsd, freebsd and windows. Also linux kernel developers do much stuff to add ability to resize rootfs online, grow partitions and other.
You work for IBM/Red Hat, so, of course, you promote systemd.
Systemd is not “innovative” as suggested by the above Koolaid drinker, and it sucks.
It wouldn’t be so bad if there weren’t so many arbitrary dependencies to systemd. The only reasons for such unnecessary lock-ins is greed and politics. It’s more difficult for Red Hat (and IBM) if they actually have to compete on merit without the aid of artificial lock-ins.