I've been on a quest lately to get computers to just "go away"™. When I've least expected it, they have alwasy broken. Taken an update, or just decided one day that it didn't want to boot, or acted in a strange way such as no audio, out of date configs, the list goes on. These days, I am not in the mood to deal with breakage at the most random of times. Things like my laptop get booted at most a few times a year, and I really don't want it to be broken on that rare time that I need it. This hasn't just applied to the laptop, but servers as well. For a long while, I was able to just avoid 99% of it by not using SystemD, but it's now even hitting my Artix systems, both the laptop, and a server that I thought I could be too lazy to switch away from because there was no SystemD. That was a mistake it turns out, and has lead me to where I am now.
Let's take a step back from rage and consider that tooling has existed for years to solve a ton of this, and I'd even been using some of it for this reason. Docker. Docker has taken the almost everything that I serve, and made it trivial. It basically doesn't break unless someone stops maintaining containers, and is a voodoo box of magic that works. There's two sides of code. The side that is dirty and impure that can break, but that's quite minimal. Writing files, touching the network, ect, can't be clean and have a safe promise that it will work, but the code that actually runs the logic, it mostly can. No more operating system to worry about. There's two things that make Docker (and other OCI containers) great.
Because systems aren't mutable, you limit the cruft to only known directories that are shared into the container. When you update it, you aren't ugrading a bunch of old packages, worrying about cruft left behind like old libraries, old programs you forgot to remove, config files that may be out of date, ect. All of it is just... Gone. When you "update", you throw everyhing away, except your limited scope that you share inside the container, meaning that the only data that can go bad, is that little data. If your image is broken, just roll back. If you want to run a completely different image based on another container, go for it. The service still runs, and can be rolled back or forward almost seamlessly, and when things do finally go wrong with your shared data, there's not much to look at to update or debug.
One could argue that declaring the outcome as opposed to the input directly is a better solution. Docker does this. You state what you want the container to do, but never tell it how to get there. Obviously someone has to create that at one point or another, but the useful bit is what comes out. Many of the best package managers for *nix systems are this way. Portage on Gentoo, Apk on Alpine. The work on the world file and decide "how to get there", but the user controls only the world file. Directly or indirectly. That only covers packages, and not configurations, but it makes rollbacks more possible, but it doesn't completely solve the problem, and why we need both declaritive and immutable together.
Many teams have been working to solve this problem for years. Each have their own drawbacks, but I'm giving some of them a test drive to see how they do for my use. Fedora Silverblue is an immutable OS that uses ostree to build in layers, much like docker. Very limited parts of the system are mutable. This leaves almost nothing to damage when you take updates, or add/remove software as you are intended to use containers such as a toolbox or distrobox which are likewise using OCI containers that can also be rolled back if there are problems. If you somehow take a broken host system update, you can roll that back, even if unbootable as the old state is stored and can be loaded from your bootloader menu as if the update never happened. This allows for easy playing with your system with almost no chance that damage can actually happen, and you throw away whatever you don't want knowing you can easily get it back, almost never even needing a recovery disk unless the bootloader gets damaged. I've started messing with Fedora, including making my own spin on Silverblue, but running Sway that matches my dotfiles at the time it was pushed to git at least. Feel free to give it a look here if you would like to see how it works. It took me a long time to get it running, and only managed to use examples from others like Fale's Desktop work, which my work is based on.
Another option that I'm messing with is Nix and NixOS .
Nix is able to run on most platforms, such as Mac/Linux and even Windows. Where
as my tool pmm was able to integrate with
other package managers to emulate world files, Nix can accomplish similar
effect, but keeping everything within a container of sorts, exposing things to
the outside system. I've started porting most of my packages out of
homebrew on my Mac, and into Nix. This allows me to make it
portable through Mac/Linux and, even useless to me but it can, Window. On my
servers, I need to do misc tasks to manage the server, but almost none of my
tooling that's nice for me to use has any bearing on the system, and only adds
more cruft that can prevent the machine from working properly. If I can't use
exa instead of
ls on my shell, it doesn't matter. I need the host to stay
stable, and running as intended. Keeping everything contained in Nix also allows
me to seperate the "me" from "the system" which leaves everything more porable.
You can see my progress, as well as how I'm using that
I'm using home-manager to keep
my personal configuration and tooling away from the system, while also using
NixOS on my laptop to declare "how the system is supposed to look" ignoring what
my user intends to use on the system. I haven't gone the full Nix route,
rewriting all of my configs, docker containers, ect all in their format as I
want to keep things portable, but we'll see if that becomes a solution to
problems that I hope won't happen yet.
I don't have all the answers, and I don't know what will go wrong next, but staying stagnant was going to leave me not wanting to remember that computers even exist. Immutable and declaritive systems are not what people are used to. It's analogous to someone that's only written procedural code and going to a fuctional language. It's a complete differet mindset. Problems have to be solved in new ways, and things that didn't used to be a problem, may now be one, but at least it should act consistent with this model. We'll see if it's consistently bad, or good, but I'll take consistency at this point over anything.