Enough with "world files are great". What are they?

I only recently realized that I've only talked about using world files, but never really explained their purpose. I'll try to break down what they are used for, and why knoming how they work is a useful thing. World files are a record of everything that is installed on the system explicitly. When you install, say Firefox, it will not be the only thing that needs installed. It uses shared libraries for audio, video, and many other things, but you generally don't care about any of that. You want Firefox, so you explicitly pull that in. All of the other dependencies are the package manager's problem, and are not recorded in a world file. This is a conenient place that you can go to in order to eassily see what's intentionally installed, let you clean out lines of things you don't care about and even things you have forgotten about over the ages.

What else can I do with world files?

Occasional cleanup is a good thing for any mutable system, but world files can also be used to reconstruct a system. If you back that simple text file up, you can transfer it to a new system, and effectively sync to that, and things will be added and removed automatically in order to make the file in sync with the system. Package sets also let you install sets of packages and enable/disable them together. This may be useful if you want to switch audio subsystems from one to another, and possibly even back. All packages can be swapped out at a time to ensure that nothing can be forgotten, and even lets you know after time what packages are used for that task. It's a great organizational tool, especially as systems get older and you forget what's installed, and why it's there. Because world files are almost exclusive to Gentoo and Alpine, I wrote pmm to emulate this behavior on other platforms.

So what about Nix?

Nix takes this concept further. Not only can you treat it like a "normal" package manager, but by default, it has access to generations. This allows you to roll back to previous versions if you decide that you don't like the changes. When paired with add-ons like NixOS, darwin-nix, or home-manager to extend this to your operating system and give you files to define "what the system looks like" similar to a world file. This isn't the limit to Nix however. You can tie other things like configs into the system, allowing Nix to manage not only installed packages, but how they are configured. You can take an nginx server config along with the package and bring it to a new system all in one tool as opposed to having packages and configs not be aware of each other in terms of the package management. Tooling like Nix Flakes is currently marked as unstable as the API is changing rapidly, but this allows you to even version lock packages or dependancies in order to increase stability and reproducability. This all goes far beyond what world files are capable of, but none of it forced on you other than generations, which is hard to complain about if you have a bit of free disk space to store it. It also allows having multiple versions of the same package installed, which is not common on most package managers, and is useful for having multiple versions of libraries, or just testing new versions before switching. Not having that ability has caused a lot of breakage on other systems, and is one of the reasons that projects like Flatpak are trying to solve.

That got quite a bit off the scope of world files, but that's why this has the "ramble" tag. Hopefully this off the wall post helped someone understand why world files are so great, or at least learned something in the ramble.

How I'm currently using nix

I've gotten a few questions surrounding how I'm using nix at this point in my workflow. Even without knowing much of it yet, I've managed to work a bit of it into my daily workflow. I'm typing this on my laptop which I rarely need to rely on as a test bed for NixOS. This lets me play with things that I may or may not like on a daily use system, server, ect, all without interrupting my daily workflow. Thus far, it's forced a few changes to my dotfiles, which isn't super uncommon for supporting a new Linux distro. Most notably, I had to remove launching Sway on tty1 login, though it's likely a lack of understanding on how that works.

My primary use thus far for nix has been using it with home-manager to replace my userspace on my Mac Mini and now this laptop. The ultimate intent is to be able to keep all of my userspace portable to my servers as well, or if I end up back on Fedora, I can still have a uniform userspace without touching the core immutable OS underneath it. With home-manager acting as a stand in for world files in the OS/pmm, it keeps it all centralized. I have not decided if I want to use tooling such as nix-darwin due to it's increased chance of breaking on MacOS updates. If my userspace is broken, that's one thing, but I'm not sure how much power it has over the host system and if it would be able to soft brick on updates. Further research and testing will need done.

Because of the lack of nix-darwin, unless I'm missing something, I don't believe that I can get full integration with GUI applications on Mac, which causes some minor inconeniences. The first place that I had noticed this is having emacs installed through homebrew which I tend to do things like compiling and other tasks through. Because it's not tightly integrated into tooling like nix-direnv, it can't see the tooling made avaliable. It's not a deal breaker to me as I'm used to working in an external terminal already, though I do intend to solve it if possible and the gains outweigh any costs associated. Only time will tell.

What is nix, and why should I care?

This exact question is why I avoid nix for so long. Specifically the "what is nix?" half. Nix is not one thing, but a collection of many things when most people say "nix". I wanted to clarify what things are to the best of my knowledge and give others what I had trouble finding in one convenient location when I was not knowing anything about nix in hopes that you understand better what things are, why I'm so pedantic, and possibly enable you to give things a shot with at least a direction to go that I had such trouble finding. With that out of the way, what the heck is it?!

Nix

Nix seems to be a catch all term for anything related to core tooling, the language, ect. If you are familiar at all with ZFS, it's likewise "a filesystem" but it's also an ecosystem of tooling that you gain massive benefit from using as opposed to using traditional tooling in it's place. Nix is much like that in regards to it being a set of tools that integrate tightly with each other.

The other definition that can be applied to nix is that it's a package manager. It's a declaritive system using Nix Expression Language (covered below). I encourage you to read about the nix package manager even as there is already a great breakdown of why it's great there, and I don't need to rehash that as I can't do a better job.

Nix expression language

Nix expression language is written in a functional style. It's used to describe anything that is in the nix ecosystem. All tools are written in this one style, which allows you to have less languages to learn to create anything that you want to integrate into the system. Because of it's pure functional nature, this means that results are in most cases able to be reproduced bit for bit identical. This reduces the classic "It works on my machine", and hard to debug errors down to almost none. If this sounds similar to what Docker or other OCI containers solve to a point, you would be correct. This extends to much much more than containers though.Nix expression language great for writing build systems, making reproducable containers, creating dev environments, building operating systems that are bit for bit identical on real hardware, the list goes on.

Nixpkgs

This one is simple. It's a list of packages avaliable to the nix package manager. There's several repositories avaliable, it supports something similar to Gentoo's overlays or the AUR from Arch. You can read more about it here.

Home-Manager

I've talked about this on more than one occasion, and never fully defined what it is. home-manager is tooling that allows you to define how user's files, applications, configs, ect are all set up. As opposed to installing packages one by one like you can use nix-env to do, it brings it all together similar to how pmm does for many systems, but gives you all of the power of Nix Expression Language to be able to set up logic instead of doing manual work. It can invoke basically "anything the nix ecosystem can do", but that's the simple answer.

NixOS

NixOS is what happens if you take home manager, and give all of that power to your operating system. Have a fully redeployable operating system on real hardware that can describe not only what's installed, but how it's configured from networking, to system services, to what docker containers will be running. All within Nix Expression Language. It has integrations with home-manager to allow everything to be fully integrated while still offering you portable users. No more installing a distro and then configuring it, worrying about config file changes with updates lingering and breaking over time. It's all written in Nix, and will all work every time, exactly the same way. Even if you try to manually edit config files, they are immutable and regenerated so nothing can really damage the system from the outside as you can simply regenerate everything. Roll back to previous definitions, all without filesystem snapshotting. It's fully extensible and the only limits are what time you are willing to put into it.

Nix-Darwin

While not as integrated as NixOS, this allows you to manage many aspects of MacOS in a similar way. Nix-Darwin has integrations with home-manager so you can get most of the power without leaving all of your Mac specific apps behind.

Nix Flakes

If you ever search Nix, you'll see this being talked about constantly. For the moment, I'm avoiding it until it gets marked as stable, but it looks promising. As I am not using this experimental feature, I'll suggest you go read about it here.

Ok, but what can I do with all of that power?

I can't even begin to pretend like I can understand the full power of what is possible, but I'll list a few things that you may find interesting. These are things that I may play with and I'll document anything that I do and share my thoughts.

• Nix-Direnv: Portable dev environments
• Comma: Run ephemeral commands
• Better PMM:: Manage packages in a central, portable, place.
• Manage docker: Docker-compose, but not learning a new language
• Nixified terraform: Why learn terraform when you already know nix?
• CI/CD: All devs need CI/CD. Why not bring that in too?

Further reading

While all of this sounds insanely appealing, there is a learning curve. Nix doesn't have to be a massive time/effort investment, but it can be if you want to get the most out of it. I believe if I stopped where I am, I would still have massive gains from where I was learning nothing more. If you want to hear some thoughts from someone who's used Nix for many years, and their thoughts on it, I'll link to Hilssner's dotfiles. For those unaware of who that is, they were the creator of Doom Emacs and their content is always a joy and a laugh to read. I highly recommend reading their FAQ as it seems to cover if you should learn nix in a way that's both funny, and brutally honest, yet remaining a nix user. I'll also link their decent into madness as it's yet another good read for anyone new. While they have quite the indepth take on it, they make a lot of good points to think about if you want to take nix seriously, or just use it as a neat toy as I have been. Until the next post, I hope you enjoyed, and were entertained by hilssner even half as much as I am.

Portable dev environments

In my journey to uncruft my system, I started looking at even uncrufting my user's footprint as well. I found myself bringing in a ton of things for misc dev work. One project I may use Nim, another Python, another I need Zola for this blog, and it started adding up. I knew there had to be a better way. In comes nix-direnv. If you have done development, you may already be familiar with direnv which is great for keeping your system clean in general. I won't repeat everything when there are good links to things, but direnv is built to keep your env vars that are project specific out of your local system, and nix-direnv extends that now to packages! Instead of rambling, I'll show you a simple demo that I actually use on this blog.

You'll need two files for each project.

.envrc

use nix

shell.nix

    { pkgs ? import <nixpkgs> {}}:

    pkgs.mkShell {
      nativeBuildInputs = with pkgs; [
        gnumake
        zola
      ];
    }

You'll obviously want to install direnv and nix-direnv in whatever way you would normally. I simply included them in my home-manager file. Deperinding on how you have your shell installed, you may need to add a hook for your shell to automatically load the environment when you change into the directory. Since I don't allow nix to manage my ZSH configuration, I simply added it to my .zshrc, though it may be more readable to most people with the shown example.

# .zshrc
eval "$(direnv hook zsh)"

If you have a different shell, follow the docs for your specific shell. That should be it in terms of global setup. .envrc files won't be loaded by default for security, so you can run direnv allow . in the directory once and it will be allowed to load from that point on when you cd into the directory, and unload when you leave.

After throughts

Not only does this allow you to keep your system cleaner by keeping env vars and packages out of the system and user's packages, it allows you to keep that portable for anyone that has a nix system with direnv and nix-direnv so there's no need to guess what packages are missing for nix as they can auto populate. If you want to adventure further into it, you can even do things like fetching remote envs, version locking to keep all users on the same dev environment, and much more, but that's outside of the scope of this. This is just meant to give you something to start with, or it may already do everything you could possibly need. Hope that helps someone understand what it does so you don't have to dig up everything like I had to.

Nix is... wild

So I decided that I was going to try nix as my attempt to remove my state from the system, and didn't realize the size of the rabbit hole that I had fallen into. I thought I was getting access to a package manager that I'd bend into shape like I had with Gentoo in the past with their world files. Turns out that Nix does that really well. I figured that I would enjoy parts of it as I've moved closer and closer to it in the years without realizing, but I didn't know what I would uncover. Before I get sidetracked, let's get this out of the way.

What even is nix?

Nix isn't an operating system. It's technically the name of a package manager, and NixOS uses that package manager. Sounds simple enough, but Nix is more of a language or an idea than either of those things. I won't pretent to have any sort of massive knowledge on nix, but there is so much out there that I wanted to make it known that you will likely be lost if you search for anything Nix related. I'll try to document what steps I'm taking and update with how I'm learning my way through it to maybe help someone out.

So what's the status?

I've started transitioning my personal state into Nix to some degree. If you want to see all of the files as they are when this was written, or as they are live, you can follow along as you please. I've started by porting out into a few modules. There's nothing special about not putting everything in one file per se, but it does allow me to not repeat things and have a more central place to edit things I want on all machines. All of this is powered by home-manager which is a nix community tool that's meant to be much like the core nix, but let you seperate users from the core system. It can also be integrated with NixOS itself, but I'll try to keep it simple for now.

what's it look like

Home-manager files are closely related to nix, but have some slight deviations, though they use nix under the hood. Here's an example from farnsworth (mac mini). I'll trim a bit to make it fit better. Mostly self comments.

    { config, pkgs, lib, ... }:

    {
      home.username = "kdb424";
      home.homeDirectory = "/Users/kdb424";

      # Set current stable
      home.stateVersion = "22.05";

      # Checks that home manager is in sync with release
      home.enableNixpkgsReleaseCheck = true;

      # Let Home Manager install and manage itself.
      programs.home-manager.enable = true;

      # pulls in other nix files
      imports = [
        ../modules/common.nix
        ../modules/mac.nix
        ../modules/commonGUI.nix
        ../modules/devel.nix

      ];

There's not really much in here. I've tried to leave the main home.nix manage only minimal things like what types of things it wants to pull in from other nix files and keep it clean. This is similar to how I managed pmm files, or using user defined sets in Gentoo/Alpine. It just lets you group things by topic and turn them all on/off in sets by chosing imports. These files are just collections of packages that look like this as a base.

    { config, lib, pkgs, ... }:

    {
      home.packages = with pkgs; [
          firefox
          nim
          ect
      ];
    }

I try to keep things organized like common tools that may be on all machines, or machines of a type, and have just them at this point to simply replace a world file with only one non standard thing in my common.nix

    { config, lib, pkgs, ... }:

    {
      home.packages = with pkgs; [
        git # shortened for readability
        vim # there's more up here in
        ect # the real file

      ] ++ lib.optionals stdenv.isDarwin [
        coreutils # provides `dd` with --status=progress
        wifi-password
        time # GNU time
      ] ++ lib.optionals stdenv.isLinux [
        iputils # provides `ping`, `ifconfig`, ...
        zsh # already on mac
        libuuid # `uuidgen` (already pre-installed on mac)
        emacs # installed with brew on Mac
      ];
    }

The use of lib.optionals will let me do an OS check and allow me to install things I determine are missing exclusively from the other. I like to try to keep everything as "linux feeling" as I can in a terminal, where mac has some things already, and Linux should likewise have them.

That's it?

Oh, not even close. That's just where I am in terms of "allowing nix to eat my files" as it can absorb even the configs related to your applications to keep them central to Nix, and easier to manage as things update and config file format drifts. They won't be silently laying around as cruft, or partial cruft if nix knows what they are, and is how they got there. I haven't decided if I want to give my dots up to Nix, but I'm playing with NixOS and it will be allowed to manage system config files wherever possible. We'll see where it goes, but that's my start into nix. Hope you enjoy reading about this other world even half as much as I'm having fun discovering it.

BONUS!

I stumbled into comma, which allows you to tell nix to fetch a command for just the line it's run on and then toss it away. This makes one off tasks, or tasks like fetching a command to fetch itself, and yes, I've tried to do that recently, insanely easy. I'm sure there's more that I haven't discovered about it's usefulness, but here it is in action.

╭─ > ~/src/zola-blog > main >
╰─ cowsay "I'm not installed"                                                                                                                           ─╯
zsh: command not found: cowsay

╭─ > ~/src/zola-blog > main >
╰─ , cowsay moo                                                                                                                                         ─╯
 _____
< moo >
 -----
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

╭─ > ~/src/zola-blog > main >
╰─ cowsay "I'm not installed"                                                                                                                           ─╯
zsh: command not found: cowsay