Recently I needed to learn more about systemd. The man-pages
are great, but here are a few semi-surprising points.
The entries in /etc/fstab are dynamically converted to systemd
mount units, and mounting is managed by systemd. After changing
fstab, you can invoke systemctl daemon-reload to regenerate the
dynamic config, and systemctl list-units --type=mount to list the mount units. Try the latter now.
This means other units can depend on mounts being ready, using the
RequiresMountsFor unit
directive.
Even though an fstab entry and a hand-written mount unit is supposed to be
equal, fstab is more equal - there are certain magic fstab options that don't
(yet) have a hand-written mount unit counterpart. Specifically,
x-systemd.device-timeout= lets you specify how long the mount unit
should wait for the device to appear. Also, x-systemd.makefs will
format the disk if it doesn't yet contain a filesystem, which is handy when
creating nodes and disks automatically. Therefore using fstab is superior to
hand-written mount units.
Sidenote: cloud-init's mounts module is not a good
way to write fstab entries, if the devices attach to the instance later - since
cloud-init will check if the device exists, and if not, omits adding the fstab
entry. Rather, use bootcmd to write the fstab entries manually.
Which device name to use? When attaching multiple EBS volumes, the
device name (/dev/nvme...) doesn't hint about the identity of the
device. But you can use the /dev/disk/by-id/....volXXX stable
identifier based on the EBS volume id.
There are orthogonal but interrelated concepts of startup ordering between
units (Before/After) and dependencies between units
(Wants/Requires/...). You can inspect unit dependencies by
issuing systemd-analyze dot the-unit, which will output the
ordering and dependency relations in graphviz format.
Dependencies control what other units a given unit will pull in to the activation set (transaction), if the given unit is to be activated itself. But it doesn't impose ordering.
Ordering relations take effect only if both units are activated in the same transaction. They are attempted to be activated in the same transaction, if they are both transitively pulled in by a top-level unit to be activated. But most often, they will be direct dependencies.
What about restarts and dependencies? I would have naively assumed that a
restart happens in response to a failure, but there was no mention in the
systemd docs that after a successful restart of a failed unit, its reverse
dependencies would be restarted as well. As far I understand now, I was
wrong. In systemd terminology, restarts don't yet cause a unit to fail.
Only exhausting the restart limit will transition the unit to a
failed state. There are no automatic attempts to recover from the
failed state.
So what determines the set of units to start up during boot? The pseudo-target multi-user.target's dependencies.
This is why you often see a reverse-dependency
WantedBy=multi-user.target in the Install section.
What is the Install section, by the way? The reverse dependencies
listed there only get added, if you enable a unit using systemctl enable the-unit (enabling actually just adds a symlink).
So a unit can declare a reverse-dependency on multi-user.target - that is, the potential to be auto-started during boot -,
but users can actually control if they want the auto-start to happen.