What systemd-delta actually shows

Why this matters more than reading one unit file

First pass: list all local changes

Use systemctl cat to see the backing files that matter

A practical example: add a restart policy as a drop-in

When systemd-delta shows masked

The rollback path: systemctl revert

A good troubleshooting workflow for “why is this unit behaving differently?”

Common mistakes to avoid

1. Editing the vendor unit directly

2. Forgetting daemon-reload

3. Treating “disabled” and “masked” as the same thing

4. Replacing a whole unit when a tiny drop-in would do

References

Final thought A lot of Linux debugging turns into archaeology. A service behaves differently from the vendor default, but nobody remembers why.

Maybe someone added a drop-in six months ago.Maybe a package shipped a unit update.Maybe the unit is masked in /etc/ and you are staring at the wrong file in /usr/lib/. This is exactly where systemd-delta earns its keep. If you use systemd regularly, I think systemd-delta should be part of your standard troubleshooting kit alongside: This guide covers the practical workflow: systemd-delta finds configuration files that override lower-priority systemd config. According to systemd-delta(1), the general priority order is: The same man page also documents the main result types you will care about: For unit troubleshooting, extended, overridden, and masked are usually the most useful. systemd.unit(5) documents that unit files are loaded from a search path, and files found earlier in that path override files found later. On this Debian host, systemd-analyze unit-paths shows system unit lookup starting with paths like: That is why reading only /usr/lib/systemd/system/foo.service is often misleading.It may not be the effective configuration at all. systemd.unit(5) also documents that drop-ins in /etc/.../*.d/ take precedence over drop-ins in /run/, which in turn take precedence over /usr/lib/. On my host, that immediately showed both a tmpfiles override and several unit drop-ins.Your output will vary, but the point is the same: it tells you where local behavior differs from vendor defaults. If you only care about system units, narrow the view: If you want just the most useful override categories: And if you want diffs for changed files: That one command saves a surprising amount of time. Once systemd-delta tells you a unit is interesting, switch to systemctl cat. systemctl(1) documents that cat prints the unit fragment and its drop-ins, with file names included as comments.That makes it one of the fastest ways to answer: You can also ask systemd where it loaded the files from: That is especially useful when a package ships a vendor unit in /usr/lib/, but the actual behavior is coming from one or more drop-ins under /etc/systemd/system/ssh.service.d/. Let us say you want a simple local override for ssh.service on Debian or Ubuntu.(If your distro uses sshd.service, substitute the real unit name.) Create a drop-in instead of copying the whole vendor unit: Reload systemd's view of unit files: Now verify the result three ways: Then, if the change is intentional, restart the unit: Why use a drop-in here instead of replacing the whole unit? Because it survives vendor updates more cleanly and keeps the local intent obvious.systemctl edit does this interactively, but writing the file directly is often easier to automate and audit. A masked unit is not just disabled.It is blocked from being started at all. systemd.unit(5) documents that a unit file that is empty or symlinked to /dev/null appears with load state masked and cannot be activated. To see masked items only: If a service refuses to start and the error feels weirdly absolute, check for masking early.It is a common cause of confusion after old troubleshooting sessions or package cleanup. This is the part many people forget exists. systemctl(1) documents that systemctl revert UNIT removes drop-ins and local overriding unit files for vendor-supplied units, and also unmasks the unit if it was masked. That makes it a clean way to get back to the packaged version. A few important details from the man page: That is a much safer habit than manually deleting random files and hoping you found all the relevant overrides. This is the sequence I recommend: If you suspect local config drift, add: That usually gets you to the answer faster than opening /usr/lib/systemd/system/*.service files by hand. Avoid changing files under /usr/lib/systemd/system/.Package upgrades can replace them, and the local intent becomes harder to track.Use a drop-in under /etc/systemd/system/UNIT.d/ unless you truly need a full replacement. systemctl(1) is explicit here: daemon-reload reruns generators, reloads unit files, and rebuilds the dependency tree.If you change files on disk and skip reload, systemctl cat may show newer content than the manager is actually using. They are not the same.Disabled means a unit is not enabled for automatic startup.Masked means it cannot be started at all.systemd-delta --type=masked makes this easy to spot. If your change is something like: then a drop-in is usually the cleaner move. Official documentation and references used for this article: When systemd behavior looks mysterious, it often is not mysterious at all.It is just layered. systemd-delta shows you the layers.systemctl cat shows you the files.

systemctl revert gives you a clean escape hatch. That combination turns a lot of vague “why is this service weird?” sessions into a short, repeatable audit instead. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse