Tools

Tools: Report: Keep Your Base OS Clean: Practical `systemd-sysext` for Linux Tools and Overrides

2026-04-20 0 views admin
Tools: Report: Keep Your Base OS Clean: Practical `systemd-sysext` for Linux Tools and Overrides

Keep Your Base OS Clean: Practical systemd-sysext for Linux Tools and Overrides

What systemd-sysext is good at

What it does not do

Anti-duplication note

Prerequisites

The compatibility rule that trips people up

Build a simple directory-based extension

Activate it

Remove it cleanly

A realistic use case: layering in a locally built binary

Mask an extension without deleting it

Troubleshooting checklist

1. Name mismatch

2. OS compatibility mismatch

3. Wrong paths inside the extension

4. You forgot to refresh

5. You expected writable merged paths

When I would use packages instead

When portable services are the better tool

Final thoughts I like keeping the base OS boring. That does not mean the machine has to stay limited. It means I want a clean line between the core system and the extra bits I only need sometimes, especially on hosts where /usr is meant to stay stable. That is where systemd-sysext gets interesting. It lets you merge additional files into /usr and /opt at runtime using overlayfs, without permanently modifying the host tree. Unmerge the extension, and those files disappear again. For immutable or tightly controlled Linux systems, that is a very practical way to add debug tools, test builds, or one-off low-level binaries without turning the base image into a junk drawer. In this guide, I will show a safe, directory-based workflow you can actually use. According to the systemd-sysext documentation, system extension images are meant to extend /usr and /opt dynamically at runtime, and they are especially useful when the base OS image is read-only or intended to remain unchanged. The merge is read-only, and while active, the host's /usr and /opt also become read-only. That makes systemd-sysext a good fit for things like: It is not a general-purpose package manager. There is no dependency solver here. The docs are pretty explicit about that. A few boundaries matter: If you need to deliver service units in an image with tighter isolation, portablectl and portable services are the closer fit. If you want runtime config layering for /etc, look at systemd-confext, not sysext. Recent posts already covered systemd-delta, systemd-tmpfiles, socket activation, systemd-oomd, and other systemd operations topics. I am intentionally taking a different angle here: runtime extension images for /usr and /opt, not unit override auditing, cleanup policy, or service lifecycle tuning. Check whether the tool exists: On many systems, extension images are searched in: For actual installed content, /var/lib/extensions/ is the normal place to use. Every sysext image needs an extension metadata file at: NAME must match the image or directory name. That file is checked against the host OS metadata. Per the man page, the extension's ID= must match the host unless you deliberately set _any. If SYSEXT_LEVEL= is present, it must match. Otherwise VERSION_ID= is used as the compatibility check. For a directory named debug-tools, the file must be: Let us create a tiny extension that drops one helper script into /usr/local/bin and one documentation file into /opt. Create the compatibility file: Add a small helper script: Add an optional file under /opt: Refresh sysext state: If the extension is accepted, you should now see the files through the live host tree: If you want to see all recognized extensions, use: To make the files disappear from the merged view: To update after changing files inside the extension directory: That refresh flow is the one you will use most in practice. One of the documented uses is to stage a newer build of a low-level component without rebuilding the whole base OS. For example, if you have a Makefile that supports DESTDIR, you can install into the extension directory instead of the live root: That gives you a reversible way to test files as if they were part of the base image, but without permanently mutating /usr. There is no classic enable or disable toggle per extension. Installed extensions are activated automatically at boot if systemd-sysext.service is enabled. But the docs provide a neat masking trick: create an empty directory with the same name in /etc/extensions/. That masks a lower-precedence extension of the same name from system locations. If your extension does not appear, check these first. The directory name and extension-release.NAME must match. For example, this is valid: If the extension says ID=ubuntu and the host is Debian, sysext should reject it. Likewise, SYSEXT_LEVEL= or VERSION_ID= must line up with the host metadata. Only /usr and /opt are merged by sysext. If you put content under /etc/myapp, sysext will ignore it. After adding or removing files in /var/lib/extensions/..., run: While sysext is active, the merged /usr and /opt views are read-only. I would still prefer normal packages when: systemd-sysext shines when the host image is treated as a base artifact and you want optional or reversible layering on top. This trips people up because both concepts involve image-based delivery. The systemd documentation explicitly calls out that difference. I would not use systemd-sysext everywhere. But on a host where the base OS should stay clean, predictable, and easy to reason about, it is a sharp tool. You get a reversible layer for binaries and support files, and the operational model stays simple: build the extension tree, add the compatibility metadata, then refresh. That is a nice trade, especially when the alternative is "just copy it into /usr/local and hope we remember later." Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Code Block

Copy

systemd-sysext --version systemd-sysext status systemd-sysext --version systemd-sysext status systemd-sysext --version systemd-sysext status /usr/lib/extension-release.d/extension-release.NAME /usr/lib/extension-release.d/extension-release.NAME /usr/lib/extension-release.d/extension-release.NAME /usr/lib/extension-release.d/extension-release.debug-tools /usr/lib/extension-release.d/extension-release.debug-tools /usr/lib/extension-release.d/extension-release.debug-tools ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 sudo mkdir -p /var/lib/extensions/debug-tools/usr/local/bin sudo mkdir -p /var/lib/extensions/debug-tools/usr/lib/extension-release.d sudo mkdir -p /var/lib/extensions/debug-tools/opt/debug-tools sudo mkdir -p /var/lib/extensions/debug-tools/usr/local/bin sudo mkdir -p /var/lib/extensions/debug-tools/usr/lib/extension-release.d sudo mkdir -p /var/lib/extensions/debug-tools/opt/debug-tools sudo mkdir -p /var/lib/extensions/debug-tools/usr/local/bin sudo mkdir -p /var/lib/extensions/debug-tools/usr/lib/extension-release.d sudo mkdir -p /var/lib/extensions/debug-tools/opt/debug-tools sudo tee /var/lib/extensions/debug-tools/usr/lib/extension-release.d/extension-release.debug-tools >/dev/null <<'EOF' ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 EOF sudo tee /var/lib/extensions/debug-tools/usr/lib/extension-release.d/extension-release.debug-tools >/dev/null <<'EOF' ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 EOF sudo tee /var/lib/extensions/debug-tools/usr/lib/extension-release.d/extension-release.debug-tools >/dev/null <<'EOF' ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 EOF sudo tee /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext >/dev/null <<'EOF' #!/usr/bin/env bash set -euo pipefail printf 'hello from systemd-sysext\n' EOF sudo chmod 0755 /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext sudo tee /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext >/dev/null <<'EOF' #!/usr/bin/env bash set -euo pipefail printf 'hello from systemd-sysext\n' EOF sudo chmod 0755 /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext sudo tee /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext >/dev/null <<'EOF' #!/usr/bin/env bash set -euo pipefail printf 'hello from systemd-sysext\n' EOF sudo chmod 0755 /var/lib/extensions/debug-tools/usr/local/bin/hello-sysext sudo tee /var/lib/extensions/debug-tools/opt/debug-tools/README.txt >/dev/null <<'EOF' This file is provided by the debug-tools system extension. EOF sudo tee /var/lib/extensions/debug-tools/opt/debug-tools/README.txt >/dev/null <<'EOF' This file is provided by the debug-tools system extension. EOF sudo tee /var/lib/extensions/debug-tools/opt/debug-tools/README.txt >/dev/null <<'EOF' This file is provided by the debug-tools system extension. EOF sudo systemd-sysext refresh sudo systemd-sysext refresh sudo systemd-sysext refresh systemd-sysext status systemd-sysext status systemd-sysext status command -v hello-sysext hello-sysext ls -l /opt/debug-tools cat /opt/debug-tools/README.txt command -v hello-sysext hello-sysext ls -l /opt/debug-tools cat /opt/debug-tools/README.txt command -v hello-sysext hello-sysext ls -l /opt/debug-tools cat /opt/debug-tools/README.txt systemd-sysext list systemd-sysext list systemd-sysext list sudo systemd-sysext unmerge sudo systemd-sysext unmerge sudo systemd-sysext unmerge sudo systemd-sysext merge sudo systemd-sysext merge sudo systemd-sysext merge sudo systemd-sysext refresh sudo systemd-sysext refresh sudo systemd-sysext refresh sudo mkdir -p /var/lib/extensions/mytest make sudo DESTDIR=/var/lib/extensions/mytest make install sudo mkdir -p /var/lib/extensions/mytest/usr/lib/extension-release.d sudo tee /var/lib/extensions/mytest/usr/lib/extension-release.d/extension-release.mytest >/dev/null <<'EOF' ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 EOF sudo systemd-sysext refresh sudo mkdir -p /var/lib/extensions/mytest make sudo DESTDIR=/var/lib/extensions/mytest make install sudo mkdir -p /var/lib/extensions/mytest/usr/lib/extension-release.d sudo tee /var/lib/extensions/mytest/usr/lib/extension-release.d/extension-release.mytest >/dev/null <<'EOF' ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 EOF sudo systemd-sysext refresh sudo mkdir -p /var/lib/extensions/mytest make sudo DESTDIR=/var/lib/extensions/mytest make install sudo mkdir -p /var/lib/extensions/mytest/usr/lib/extension-release.d sudo tee /var/lib/extensions/mytest/usr/lib/extension-release.d/extension-release.mytest >/dev/null <<'EOF' ID=debian VERSION_ID=12 SYSEXT_SCOPE=system ARCHITECTURE=x86-64 EOF sudo systemd-sysext refresh sudo mkdir -p /etc/extensions/debug-tools sudo systemd-sysext refresh sudo mkdir -p /etc/extensions/debug-tools sudo systemd-sysext refresh sudo mkdir -p /etc/extensions/debug-tools sudo systemd-sysext refresh sudo rmdir /etc/extensions/debug-tools sudo systemd-sysext refresh sudo rmdir /etc/extensions/debug-tools sudo systemd-sysext refresh sudo rmdir /etc/extensions/debug-tools sudo systemd-sysext refresh sudo systemd-sysext refresh sudo systemd-sysext refresh sudo systemd-sysext refresh - shipping optional troubleshooting tools - testing a newer build of a low-level binary - layering in site-specific files on top of a controlled base image - keeping the base OS reproducible while still allowing operational flexibility - systemd-sysext merges only /usr and /opt - files inside /etc and /var in the extension are ignored by sysext - it is additive by design, even though overlayfs technically allows replacement behavior - it is not the right tool for shipping system services early in boot - a Linux host with systemd-sysext available - root access for installation into system extension paths - overlayfs support in the kernel - /etc/extensions/ - /run/extensions/ - /var/lib/extensions/ - ID= should match your host OS family - VERSION_ID= is the fallback compatibility gate - ARCHITECTURE= should match the host architecture when set - do not put os-release in the extension's /usr/lib, because that would shadow the host metadata - directory: debug-tools - file: extension-release.debug-tools - I want dependency management - I want normal upgrade and removal tracking - I am distributing software broadly to many mixed systems - the host is not trying to keep /usr controlled or reproducible - use sysext when you want extra files to appear in /usr or /opt - use portable services when you want to ship services in an image and manage them as services, with service-level sandboxing - systemd-sysext man page: https://manpages.debian.org/bookworm-backports/systemd/systemd-sysext.8.en.html - Portable Services introduction: https://systemd.io/PORTABLE_SERVICES/ - extension-release format reference: https://www.freedesktop.org/software/systemd/man/extension-release.html - Discoverable Partitions Specification: https://uapi-group.org/specifications/specs/discoverable_partitions_specification/

🏷️ Tags

toolsutilitiessecurity toolsreportcleanpracticalsystemdsysextlinuxoverrides

More from Tools

Tools: How to Fix Kubernetes CrashLoopBackOff (Real Commands) (2026)

Tools: How to Fix Kubernetes CrashLoopBackOff (Real Commands) (2026)

2026-04-20 0
Tools: Linux File System Hunting: I Dug So Deep I Found Secrets That Could Get This Post Taken Down (2026)

Tools: Linux File System Hunting: I Dug So Deep I Found Secrets That Could Get This Post Taken Down (2026)

2026-04-20 0
Tools: Background Tasks: The One Actor in the Codebase and the SIGTERM Bug That Only Broke on Linux

Tools: Background Tasks: The One Actor in the Codebase and the SIGTERM Bug That Only Broke on Linux

2026-04-20 0
Tools: Breaking: Photo Enlarger: How to Upscale Images Without Quality Loss

Tools: Breaking: Photo Enlarger: How to Upscale Images Without Quality Loss

2026-04-20 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views