# systemd > Inspect and debug services, query the journal, write and override units, schedule timers and apply cgroup resource limits with systemd. Canonical: https://www.wiki.jodisand.me/systemd/ Reviewed: 2026-09-24 Related: [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Is it running, and if not why | `systemctl status my-app` | | Everything that failed | `systemctl --failed` | | Start, stop, restart (needs root) | `systemctl restart my-app` | | Reload config, keep the process | `systemctl reload my-app` | | Start now and at every boot | `systemctl enable --now my-app` | | Follow logs | `journalctl -u my-app -f` | | Logs of the latest run only (v257+) | `journalctl -u my-app -I` | | Logs since boot | `journalctl -u my-app -b` | | Errors and worse, last hour | `journalctl -p err --since -1h` | | Effective unit file with drop-ins | `systemctl cat my-app` | | Machine-readable properties | `systemctl show my-app -p Result -p ExecMainStatus` | | Override safely | `systemctl edit my-app` | | After editing a unit file by hand | `systemctl daemon-reload` | | Clear failed state and start limit | `systemctl reset-failed my-app` | | What slowed boot | `systemd-analyze critical-chain` | | Dependency tree | `systemctl list-dependencies my-app` | | Run a command with limits | `systemd-run --scope -p MemoryMax=1G ./cmd` | | Timer schedule | `systemctl list-timers --all` | Verified against systemd 259. Options below that arrived recently state the version. Reference: [systemd man pages](https://www.freedesktop.org/software/systemd/man/latest/). ## A failing service `systemctl status` shows the state, the main PID, the last exit status and the last ten journal lines. The exit status and `Result=` say what happened; the journal says why. ```sh systemctl status my-app --no-pager -l journalctl -u my-app -I --no-pager # everything from the latest invocation (v257+) journalctl -u my-app -b --no-pager -n 50 # older systemd: last 50 lines this boot systemctl show my-app -p Result -p ExecMainStatus -p NRestarts systemd-analyze verify /etc/systemd/system/my-app.service # syntax, missing binaries, bad dependencies ``` ```text Result=exit-code ExecMainStatus=203 NRestarts=4 ``` | `Result=` | Meaning | | --- | --- | | `exit-code` | Main process exited non-zero; `ExecMainStatus` has the code | | `signal` | Killed by a signal. `SIGKILL` usually follows a stop timeout or an external kill | | `core-dump` | Killed by a signal and dumped core; `coredumpctl list` | | `timeout` | A start, stop or runtime step exceeded its `Timeout*Sec=` | | `oom-kill` | The kernel OOM killer ended a process in the unit (cgroup `MemoryMax=` or system-wide) | | `protocol` | The service did not follow its `Type=`: no `READY=1` for `notify`, or no PID file for `forking` | | `watchdog` | `WatchdogSec=` set and the service stopped pinging | | `start-limit-hit` | More than `StartLimitBurst=` starts in `StartLimitIntervalSec=`; run `systemctl reset-failed` after fixing | | `exec-condition` | `ExecCondition=` exited 1 to 254, so the service was skipped | | `resources` | systemd could not set something up (missing directory, namespace, credentials) | Exit statuses 200 and above come from systemd while it prepared the process, before your program ran. The full table is in [systemd.exec(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Process%20Exit%20Codes). | `status=` | Name | Usual cause | | --- | --- | --- | | 200 | `CHDIR` | `WorkingDirectory=` does not exist or is not accessible | | 203 | `EXEC` | `ExecStart=` path wrong, not executable, bad shebang, or SELinux denies execution | | 209 | `STDOUT` | `StandardOutput=` target cannot be opened | | 217 | `USER` | `User=` or `Group=` does not exist | | 226 | `NAMESPACE` | A path in `ReadWritePaths=` or similar sandbox setting does not exist | `Active: activating (start)` that never completes is usually a `Type=` mismatch: a daemon that forks declared as `Type=notify`, or a service that never sends `READY=1`. It ends in `timeout` after `TimeoutStartSec=` (default 90 seconds). ## Inspecting units ```sh systemctl list-units --type=service --state=running systemctl list-unit-files --state=enabled systemctl cat my-app # unit file plus every drop-in, in the order applied systemctl show my-app -p ExecStart -p User -p MemoryMax systemctl list-dependencies my-app --reverse # units that pull this one in systemctl is-enabled my-app; systemctl is-active my-app # scriptable: exit status 0 when true systemd-delta # overridden, extended and masked units system-wide ``` Units in `/etc/systemd/system/` override `/run/systemd/system/`, which override `/usr/lib/systemd/system/`. Drop-ins in `my-app.service.d/*.conf` change individual settings. `systemctl cat` shows the result, which is the only reliable view of what is in effect. ## Journal The journal is an indexed binary store of structured records. Every field (`_PID`, `_SYSTEMD_UNIT`, `PRIORITY` and so on) can be filtered, not only the message text. ```sh journalctl -u my-app -f # follow journalctl -u my-app --since '2 hours ago' --until '10 min ago' journalctl -u my-app -b -1 # previous boot (needs persistent storage) journalctl -p warning..err -b # priority range journalctl _PID=1234 # one process journalctl -u my-app -o json-pretty -n 1 # every field of the latest record journalctl -u my-app -g 'timeout|refused' # regex on MESSAGE; case-insensitive if all lower case journalctl -k -b # kernel messages this boot journalctl --list-boots journalctl --disk-usage journalctl --vacuum-time=7d # deletes archived journal files older than 7 days ``` Storage is set by `Storage=` in `journald.conf`. With `auto`, logs persist only if `/var/log/journal` exists; otherwise they live in `/run/log/journal` and vanish at reboot. systemd 259 changed the default from `auto` to `persistent`. On older systems with no `/var/log/journal`, `journalctl -b -1` returns nothing. ## Writing a unit Local units go in `/etc/systemd/system/`. Run `systemctl daemon-reload` after creating or editing one by hand. ```ini [Unit] Description=Ingest worker Documentation=https://docs.example.com/ingest After=network-online.target postgresql.service Wants=network-online.target StartLimitIntervalSec=300 StartLimitBurst=5 [Service] Type=notify User=ingest Group=ingest WorkingDirectory=/opt/ingest EnvironmentFile=-/etc/ingest/env ExecStart=/opt/ingest/bin/worker --config /etc/ingest/config.yaml ExecReload=kill -HUP $MAINPID Restart=on-failure RestartSec=5 TimeoutStopSec=30 # sandboxing: each line removes access the service does not need NoNewPrivileges=yes PrivateTmp=yes ProtectSystem=strict ProtectHome=yes StateDirectory=ingest ProtectKernelTunables=yes ProtectControlGroups=yes RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX SystemCallFilter=@system-service CapabilityBoundingSet= [Install] WantedBy=multi-user.target ``` `StateDirectory=ingest` creates `/var/lib/ingest` owned by the service user and makes it writable under `ProtectSystem=strict`. Use `ReadWritePaths=` for paths systemd should not create. | Setting | Effect | | --- | --- | | `Type=simple` | Default. Started as soon as the process is forked, before `execve`, so a missing binary still counts as started | | `Type=exec` | Started once `execve` succeeds, so a bad `ExecStart=` fails the start job. Recommended over `simple` (v240+) | | `Type=forking` | Traditional daemon: started when the parent exits. Set `PIDFile=` so systemd tracks the right main process | | `Type=notify` | Started when the service sends `READY=1` through `sd_notify()`. Accurate ordering for dependants | | `Type=notify-reload` | Like `notify`, and `systemctl reload` sends `SIGHUP` and waits for the service to report completion (v253+) | | `Type=oneshot` | Started when the process exits. Add `RemainAfterExit=yes` to keep it "active" afterwards | | `Restart=on-failure` | Restart after non-zero exit, unclean signal, timeout, watchdog or OOM kill; not after a clean exit or `systemctl stop` | | `StartLimitBurst=` / `StartLimitIntervalSec=` | In `[Unit]`. Too many starts in the window leaves the unit `failed` with `start-limit-hit` | | `After=` | Ordering only: start after X if both are being started. Pulls nothing in | | `Wants=` / `Requires=` | Pull X in. `Requires=` also stops this unit if X is stopped. Combine with `After=` for ordering | | `EnvironmentFile=-/path` | Leading `-` makes a missing file non-fatal | `After=network-online.target` needs `Wants=network-online.target` too, or nothing pulls the target in and the unit may start before the network is configured. Check the sandbox you achieved: ```sh systemd-analyze security my-app # exposure score 0 to 10 and the setting behind each point ``` ## Overriding a packaged unit Do not edit files under `/usr/lib/systemd/system/`: the next package update overwrites them. ```sh systemctl edit my-app # writes /etc/systemd/system/my-app.service.d/override.conf and reloads systemctl edit --full my-app # copies the whole unit to /etc/systemd/system/ for larger changes systemctl revert my-app # deletes local overrides and drop-ins, back to the vendor unit ``` ```ini # /etc/systemd/system/my-app.service.d/override.conf [Service] ExecStart= ExecStart=/opt/ingest/bin/worker --config /etc/ingest/other.yaml MemoryMax=2G ``` List-valued settings such as `ExecStart=`, `Environment=` and `After=` accumulate across drop-ins. An empty assignment clears the list. Without the empty `ExecStart=` line the unit has two `ExecStart=` commands and fails to load with "Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services". ```sh systemctl restart my-app # systemctl edit already ran daemon-reload; the process needs a restart ``` ## Timers A timer unit activates a service unit of the same name (or the one named in `Unit=`). Compared with cron, timers log to the journal, record failures in unit state, catch up after downtime with `Persistent=true`, and run with the service's sandboxing and resource limits. ```ini # /etc/systemd/system/backup.timer [Unit] Description=Nightly backup [Timer] OnCalendar=*-*-* 02:30:00 RandomizedDelaySec=300 Persistent=true [Install] WantedBy=timers.target ``` ```sh systemctl enable --now backup.timer # enable the timer, not the service systemctl list-timers --all # next and last run for each systemd-analyze calendar --iterations=3 'Mon *-*-* 06:00:00' # validate and show the next elapses systemctl start backup.service # run it now, independent of the timer journalctl -u backup.service --since today ``` `OnCalendar=daily` means `*-*-* 00:00:00`, when many other jobs also run. `RandomizedDelaySec=` spreads the start. For a one-off delayed job without writing files: `systemd-run --on-active=30m /usr/local/bin/cleanup`. ## Resource control Every unit runs in its own cgroup, so the kernel enforces limits whether or not the process cooperates. ```ini [Service] MemoryMax=2G # hard limit: the OOM killer acts inside the cgroup above this MemoryHigh=1.5G # throttle and reclaim above this, before MemoryMax CPUQuota=150% # at most 1.5 CPUs of time CPUWeight=50 # relative share under contention (default 100) IOWeight=50 TasksMax=512 # processes plus threads ``` ```sh systemd-cgtop # live CPU, memory, I/O per cgroup systemctl show my-app -p MemoryCurrent -p MemoryPeak -p CPUUsageNSec systemctl set-property my-app MemoryMax=1G # apply now and persist as a drop-in systemd-run --scope -p MemoryMax=1G -p CPUQuota=50% ./heavy-job # limit an ad-hoc command ``` For reading pressure and throttling counters from these cgroups, see [Linux performance](https://www.wiki.jodisand.me/linux-performance/#cpu). ## Boot problems ```sh systemd-analyze # time in firmware, loader, kernel, initrd, userspace systemd-analyze blame # units by start time (parallel starts overlap, so not additive) systemd-analyze critical-chain # the chain that delayed the default target systemctl --failed journalctl -b -p err systemctl list-jobs # jobs still queued when boot hangs ``` A unit stuck in `activating` holds up every unit ordered `After=` it. `systemctl list-jobs` names the waiting jobs, which is faster than reading the whole journal. ## Socket activation systemd opens the listening socket itself and starts the service on the first connection. The service receives the socket as file descriptor 3 onwards, with `$LISTEN_FDS` set to the count and `$LISTEN_PID` to its own PID, and calls `sd_listen_fds()` (or the language equivalent) instead of `bind()`. Connections that arrive while the service is stopped or restarting wait in the kernel backlog instead of being refused, and a daemon that is rarely used stays stopped until something connects. ```ini # /etc/systemd/system/my-app.socket [Socket] ListenStream=8080 # TCP on every address; 127.0.0.1:8080 or [::1]:8080 to restrict ListenStream=/run/my-app/api.sock # AF_UNIX path; both sockets are passed to the same service SocketMode=0660 SocketUser=my-app NoDelay=true # TCP_NODELAY on accepted connections FreeBind=true # bind an address that is not configured yet at boot [Install] WantedBy=sockets.target ``` ```ini # /etc/systemd/system/my-app.service: no [Install] section, the socket starts it [Service] Type=exec User=my-app ExecStart=/opt/my-app/bin/server --systemd-socket ``` With the default `Accept=no` one service instance gets every listening socket. `Accept=yes` spawns `my-app@.service` per connection and passes only the accepted socket, with `$REMOTE_ADDR` and `$REMOTE_PORT` in the environment; `MaxConnections=` (default 64) and `MaxConnectionsPerSource=` bound it. `Accept=yes` costs a fork per connection and suits inetd-style programs, not servers. `Service=` names a service other than the one matching the socket name and is only allowed with `Accept=no`. ```sh systemctl enable --now my-app.socket # enable the socket, not the service systemctl list-sockets # listening address, socket unit and the unit it activates systemctl status my-app.socket # "Triggers:" shows the service; "Listen:" the addresses ss -ltnp 'sport = :8080' # the owner is systemd until the service starts systemctl stop my-app.socket my-app # stop both; stopping only the service lets the next connection restart it ``` Test a socket-activated program without a unit. `systemd-socket-activate` listens, then execs the command with the same `$LISTEN_FDS` protocol when a connection arrives: ```sh systemd-socket-activate -l 8080 /opt/my-app/bin/server --systemd-socket # -l address or port; repeatable systemd-socket-activate -l 8080 -a /usr/bin/my-handler # -a: one instance per connection, like Accept=yes systemd-socket-activate -l 8080 --inetd /usr/bin/my-handler # connection on stdin/stdout, inetd style ``` `TriggerLimitIntervalSec=` and `TriggerLimitBurst=` (defaults 2 s and 20 activations for `Accept=no`, 200 for `Accept=yes`) put the socket into a failed state when a client floods it; `systemctl reset-failed my-app.socket` clears it. ## Sandboxing Each directive under `[Service]` removes access through namespaces, seccomp filters or capability drops before `ExecStart=` runs. They cost nothing at runtime and turn a compromised service into a process that can read its own state directory and little else. The unit in [Writing a unit](#writing-a-unit) uses the common set; the table gives what each one does and what it breaks. | Directive | Effect | Breaks when | | --- | --- | --- | | `ProtectSystem=strict` | Whole file system read-only except `/dev`, `/proc`, `/sys` | The service writes anywhere not in `ReadWritePaths=` or a `*Directory=` setting | | `ProtectHome=yes` | `/home`, `/root`, `/run/user` empty and inaccessible; `read-only` or `tmpfs` are the alternatives | Config or data lives under a home directory | | `PrivateTmp=yes` | Private `/tmp` and `/var/tmp`, removed on stop | Another unit expects to find its files in `/tmp` | | `PrivateDevices=yes` | Only pseudo devices in `/dev`, no `/dev/sda`, `/dev/mem` | Hardware access, some GPU and audio work | | `PrivateNetwork=yes` | New network namespace with only `lo` | Any network use, including AF_UNIX to the host and D-Bus | | `NoNewPrivileges=yes` | `execve()` cannot gain privileges: setuid binaries, file capabilities | The service runs `sudo`, `ping` with setuid, or similar | | `CapabilityBoundingSet=` | Empty value drops every capability; list names to keep, prefix `~` to drop the named ones | Binding ports below 1024 needs `CAP_NET_BIND_SERVICE`; use `AmbientCapabilities=` for a non-root `User=` | | `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX` | `socket()` fails for other families | Netlink (`AF_NETLINK`) for interface queries, `AF_PACKET` for raw capture | | `SystemCallFilter=@system-service` | Allow-list of system calls a normal service needs; the process is killed with `SIGSYS` on any other | Old binaries, JIT runtimes, anything mounting or using `ptrace`; `SystemCallErrorNumber=EPERM` returns an error instead of killing | | `SystemCallArchitectures=native` | Refuse 32-bit calls on a 64-bit kernel | Nothing in practice | | `RestrictNamespaces=yes`, `LockPersonality=yes`, `RestrictRealtime=yes`, `RestrictSUIDSGID=yes`, `MemoryDenyWriteExecute=yes` | Block namespace creation, `personality()`, realtime scheduling, creating setuid files, writable-and-executable memory | JIT compilers need write-execute memory; container tooling needs namespaces | | `ProtectKernelTunables=yes`, `ProtectKernelModules=yes`, `ProtectKernelLogs=yes`, `ProtectControlGroups=yes`, `ProtectClock=yes`, `ProtectHostname=yes` | `/proc/sys` and `/sys` read-only, no module loading, no `/dev/kmsg`, cgroup tree read-only, no clock changes, own UTS namespace | Monitoring agents that read `/dev/kmsg` or write sysctls | | `ProtectProc=invisible` | Other users' processes hidden from `/proc` | Process monitors | | `DynamicUser=yes` | UID and GID allocated at start and released at stop; implies `ProtectSystem=strict`, `ProtectHome=read-only`, `PrivateTmp=`, `NoNewPrivileges=`, `RestrictSUIDSGID=` and `RemoveIPC=` | Files the service must keep must live in `StateDirectory=`, `CacheDirectory=` or `LogsDirectory=`, which systemd chowns to the current UID | | `IPAddressDeny=any` with `IPAddressAllow=` | cgroup BPF filter on every socket; in `systemd.resource-control(5)` | Any peer not in the allow list | `RuntimeDirectory=`, `StateDirectory=`, `CacheDirectory=`, `LogsDirectory=` and `ConfigurationDirectory=` create `/run/NAME`, `/var/lib/NAME`, `/var/cache/NAME`, `/var/log/NAME` and `/etc/NAME` owned by the service user and export `$RUNTIME_DIRECTORY` and friends. They are the right way to give a sandboxed service somewhere to write; `ReadWritePaths=` is for paths that already exist and belong to something else. Secrets go in through credentials rather than `Environment=`, which any process in the unit can read from `/proc/PID/environ`: ```ini [Service] LoadCredential=db-password:/etc/my-app/db-password # readable only by root on disk ExecStart=/opt/my-app/bin/server --password-file ${CREDENTIALS_DIRECTORY}/db-password ``` The service sees the file under `$CREDENTIALS_DIRECTORY`, on a tmpfs owned by its user, and nothing else on the system can open it. `LoadCredentialEncrypted=` reads files produced by `systemd-creds encrypt`, which binds them to the TPM or the host key. Score the result and find the setting behind each point: ```sh systemd-analyze security my-app # per-directive table; 0 is fully sandboxed, 10 is unrestricted systemd-analyze security my-app --json=short | jq '[.[] | select(.set == false)] | length' # settings not applied systemd-analyze security --threshold=5 my-app # exit non-zero when exposure is above 5, for CI systemd-analyze security --offline=yes /etc/systemd/system/my-app.service # score a file without loading it ``` When a hardened unit fails, the exit status names the layer: `226/NAMESPACE` for a path in `ReadWritePaths=` or `BindPaths=` that does not exist, `228/SECCOMP` when the filter could not be applied, `238/STATE_DIRECTORY` when the directory could not be created, and `243/CREDENTIALS` for a missing credential file. A service that starts and then dies with `code=killed, status=31/SYS` made a system call outside its `SystemCallFilter=`. Reproduce a sandbox without editing the unit: ```sh systemd-run --pty --wait --uid=my-app -p ProtectSystem=strict -p PrivateTmp=yes \ -p SystemCallFilter=@system-service -p StateDirectory=my-app /opt/my-app/bin/server --check ``` Drop one `-p` at a time until it works; that is the directive to relax or the path to add. ## User units Every logged-in user gets a `systemd --user` manager that runs units from the user's own directories, as that user, with no root involved. It has its own `default.target`, timers, sockets and journal namespace, and the same `systemctl` and `journalctl` commands work with `--user`. | Location | Contents | | --- | --- | | `~/.config/systemd/user/` | The user's own units; `systemctl --user enable` links into `default.target.wants/` here | | `/etc/systemd/user/` | Units the administrator provides to every user | | `/usr/lib/systemd/user/` | Package-provided user units (`pipewire.service`, `dbus.service`) | | `$XDG_RUNTIME_DIR/systemd/user/` | Runtime units, gone at logout | ```sh mkdir -p ~/.config/systemd/user systemctl --user daemon-reload systemctl --user enable --now sync.timer systemctl --user status sync.service journalctl --user -u sync.service -f systemctl --user show-environment # the environment block user services inherit systemctl --user import-environment PATH # copy a variable from this shell into the manager systemd-run --user --unit=build -p MemoryMax=4G make -j8 # transient user unit with a cgroup limit ``` The user manager starts at first login and stops after the last session closes, taking every user service with it. Enable lingering for a user whose services must survive logout and start at boot: ```sh sudo loginctl enable-linger alice # creates /var/lib/systemd/linger/alice; user@1000.service now starts at boot loginctl show-user alice -p Linger ``` `systemctl --user` needs `$XDG_RUNTIME_DIR` (`/run/user/UID`) and `$DBUS_SESSION_BUS_ADDRESS`, which `pam_systemd` sets at login. From `sudo` or a cron job they are missing, so the command fails with `Failed to connect to bus`. Address the user's manager explicitly instead: ```sh sudo systemctl --user -M alice@ status sync.timer # -M USER@ targets that user's manager through the system bus sudo journalctl _UID="$(id -u alice)" --user-unit=sync.service ``` User units cannot use `User=`, `CapabilityBoundingSet=` or other root-only directives, bind to ports below 1024 without a sysctl, or start before the user manager exists; anything that must run as a system service belongs in `/etc/systemd/system/` with `User=` set. ## journald configuration `systemd-journald` reads `/etc/systemd/journald.conf` and then `/etc/systemd/journald.conf.d/*.conf` in lexical order; a drop-in keeps the package-owned file untouched. The defaults cap the persistent journal at 10% of the file system or 4 GiB, whichever is smaller, keep 15% free, and rate limit each service to 10,000 messages in 30 seconds. ```ini # /etc/systemd/journald.conf.d/10-retention.conf [Journal] Storage=persistent # /var/log/journal, created if missing; the default from v259 SystemMaxUse=2G # cap for /var/log/journal; the smaller of this and SystemKeepFree= wins SystemKeepFree=5G # leave this much free for other users of the file system SystemMaxFileSize=128M # rotation granularity; vacuuming deletes whole archived files MaxRetentionSec=1month # delete entries older than this; default 0 disables age-based deletion MaxFileSec=1week # rotate at least weekly so old data is deleted in small steps RateLimitIntervalSec=30s RateLimitBurst=10000 # per service; 0 in either disables rate limiting ForwardToSyslog=no # yes when rsyslog or a collector reads /run/systemd/journal/syslog MaxLevelStore=debug # drop records above this priority before storing Compress=yes ``` ```sh systemctl restart systemd-journald # apply; records in flight are kept in /run and re-read journalctl --disk-usage # active plus archived files journalctl --rotate --vacuum-size=500M # archive the active files first, then delete the oldest archives journalctl --vacuum-time=14d # by age journalctl --vacuum-files=10 # keep at most this many archived files journalctl --verify # checksum every journal file; reports corrupt ones journalctl --header # file headers; useful for out-of-order timestamps after a clock jump journalctl -N # every field name in use journalctl -F _SYSTEMD_UNIT # every value of one field, for building filters journalctl --flush # /run/log/journal into /var/log/journal once persistent storage exists ``` Vacuuming only removes archived files, so `--disk-usage` drops less than expected until the active file rotates; `--rotate` first fixes that. A single noisy unit is better limited in its own unit than globally: ```ini [Service] LogRateLimitIntervalSec=10s LogRateLimitBurst=200 LogFilterPatterns=~^DEBUG # drop matching lines before they reach the journal (v253+) LogExtraFields=TEAM=platform # add a field to every record for filtering ``` The message `Suppressed 3120 messages from my-app.service` in the journal means the rate limit fired and those records are gone. ## Oneliners ```sh # Result and exit status of every failed service systemctl --failed --no-legend --plain | awk '{print $1}' | xargs -r -I{} systemctl show {} -p Id -p Result -p ExecMainStatus # Services by memory use systemd-cgtop -m -b -n 1 --depth=2 | head -15 # Which unit owns a process systemctl status "$(pgrep -o -f worker)" # Restart count since the unit was last reset systemctl show my-app -p NRestarts --value # Processes in a unit's cgroup systemd-cgls -u my-app.service # Run a command as the service user in a transient unit and wait for it systemd-run --uid=ingest --same-dir --wait --pty /opt/ingest/bin/worker --check # Prevent a unit from being started by anything (symlinks it to /dev/null) systemctl mask --now my-app # Journal fields available for a unit journalctl -u my-app -o verbose -n 1 # Least-sandboxed services first systemd-analyze security --no-pager | sort -k2 -nr | head # Units that failed since boot, with the last three journal lines of each systemctl --failed --no-legend --plain | awk '{print $1}' | while read -r u; do echo "== $u"; journalctl -u "$u" -b -n 3 --no-pager -o cat; done # Every service that restarted at least once this boot systemctl list-units --type=service --no-legend --plain | awk '{print $1}' | xargs -r -I{} sh -c 'n=$(systemctl show {} -p NRestarts --value); [ "$n" -gt 0 ] && echo "$n {}"' | sort -rn # Effective value of one setting after all drop-ins systemctl show my-app -p ExecStart --value # Where a unit's files and drop-ins come from systemctl show my-app -p FragmentPath -p DropInPaths # Wait until a unit is active before continuing (exit 1 after 60 s) timeout 60s bash -c 'until systemctl is-active --quiet my-app; do sleep 1; done' # Time since the unit last became active systemctl show my-app -p ActiveEnterTimestamp --value # Sockets systemd is listening on, and the units they trigger systemctl list-sockets --all # Timers whose last run failed systemctl list-timers --all --no-legend --plain | awk '{print $NF}' | xargs -r -I{} sh -c 'systemctl is-failed --quiet {} && echo {}' # Next three elapses of a calendar expression systemd-analyze calendar --iterations=3 'Mon..Fri *-*-* 09:00' # Run a job once, 20 minutes from now, as a transient timer systemd-run --on-active=20m --unit=cleanup-once /usr/local/bin/cleanup # Journal lines of a unit as one JSON object per line, for jq journalctl -u my-app -b -o json | jq -r 'select(.PRIORITY|tonumber <= 3) | .MESSAGE' # Units logging most this boot journalctl -b -o json | jq -r '._SYSTEMD_UNIT // empty' | sort | uniq -c | sort -rn | head # Everything a process wrote, including its children, by cgroup journalctl -b _SYSTEMD_CGROUP=/system.slice/my-app.service # Messages the journal suppressed through rate limiting journalctl -b -g 'Suppressed [0-9]+ messages' # Kernel OOM kills this boot journalctl -k -b -g 'Out of memory|oom-kill' # Follow the journal of a user's service from root sudo journalctl -f _UID="$(id -u alice)" --user-unit=sync.service # Unit files that differ from the vendor version systemd-delta --type=extended,overridden # Verify every local unit file without starting anything systemd-analyze verify /etc/systemd/system/*.service # Run the security scan and fail above a threshold, for CI systemd-analyze security --threshold=6 my-app --no-pager # Exposure score for every service, as JSON for u in $(systemctl list-units --type=service --state=running --no-legend --plain | awk '{print $1}'); do printf '%s ' "$u"; systemd-analyze security "$u" --json=short 2>/dev/null | jq -r 'map(select(.set==false)) | length'; done # Environment the service will actually see (not your shell's) systemctl show my-app -p Environment -p EnvironmentFiles # Memory pressure stall time for a unit's cgroup cat "/sys/fs/cgroup/system.slice/my-app.service/memory.pressure" # Set a hard memory limit until the next boot only systemctl set-property --runtime my-app MemoryMax=512M # Stop a unit and everything that depends on it systemctl stop my-app --with-dependencies # Reboot into the boot menu or a rescue shell systemctl reboot --boot-loader-menu=10; systemctl rescue ``` ## Scripts Report every failed unit on a list of hosts, with its result, exit status and last journal lines, so one run covers a fleet. ```sh #!/usr/bin/env bash # usage: failed-units.sh host1 host2 ... (ssh as a user who can read the journal) set -euo pipefail for host in "$@"; do printf '== %s\n' "$host" ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" bash -s <<'EOF' || printf 'unreachable\n' set -euo pipefail units=$(systemctl --failed --no-legend --plain | awk '{print $1}') [[ -n $units ]] || { echo ok; exit 0; } for u in $units; do systemctl show "$u" -p Id -p Result -p ExecMainStatus -p NRestarts --value | paste -sd ' ' journalctl -u "$u" -b -n 5 --no-pager -o short-iso | sed 's/^/ /' done EOF done ``` Generate a matching service and timer pair for a periodic job, with sandboxing already applied, and enable the timer. ```sh #!/usr/bin/env bash # usage: mktimer.sh NAME 'OnCalendar expression' /path/to/command [args...] set -euo pipefail name=$1; cal=$2; shift 2 systemd-analyze calendar "$cal" >/dev/null # fails on a bad expression before anything is written unit=/etc/systemd/system/$name cat > "$unit.service" < "$unit.timer" < t) }'; then printf '%-40s %s\n' "$unit" "$score" rc=1 fi done < <(systemd-analyze security --no-pager 2>/dev/null | awk 'NR > 1 && $2 ~ /^[0-9.]+$/ {print $1, $2}' | sort -k2 -nr) exit "$rc" ``` ## Troubleshooting | Symptom | Cause | Check or fix | | --- | --- | --- | | `Unit my-app.service not found` after creating it | Manager has not re-read unit files | `systemctl daemon-reload` | | `Warning: The unit file ... changed on disk` | Edited without reloading | `systemctl daemon-reload` | | `start request repeated too quickly` | `start-limit-hit` | Fix the cause, then `systemctl reset-failed my-app` | | `status=203/EXEC` | Wrong path, missing execute bit, CRLF shebang, SELinux label | `ls -lZ` the binary; `ausearch -m avc -ts recent` | | Starts by hand, fails as a service | Different user, `PATH`, working directory or sandbox | `systemd-run --uid=... --pty` to reproduce; relax one sandbox setting at a time | | Changes to a drop-in have no effect | Wrong section header, or a later drop-in overrides it | `systemctl cat my-app`, `systemd-delta` | | Service killed after 90 s on stop | Ignores `SIGTERM` | `TimeoutStopSec=`, `KillSignal=`, fix signal handling | | Enabled but not started at boot | `WantedBy=` target not reached, or condition failed | `systemctl list-dependencies multi-user.target`, `systemctl status` shows `Condition` lines | | Timer never fires | Timer not enabled, or the service is enabled instead | `systemctl list-timers --all`, `systemctl enable --now x.timer` | | No logs from a previous boot | Volatile journal storage | Create `/var/log/journal` or set `Storage=persistent`, then `journalctl --flush` |