Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
Running apps as systemd services: a production unit file guide How we run Node, Python and Go apps under systemd: a dedicated user, restart limits, secrets, hardening, resource limits and logs that do not fill the disk.
We regularly inherit servers where the main application runs inside a tmux session, or under nohup started by someone who has since left. It works until the first reboot. Then the app is down, nobody knows the exact command that started it, and the environment variables it needed lived only in that person's shell history.
The short answer: to run an app as a systemd service, write a unit file in /etc/systemd/system, run the process as a dedicated unprivileged user, set Type=exec and Restart=on-failure, load configuration from a root-owned file, add sandboxing options, and let the journal collect stdout. Enable it so it starts on boot. The detail behind each choice follows.
A unit file we would actually ship Here is a complete unit for a FastAPI application served by uvicorn on port 8000 behind NGINX. The same shape works for a Node, Go or Java service; only ExecStart changes.
# /etc/systemd/system/orders-api.service
[Unit]
Description=Orders API
Wants=network-online.target
After=network-online.target postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Type=exec
User=orders
Group=orders
WorkingDirectory=/opt/orders-api/current
EnvironmentFile=/etc/orders-api/env
LoadCredential=db_password:/etc/orders-api/db_password
ExecStart=/opt/orders-api/current/.venv/bin/uvicorn app.main:app --host 127.0.0.1 --port 8000 --workers 4
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
# Writable paths, created and owned for us
StateDirectory=orders-api
RuntimeDirectory=orders-api
# Resource limits (cgroup v2)
MemoryHigh=768M
MemoryMax=1G
TasksMax=256
LimitNOFILE=65536
# Sandboxing
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
CapabilityBoundingSet=
[Install]
WantedBy=multi-user.targetBefore enabling it, create the user and check the file. systemd-analyze verify catches typos in directive names, which systemd otherwise only logs as warnings and ignores.
sudo useradd --system --user-group --home-dir /opt/orders-api --shell /usr/sbin/nologin orders
sudo systemd-analyze verify /etc/systemd/system/orders-api.service
sudo systemctl daemon-reload
sudo systemctl enable --now orders-api
systemctl status orders-apiThe choices that matter Most of the lines above have a reason we have learned the hard way. These are the ones people most often get wrong.
Type=exec rather than the default Type=simple. With simple, systemctl start reports success even if the binary path is wrong. With exec, it waits until the program has actually been executed and fails the start if it could not be. If your app supports sd_notify readiness, Type=notify is better still. Restart=on-failure, not Restart=always. It restarts after crashes, non-zero exits and timeouts, but a clean exit or a deliberate systemctl stop stays stopped. StartLimitIntervalSec and StartLimitBurst belong in [Unit], not [Service]. Here, a sixth start within 300 seconds is refused and the unit stays failed instead of restarting forever. That failed state is what your monitoring should alert on. Clear it with systemctl reset-failed orders-api once you have fixed the cause. Wants= plus After=network-online.target if the app connects to anything at startup. After= alone only orders units; it does not pull the target in. Bind to 127.0.0.1 when a reverse proxy sits in front. The app should not be reachable from outside on its own port. Absolute paths in ExecStart, pointing at the virtualenv or node binary you deployed, not whatever happens to be first on PATH. After= on another local unit, such as postgresql.service, only matters if both start on the same machine during boot. It does not wait for the database to accept connections, so the application should still retry its first connection instead of crashing.
Configuration and secrets EnvironmentFile is read by systemd itself, as root, before it drops privileges. That means the file can be owned by root with mode 600 and the application user still gets the values. Plain Environment= lines in the unit are fine for non-sensitive settings, but unit files are world-readable and systemctl show prints them, so passwords do not belong there.
For real secrets we prefer LoadCredential. systemd copies the file into a private, read-only directory that only this service can see, and exposes its path as $CREDENTIALS_DIRECTORY. The secret never appears in the process environment, so it does not leak through /proc, crash reports or a debug endpoint that dumps environment variables.
# Python: read the credential at startup
import os, pathlib
cred_dir = pathlib.Path(os.environ["CREDENTIALS_DIRECTORY"])
db_password = (cred_dir / "db_password").read_text().strip()If you want the file encrypted at rest on the server, systemd-creds encrypt can produce a blob tied to the host (and the TPM, where one exists), which you then load with LoadCredentialEncrypted=. It is worth it on machines where disk images or backups travel further than you would like.
Hardening without breaking the app The sandboxing block is where most of the security value is, and it costs almost nothing at runtime. ProtectSystem=strict makes the whole filesystem read-only for the service, apart from /dev, /proc and /sys. StateDirectory=orders-api creates /var/lib/orders-api, owned by the service user, and makes it writable; RuntimeDirectory does the same for /run/orders-api and removes it on stop. If the app needs another writable path, such as an upload folder, add it explicitly with ReadWritePaths=.
An empty CapabilityBoundingSet= drops every Linux capability. If you need to bind a port below 1024 without root, use AmbientCapabilities=CAP_NET_BIND_SERVICE and list the same capability in the bounding set. In practice we rarely need it, because NGINX or Traefik owns ports 80 and 443.
systemd will score a unit for you. Run it before and after adding options and look at the items still marked as exposed:
systemd-analyze security orders-api.serviceThe usual breakages are predictable. ProtectHome=yes fails if the app lives under /home, which is one reason we deploy to /opt. PrivateTmp=yes surprises people who pass files through /tmp to another process. Apps that use a JIT, such as Node and Java, will crash if you add MemoryDenyWriteExecute=yes, so we leave that one out for them. Add options in small groups and restart between them.
Logs, memory and stopping cleanly Let the application log to stdout and stderr. journald captures it with timestamps and the unit name, and you query it per service instead of hunting through files in /var/log. Python block-buffers print() output when stdout is not a terminal, so add Environment=PYTHONUNBUFFERED=1 if lines arrive late or in bursts.
journalctl -u orders-api -f
journalctl -u orders-api --since "1 hour ago" -p warning
journalctl --disk-usage
# /etc/systemd/journald.conf.d/size.conf
[Journal]
SystemMaxUse=2GOn a persistent journal the default cap is 10% of the filesystem, up to 4G. On a small VPS we set SystemMaxUse explicitly so it is a decision rather than an accident, then restart systemd-journald.
For memory, MemoryHigh throttles the service and pushes it to reclaim memory before MemoryMax, where the kernel kills it. That gives you slow responses and a warning sign before an outright crash. TasksMax stops a thread or process leak from taking the whole server down. These limits rely on cgroup v2, which is the default on current Debian, Ubuntu, RHEL and their derivatives.
On stop, systemd sends SIGTERM and waits for TimeoutStopSec before sending SIGKILL. Your app should handle SIGTERM by finishing in-flight requests and closing database connections. If it needs longer than 30 seconds to drain, raise the timeout rather than letting it be killed mid-write.
Changing and deploying Never edit a unit file shipped by a package under /usr/lib/systemd/system; the next upgrade overwrites it. Use a drop-in instead. systemctl edit opens an override file in /etc/systemd/system/<unit>.d/ and reloads systemd when you save.
sudo systemctl edit nginx.service
# add, for example:
# [Service]
# LimitNOFILE=65536
systemctl cat nginx.service # shows the unit plus every drop-inFor our own services, we keep the unit file in the application repository and install it during deployment. Each release goes into its own directory under /opt/orders-api/releases, and current is a symlink. A deploy installs dependencies into the new release, switches the symlink, and runs systemctl restart. Rolling back is pointing the symlink at the previous release and restarting again. Run systemctl daemon-reload whenever the unit file itself changes, or systemd keeps using the old definition.
If the deploy user is not root, give it a narrow sudoers rule rather than general sudo. sudo matches the arguments exactly, so the deploy script must run precisely this command.
# /etc/sudoers.d/orders-api (edit with: sudo visudo -f /etc/sudoers.d/orders-api)
deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart orders-apiA checklist before you call it done The service runs as its own system user with a nologin shell, from /opt, not /home or /root. Type=exec or Type=notify, Restart=on-failure, and a start limit that ends in a failed state you alert on. Secrets come from LoadCredential or a root-only EnvironmentFile, never from the unit file. ProtectSystem=strict and friends are on, writable paths are explicit, and systemd-analyze security has been read. MemoryMax and TasksMax are set, and the journal has a size cap. The app handles SIGTERM, and a reboot has been tested: everything comes back without anyone logging in. That last check is the one that catches the tmux-session servers. Reboot a staging copy on purpose, at a time of your choosing, so you are not doing it for the first time during an outage.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789