The toolkit is complete and scheduled, but cron is starting to fall short. It cannot wait for the network to be ready before launching the watchdog. It does not recover the report's run if the server was powered off at 06:30. It does not limit the memory of a job that goes wild, nor isolate its access to the filesystem. And its logging is scattered across four logs that have to be consulted one by one. systemd solves all four things, in exchange for more files and more syntax. This lesson teaches exactly as much systemd as a Bash script needs: how a service is declared, how it is scheduled with a timer, and when migrating from cron is worth it and when it is not.

1. What systemd is, units and where they live
2. Anatomy of a .service, and oneshot versus simple
3. Anatomy of a .timer
4. OnCalendar and how to validate it
5. Persistent=true: what cron cannot do
6. Cron versus timers: the honest table
7. Day-to-day commands
8. Hardening a unit
9. Long-running services written in Bash
10. User units and loginctl enable-linger
11. Migrating a cron entry step by step
12. Application: the toolkit on systemd

systemd is process number 1 on almost every current distribution: the first one the kernel starts and the one that brings up everything else. Its job is to manage units — services, mounts, sockets, timers — and the dependencies between them: what must be ready before what. A Bash script runs into it by three routes: timers as a modern alternative to cron, services when something must run continuously and restart if it dies, and querying — you have already used systemctl is-active in 06-03 and journalctl -u cron in 07-01 without calling it systemd. You do not need to master it: with the fifteen directives in this lesson you cover 95% of what an operations script needs. A unit is a text file in INI format. The types we care about are .service (a process that runs), .timer (when another unit is activated) and .target (a synchronization point, like network-online.target). And three locations, in increasing order of precedence:

The rule is the one from 05-06: what the system ships is not touched, yours goes in /etc. If you need to change a single directive of a system unit, systemctl edit name.service creates a drop-in at …/name.service.d/override.conf that layers on top without replacing the original.

[Unit] holds metadata and dependencies: Description is what you will see in systemctl status, and After=network-online.target means "do not start before the network is ready", with Wants= to activate it if it was not — this is exactly what cron cannot do with @reboot (07-01). [Service] holds how it is run. Type=oneshot for a process that does its work and finishes (the next block); User/Group so as not to run as root if you do not have to; WorkingDirectory sets the working directory, although you will use absolute paths anyway; EnvironmentFile=-/path loads KEY=value pairs as the environment and the leading dash keeps it from failing if the file does not exist; Environment= defines a single variable and is repeated as many times as needed; TimeoutStartSec is the timeout from 07-02 built in; and StandardOutput/StandardError decide where the output goes, with journal as today's default. Two warnings about ExecStart. The path must be absolute, always: systemd has no PATH worth the name and will fail with "No such file or directory" even if the script is in yours. And there is no shell: ExecStart=/bin/my.sh > /tmp/output.log redirects nothing, because the > is passed to the script as a literal argument; if you need shell constructs, invoke them explicitly with bash -c '…'. In practice you do not need to, because StandardOutput=journal already captures the output. And the [Install] section is missing, which says what happens on enable: in a service triggered by a timer you do not put one, because we do not want it to start at boot but when the timer says so.

This choice confuses everybody, and getting it wrong produces odd symptoms:

Your operations jobs are all oneshot: they do their work and exit. A Type=simple for them would make systemctl status show the unit as failed every time it finishes correctly; conversely, a long-running service declared oneshot would leave systemd waiting forever for it to "start". RemainAfterExit=yes is a complement to oneshot that leaves the unit as active after finishing, for jobs that establish a state (mounting something, applying a configuration), not for periodic ones.

A timer activates another unit at a given moment, and the relationship is by name: veloz-report.timer triggers veloz-report.service without declaring it (it can be changed with Unit=).

The [Timer] directives come in two families. The clock one is OnCalendar=, which fires at specific moments like cron. The relative ones are OnBootSec= (that much time after the system boots), OnStartupSec= (after systemd starts), OnUnitActiveSec= (after the last run) and OnUnitInactiveSec= (after it finished). To those add Persistent=true (section 5), RandomizedDelaySec= and AccuracySec=. OnUnitActiveSec deserves attention because it solves something cron does badly. OnBootSec=5min plus OnUnitActiveSec=5min means "five minutes after boot, and then every five minutes from when the previous one finished". With cron, */5 fires by the clock even if the previous run is still alive; here the interval is counted from the actual run, so there is no overlap by design and flock stops being indispensable — although it is worth keeping, because it also protects against manual runs. RandomizedDelaySec is the answer to the @daily problem: if twenty servers have the same timer, the random delay spreads the load. And AccuracySec lets systemd group activations to save power; the default is one minute, so if you need punctuality set it to 1s explicitly. This is also cron's great absentee: granularity. OnUnitActiveSec=30s is perfectly valid; in cron, "every 30 seconds" does not exist.

The syntax is DayOfWeek Year-Month-Day Hour:Minute:Second, with * for "any" and /N for steps:

And here comes one of the best reasons to use timers: you can check the expression before installing it with systemd-analyze calendar '*-*-* 06:30:00' --iterations=3.

It tells you whether the expression is valid, how it interprets it and when it would run; with --iterations it shows the following times, which is the definitive way to check that "every five minutes" means what you think. Cron offers nothing like it: there, the only way to check is to wait. OnCalendar also accepts time-zone suffixes (Mon *-*-* 03:00:00 UTC), which cleanly solves the daylight-saving problem you saw in 07-01.

A real scenario: the backup is scheduled at 03:15 and the server spends the night powered off for maintenance, booting at 09:00. With cron, the 03:15 run simply did not happen, and nobody will notice until that backup is needed. Persistent=true changes that: systemd stores on disk when each timer last ran and, at boot, checks whether an activation was missed; if so, it launches it immediately. It is anacron's functionality (07-01) built in, without a separate tool and with full granularity. It must be set on every job whose run matters — reports, backups, cleanups — and it must be left off on those that only make sense at their moment: a status check from eight hours ago contributes nothing at boot. Combine it with RandomizedDelaySec so that the overdue jobs do not all launch at once.

The sensible conclusion: cron for the simple and portable, timers for what matters. Cleaning up temporary files is still fine in cron; a backup that must not be lost, a job that needs the network or a service that must restart on its own call for a timer. There is no obligation to migrate everything: coexistence is normal.

After creating or modifying any unit you have to reload with sudo systemctl daemon-reload; forgetting it is the number-one mistake with systemd.

list-timers is the view that replaces crontab -l, and it gives quite a lot more information:

At a glance you know what is coming, when the last one was and whether something has gone too long without running: that LAST is exactly the datum check_backup (07-04) had to deduce by looking at directories. And systemctl start deserves separate praise, because it tests the job exactly under the conditions it will run in — same user, same environment, same limits: it is 07-01's env -i, but for real.

An advantage cron cannot offer: limiting what the job can do even if it has a bug.

With this, a bug in your backup script cannot write outside /backups and logs/, and PrivateTmp eliminates a whole family of attacks based on predictable temporary files: defense in depth almost for free, with the full implications in 08-03. Start soft (ProtectSystem=full, PrivateTmp=true, NoNewPrivileges=true) and harden afterwards, testing with systemctl start: if the job stops working, you now know which path is missing from ReadWritePaths.

Sometimes you do want a permanent process: a watcher that checks every 10 seconds, or something consuming a queue. The unit becomes Type=simple and adds Restart=on-failure (relaunches if it ends with a non-zero code; always relaunches even after a clean exit), RestartSec=10 to wait between attempts, and StartLimitBurst=5 with StartLimitIntervalSec=300 to avoid the infinite loop: if it fails five times in five minutes, systemd gives up and leaves the unit in failed, which is correct because a restart that fixes nothing only hides the problem. Restart= does not fix a badly written script: restarting something every 10 seconds when it fails because of a wrong path only fills the journal. It must also handle SIGTERM, which is what systemctl stop sends, picking up the trap from 05-03:

Two details make it work. The RUNNING flag lets it finish the work in progress before exiting, instead of dying halfway and leaving a corrupt file. And the wait is sliced into one-second sleeps instead of sleep 10, because Bash does not interrupt a long sleep until it finishes: with sleep 10, systemctl stop would take up to ten seconds to take effect and systemd would end up sending SIGKILL. If the script does not respond to SIGTERM, systemd waits TimeoutStopSec (90 s by default) and then kills it brutally, with all the corruption that may cause.

Not everything needs sudo: each user has their own systemd instance and their units in ~/.config/systemd/user/.

It is the same with --user added and without User=/Group=, since it runs as you; the advantage is that you need no privileges and the unit lives in your $HOME, version-controllable with the toolkit. There is one trap: by default, the user instance dies when you log out, so your timers stop firing. The solution is sudo loginctl enable-linger veloz — linger means "stay behind" — which keeps that user's instance alive even with no open session; without it, a user unit works while you are connected and mysteriously stops working when you leave. For server jobs the norm is still system units with User=veloz.

Let us take the watchdog line (*/5 * * * * flock -n … watchdog.sh >> … 2>&1) in six steps. Step 1: the service, which holds everything that is not the schedule; it is the one from section 2 with these directives changed.

SuccessExitStatus=1 is a fine adjustment: the watchdog returns 1 when there is some WARNING, and we do not want systemd to mark the unit as failed over a warning; the redirection to the log disappears because the output goes to the journal. Step 2: the timer, where OnUnitActiveSec is chosen instead of OnCalendar=*:0/5 so that the interval is counted from when the previous run finished and they never overlap, with no Persistent=true because an overdue check is of no interest. Step 3: validate with systemd-analyze verify …{service,timer}. Step 4: reload and test by hand with daemon-reload, systemctl start veloz-watchdog.service and journalctl -u veloz-watchdog -n 30 --no-pager. Step 5: enable with systemctl enable --now veloz-watchdog.timer and check with list-timers. Step 6: remove the cron line — it is constantly forgotten, and the result is the job running twice: comment it out instead of deleting it, with the date and the reason.

Compare what you gain against the crontab from 07-01: the report is recovered if the server was powered off, the watchdog waits for the network and never overlaps, the three logs are consulted with the same command filtering by unit, list-timers shows the state of everything at a glance, and systemctl start tests each job under conditions identical to the real ones.

• Forgetting daemon-reload, or enabling the .service instead of the .timer. Without a reload, you edit the unit and nothing happens; and enable veloz-report.service schedules nothing, because it is the timer that gets enabled.
• Type=simple on a job that finishes. The unit shows up as failed every time it works fine. Use oneshot.
• Redirections, globs or relative paths in ExecStart. There is no shell: >, | and $VAR are passed literally, and a relative path always fails.
• Leaving the cron line after migrating, or a user unit without enable-linger. The first runs the job twice; the second works while you have a session and stops working when you disconnect.
• A long sleep in a service, or Restart=always over a real failure. The first makes systemctl stop end in SIGKILL (slice the wait); the second hides the problem and fills the journal (use on-failure with StartLimitBurst).
• Tip: always validate with systemd-analyze calendar before installing a schedule, and with --iterations=5 to see the next five runs. Discovering a week later that your "every 5 minutes" was "at 5 in the morning" hurts.

Exercise 1. Write the .service + .timer pair for backup.sh: it must run at 03:15 as user veloz, wait for the filesystems to be mounted, recover the run if the server was powered off, take no longer than an hour, and only be able to write in /backups and in the logs directory.

Exercise 2. Write a veloz_timer_status function that takes a timer's name and returns 0 if it is active and its last run was successful, 1 if it has gone longer than expected without running, and 2 if the associated service is in a failed state.

Solution 1.

RequiresMountsFor= is more precise than a generic After=local-fs.target: it guarantees that those specific mount points are available, which is what a backup needs — if /backups is an external disk or an NFS share that has not mounted yet, the script would write into the empty directory of the local system without noticing. And ProtectHome=read-only allows reading ~/veloz-ops but not writing: that is why ReadWritePaths explicitly includes the logs directory.

Solution 2.

The interesting piece is systemctl show -p PROPERTY --value, which is the machine-readable way of querying systemd: systemctl status is designed for humans and its format changes between versions, whereas show returns the bare value and is stable. ExecMainExitTimestamp gives the date in a format date -d understands, so subtracting in seconds since the epoch (04-06) gives the age directly. is-failed distinguishes a real failure from "it never ran", which deserve different answers. This function is the check that was missing in watchdog.sh (07-04): watching that the scheduled jobs themselves are still working.

systemd manages units, and a Bash script cares about two: the .service (how it runs) and the .timer (when). They go in /etc/systemd/system/ or, for user ones, in ~/.config/systemd/user/ with loginctl enable-linger to survive logout. In the service, Type=oneshot for jobs that finish and simple for those that do not, ExecStart with an absolute path and no shell constructs, User/Group so as not to run as root, EnvironmentFile=- to load the configuration, TimeoutStartSec as the built-in timeout and StandardOutput=journal to centralize the logging. In the timer, OnCalendar for clock schedules — validatable with systemd-analyze calendar, something cron does not offer — the OnBootSec/OnUnitActiveSec family for relative intervals that do not overlap by design, Persistent=true to recover what was missed while the server was off, and RandomizedDelaySec to spread the load. The day to day is six commands: daemon-reload after every change (the most common omission), enable --now on the timer and not on the service, start to test the job under conditions identical to the real ones, status and journalctl -u to see what happened, and list-timers as a much improved replacement for crontab -l. Add the almost-free hardening of NoNewPrivileges, PrivateTmp, ProtectSystem and ReadWritePaths, and remember that in a long-running service Restart=on-failure with StartLimitBurst does not fix a badly written script, and that you have to handle SIGTERM with a trap and by slicing the sleeps. When migrating from cron, six steps and one that is always forgotten: removing the crontab line, or the job will run twice. With this, the toolkit's four jobs run on their own, recover from a shutdown, wait for the network and share their logging. But all of this happens on a single server, and in production there are three — srv-veloz-01, srv-veloz-02 and srv-veloz-03: checking the fleet's status by logging into each one by hand is not automation. In 07-06 we take the leap: ssh from a script's point of view, key authentication and its risks, ~/.ssh/config with connection reuse, the options without which an ssh inside a loop eats standard input, remote blocks with here-documents, and fleet walks in parallel with per-host error handling. fleet.sh is born.