The toolkit stops waiting for your orders. Until now, daily-report.sh only existed if somebody remembered to launch it, and service-status.sh ran when you already suspected something was wrong — that is, too late. cron is the daemon that has been solving exactly that since the seventies: a process that wakes up every minute, looks at a schedule table and runs whatever is due. It is simple, it is everywhere and it works. But it has one quirk that trips everybody up the first time: the environment in which it runs your scripts looks nothing like your terminal. This lesson covers cron's model, its field-by-field syntax and, above all, how to debug the classic "but it works in my terminal".

1. Cron's model and where jobs live
2. crontab -l, -e, -r and the keyboard hazard
3. The five-field syntax
4. Lists, ranges, steps and shortcuts
5. Cron's minimal environment
6. The %, MAILTO and capturing the output
7. Debugging a cron job
8. Overlap, permissions and time zone
9. Cron's relatives: at, anacron and timers
10. Application: the toolkit in the crontab

cron is a daemon: it starts with the system and stays alive. Its cycle is simple: it wakes up at the beginning of every minute, reads the job tables (crontabs), launches the commands whose time specification matches that minute, and goes back to sleep. Two consequences follow. First: the minimum granularity is the minute; "every 30 seconds" does not exist in cron (that is what the timers in 07-05 are for). Second: cron does not wait for the previous run to finish, so jobs can overlap (section 8).

Check it with what you learned in 06-03: systemctl is-active cron should return active (crond in the Red Hat family). If the daemon is not running, your jobs do not run and nobody tells you. Now the first source of confusion: there is not one crontab, there are several places and they do not share a format.

The key difference is in the middle column: system crontabs carry an extra field with the user, right after the five time fields. Per-user ones do not, because the user is already known.

Putting a user-style line in /etc/cron.d/ is a common mistake: cron will read /home/veloz/... as the user name and fail. The cron.daily directories and friends are another case: there are no schedules there, there are executable scripts (with no .sh extension and with execute permission) that the system launches once a day, at whatever time /etc/crontab or anacron decides.

Your personal crontab is always managed with the crontab command, never by editing files by hand.

-l and -r are adjacent keys on QWERTY, and there is no recycle bin. Two habits prevent disaster:

The second line is the real best practice: keep the crontab as a text file in your repository (Git in 08-04) and install it from there. It is actually stored in /var/spool/cron/crontabs/<user> (Debian/Ubuntu), but do not edit it there: crontab -e also validates the syntax and notifies the daemon.

Every line starts with five space-separated fields, from the smallest unit to the largest:

The asterisk means "any value", so 30 6 * * * reads: minute 30, hour 6, any day, any month, any day of the week → every day at 06:30. One detail that surprises people: when day of month and day of week are both restricted (neither is *), cron combines them with a logical OR. 0 0 13 * 5 is not "Friday the 13th", it is "every 13th and also every Friday"; for "Friday the 13th" you have to put the condition inside the command.

Each field accepts four constructs, which can be mixed:

*/N does not mean "every N minutes from now", it means "the values divisible by N". That is why */7 gives 0, 7, … 56 and then jumps back to 0: only 4 minutes pass between the two runs straddling the hour. With divisors of 60 (2, 3, 5, 10, 15, 20, 30) the spacing is exact.

There are also shortcuts: @yearly (0 0 1 1 *), @monthly (0 0 1 * *), @weekly (0 0 * * 0), @daily (0 0 * * *), @hourly (0 * * * *) and @reboot, which runs when cron starts. They are readable, but they all fire at the same time: if ten servers use @daily, all ten start at 00:00 sharp and compete for network and disk. In production 17 3 * * * with a deliberately chosen time is better. And @reboot fires when the daemon starts, not when the system is ready: if your script needs the network, it may arrive too early (systemd dependencies solve this, 07-05).

This is the section of the lesson: when a script works in your terminal and fails under cron, the answer is almost always here.

Cron does not open a login shell or an interactive one. Review 01-04: ~/.bashrc is not read, nor ~/.bash_profile, nor /etc/profile.

Three practical consequences: a jq installed in /usr/local/bin is not found; an alias or function from your .bashrc does not exist; and SHELL=/bin/sh means the crontab line is interpreted by sh, with no [[ ]], arrays or <( ) (inside your script you do have them, because its shebang rules). The most useful experiment you can run is to schedule * * * * * env > /tmp/cron-env.txt and compare it with your own env.

There are three solutions, all three legitimate:

Crontab assignments (option 2) are ordinary environment variables, they apply to every job in that table and they do not support expansion: PATH=$PATH:/opt/bin does not do what you expect, you have to write the full value.

Inside a crontab line, the first % marks the end of the command and everything after it is sent as standard input; the following ones become newlines. It is a perfect trap because % appears in every date format string:

The way to never have to remember this rule is not to put logic in the crontab: a line should be an absolute path, its options and a redirection; nothing else.

About the output: if the job writes anything to stdout or stderr, cron tries to mail it to you locally. On a server with no mail agent, that message is lost and your errors disappear. MAILTO=ops@veloz.example sets the destination and MAILTO="" disables mail. The right pattern on a modern server is a log of your own:

Review 02-04: >> appends without truncating and 2>&1 afterwards sends errors to the same file too. And the bad practice you will see everywhere: a blind >/dev/null 2>&1 silences the job completely, so the day it fails there will be no mail, no log and no trace. It is only acceptable when the script already writes its own record (07-04), and even then it is worth keeping a 2>> to an error file.

When something scheduled does not work, follow this order.

Was it even attempted? Cron records every launch in the system log:

If that line is missing, the problem is the time specification or the crontab, not your script; if it is there, the problem is inside.

Reproduce the environment. env -i clears it completely and lets you run the way cron would:

If it fails here but not in your terminal, you have your diagnosis: you depend on something in the environment, usually a binary in /usr/local/bin. Next, look at the job's log (if you did not redirect it, do it now) and, if necessary, turn up the detail with set -x and the custom PS4 from 05-03, or by calling the script with bash -x.

Overlap. Cron does not check whether the previous run is still alive. Schedule service-status.sh every 10 minutes, let the API hang for 40, and you will have four processes fighting over the same log. You have known the solution since 05-02:

-n means "if the lock is taken, exit immediately", so the overlapping run is discarded instead of piling up. In 07-02 you will see the full version, with the lock inside the script itself.

Permissions. If /etc/cron.allow exists, only the listed users may have a crontab; if it does not exist but /etc/cron.deny does, everybody may except those listed. A "you are not allowed to use this program" is always explained by these two files.

Time zone and daylight saving. Cron uses the system's (timedatectl), so the same line runs at different times on servers in different zones: the usual practice is to put them all on UTC. And in spring, when the clock jumps from 02:00 to 03:00, a job at 02:30 does not run that day; in autumn it may run twice. Rule of thumb: do not schedule anything between 02:00 and 03:00 if you care about it happening exactly once — and if the job is idempotent (07-02), this stops being a problem.

at runs a command exactly once, at a specific moment: echo '/path/backup.sh' | at 23:00, with at -l to list and atrm N to cancel. It is ideal for "launch this when the maintenance window ends" without leaving anything permanent. anacron, in turn, solves a design problem of cron: if the machine is powered off at the scheduled time, the run is lost. Anacron does not work with times but with periods in days and, on boot, it checks how long ago each job ran. That is why it is the norm on laptops, and why on many systems the cron.daily directories are actually triggered by anacron.

Timers cover everything above and add dependencies, centralized logging and isolation, in exchange for more files. They are the subject of 07-05.

This is srv-veloz-01's crontab, kept as a version-controlled file in ~/veloz-ops/etc/crontab.veloz:

Every decision answers something from this lesson. SHELL=/bin/bash so the lines are interpreted by Bash. A full PATH because jq lives in /usr/local/bin. MAILTO="" because the server has no mail and we already redirect to a log. VELOZ_OPS as a variable of our own, because cron does expand inside commands the variables defined in the crontab. 30 6 and not @daily: a chosen time, not competing with everything else at midnight. */10 because 10 divides 60 and the spacing is exact. flock -n only on the second one, which is the one that can take longer than its interval. >> … 2>&1 on both, never /dev/null. And no logic and no % in the lines: dates and decisions live inside the scripts.

You install it with crontab -l > ~/veloz-ops/etc/crontab.bak (safety net) followed by crontab ~/veloz-ops/etc/crontab.veloz, and the next day you verify it with journalctl -u cron --since "06:00". Those two logs will grow forever: rotation with logrotate is solved in 07-04.

• Relative paths. Cron runs from $HOME with a minimal PATH: ./daily-report.sh does not work. Always an absolute path.
• Assuming .bashrc is read. It is not: no aliases, no functions, and not the PATH you added there.
• Forgetting 2>&1, or using >/dev/null 2>&1 out of habit. The first leaves errors out of the log; the second silences the only warning you were going to get.
• The unescaped %. It cuts the command dead. Better: do not put date in the crontab.
• Confusing the user crontab with /etc/cron.d/. The latter carries a user field; the former does not.
• A missing final newline, or a script that is not executable. Some versions ignore the last line without a \n; and without chmod +x (02-03) you have to invoke it as /bin/bash /path/script.sh.
• crontab -r instead of -l. Make a copy before touching anything and keep the crontab in a version-controlled file.
• Tip: when you roll out a new job, schedule it every 2 minutes first with output to a log, check that it works and only then give it its final schedule. Waiting 24 hours to discover a path typo is wasted time.

Exercise 1. Translate into cron syntax: (a) every 5 minutes between 07:00 and 21:59, Monday to Saturday; (b) the first day of each month at 04:15; (c) every half hour every day; (d) Mondays, Wednesdays and Fridays at 22:00.

Exercise 2. This line "does not work". Find the four problems and rewrite it:

Exercise 3. Write a veloz_cron_install function for lib/common.sh that installs a crontab file safely: it saves a copy of the current one, validates that the new one exists and is not empty, installs it and verifies that it really got installed, restoring the copy if anything fails.

Solution 1.

The typical mistake in (a) is writing 7-22 thinking "up to 21", which would add the runs from 22:00 to 22:55.

Solution 2. The problems: (1) ~ is not expanded reliably in the crontab; (2) an unescaped %, which cuts the command at date +; (3) > /dev/null 2>&1 silences the backup, so if it fails nobody finds out; (4) a relative path that depends on a cd which, if it fails, silently leaves the && unexecuted.

The destination's date is computed inside the script, which is where it belongs: that way the crontab has no logic, there is no % to escape and the script is tested by hand exactly as it runs on a schedule.

Solution 3.

Three decisions deserve a comment. [[ -s ]] checks in a single test that the file exists and is not empty (03-04): installing an empty crontab is the same as an accidental crontab -r. The || : after crontab -l neutralizes the non-zero exit code it returns when the user has no crontab yet, which is not a failure; without it, set -e (05-03) would abort the function. And the verification with diff -q <(crontab -l) uses the process substitution from 05-05 to compare what actually got installed with what you wanted, instead of trusting crontab's exit code.

cron wakes up every minute, reads some tables and runs whatever is due. Jobs live in the user crontab (no user field), in /etc/crontab and /etc/cron.d/ (with one) or as standalone scripts in /etc/cron.{hourly,daily,weekly,monthly}. The five fields are minute, hour, day of month, month and day of week, and they accept values, lists 1,15, ranges 1-5, steps */10 and combinations; the @daily or @reboot shortcuts are readable but concentrate the load on round hours. Manage the table with crontab -l, -e and crontab file, taking great care with -r, which deletes without asking. What really separates a job that works from one that does not is the environment: cron does not read ~/.bashrc, gives you a two-directory PATH and SHELL=/bin/sh, and that is where almost every "it worked in my terminal" comes from; you fix it with absolute paths, by defining PATH in the crontab or by loading the configuration explicitly. Add the % that has to be escaped, always redirecting with >> file 2>&1 instead of throwing the output at /dev/null, wrapping anything that might overlap with flock -n, and debugging in order: first journalctl -u cron to find out whether it was attempted, then env -i to reproduce the environment.

You now have the report at 06:30 and the status check every 10 minutes running on their own. But there is an enormous difference between scheduling a script and having a script that can run with nobody in front of it. What happens if it asks something at the keyboard and there is no keyboard? If it runs twice because of a retry? If it hangs waiting for a server that never answers? How do you know, the next morning, whether it did its job? In 07-02 we turn that into seven concrete properties — non-interactive, idempotent, exclusive, observable, bounded, fail-safe and configurable — and implement them one by one on daily-report.sh, including the --dry-run that will let you test an automated job without fear.