With awk and sed the toolkit already knows how to read, summarize and transform any text you put in front of it. But all that text still comes from files somebody left on disk. An operations script needs something more: to ask the system itself what state it is in —how much disk is left, what load it is carrying, who is connected, what operating system version it runs, whether veloz-api is alive— so it can decide based on the answer. This lesson goes through the commands that interrogate the system, which of them are reliable for a script and how to turn their answers into thresholds and decisions, which is what separates a script that reports from one that acts.

1. Identity and context: who am I and where am I
2. Detecting the distribution correctly
3. Time and locale settings
4. Disk: df and du
5. Memory, CPU and load
6. /proc: the reliable source
7. Storage and open files, in broad strokes
8. Users and sessions
9. Dependencies: check before failing
10. Asking about services
11. The process environment
12. Reporting back: stderr, logger and email
13. Thresholds and decisions
14. Application: service-status.sh is born

The first thing a script should know is what identity it is running under, because whether it can read /var/log/veloz/ or restart a service depends on it:

In a script, the check is written [[ $EUID -eq 0 ]] && veloz_die 1 "do not run this as root". $EUID is a Bash variable and spawns no process, so it is preferable to $(id -u) inside a loop. The golden rule is the opposite of the intuition: a reporting script should refuse to run as root, because it does not need to and any mistake of its own gets amplified (08-03).

uname tells you the kernel (Linux), not the distribution. For that there is /etc/os-release, a standard file in every modern distribution with KEY=value format designed to be loaded from a script:

With $ID (ubuntu, debian, fedora…) and $ID_LIKE you pick the right branch in a case (04-05) without guessing. The alternatives you see around are worse: lsb_release -a spawns a process and is often not installed, and checking whether /etc/debian_version exists is fragile. One security nuance: loading the file with . executes its contents, which is acceptable in /etc/os-release (owned by root) but must not be done with files anyone can write.

uptime gives you at a glance how long the machine has been up and its load; uptime -p says it in prose and uptime -s gives the boot date, which is the useful one for spotting an unexpected reboot. For dates, date was already used in 04-06: remember date +%F (2026-08-03), date +%s (epoch) and date -d 'yesterday' +%F. Time zones are controlled through the environment: TZ=UTC date +'%F %T %Z' prints 2026-08-03 17:42:10 UTC without touching the machine's configuration, and timedatectl shows the system zone and whether it is synchronized by NTP. Logging in UTC saves grief: daylight saving changes make one hour repeat and another not exist, which breaks reports and comparisons. And here comes one of the most profitable lessons of the module: LC_ALL=C in scripts. The locale changes the behavior of the tools, not just the messages.

The practical consequence: if your script compares, sorts or searches for patterns in a command's output, put LC_ALL=C in front of it. If a person is going to read that output, leave it in their language.

df reports the space per mounted filesystem; du measures what a specific directory takes up. They are not interchangeable: df asks the filesystem and is instantaneous, du walks the tree and can take minutes.

-P is the key option for scripts, and it deserves an explanation. Without it, df splits the line in two when the device name is long, and your awk '{print $5}' reads the wrong column of a split line. -P (POSIX format) guarantees one line per filesystem. Note as well that -h and automatic processing are incompatible: 1.9G cannot be compared with > (04-06). In scripts you use df -P (blocks) or df -PBM for megabytes, and -h only for what a person reads. sort -h does understand suffixes like 1.9G, which is why it works with du -sh.

free -m shows memory in megabytes, nproc the number of available cores and uptime the load figures at the end of the line. From free -m the number that matters is available, not free. Linux uses free memory as a disk cache, so free almost always looks low and means nothing; available estimates how much is really there for starting something new. A script that warns on low free will raise false alarms every single day.

The load average is three numbers: the average of processes runnable or waiting on disk over 1, 5 and 15 minutes. Interpreting it depends on the number of cores: a load of 4 is comfortable on an 8-core machine and serious on a 2-core one. That is why the correct threshold is never a fixed number but a comparison against nproc, and the one worth deciding on is the 5- or 15-minute figure —the 1-minute one is too jumpy and fires alerts on any spike—.

Almost all the previous commands are nothing but pretty readers of /proc, a virtual filesystem where the kernel publishes its state as plain text. Reading it directly is faster (no processes spawned) and far more stable, because the format of /proc does not change with the command version or with the language:

That read -r load1 load5 _ < /proc/loadavg from 03-05 spawns not one single process and replaces uptime | awk ...: two processes fewer and no format that can change. It is the difference between parsing the output of an interactive command —meant for humans, liable to change format, translated— and reading a kernel interface, which is a stable contract. Whenever the second option exists, take it.

lsblk shows the disks and partitions as a tree, with their mount points and sizes; lsblk -f adds the filesystem and the UUID. mount (or better, findmnt, which gives tabulated, filterable output) lists what is mounted and with what options —checking that a /mnt/backup is mounted before writing the backup avoids filling the root disk, a classic we will see in 07-03—. And lsof lists open files: lsof /var/log/veloz/access.log says which process has it open, and lsof -p 1234 what a process has open. It is the tool for finding out why something cannot be unmounted or who is holding an already-deleted file that still takes up disk.

who lists the open sessions, w adds what each one is running plus the load, and last shows the history of logins and reboots by reading /var/log/wtmp —last -x reboot is the quick way to see when the machine was restarted—.

To query accounts, the temptation is grep alopez /etc/passwd, and it is a mistake: it only works if the users are local, and as soon as there is LDAP or a corporate directory it returns nothing. The correct way is getent, which queries the same databases as the system (file, LDAP, DNS…) according to /etc/nsswitch.conf:

So getent passwd alopez || veloz_die 1 "user alopez does not exist" is a valid guard on any machine, and getent group veloz | cut -d: -f4 lists the group's members. It is the same command that in 06-04 will serve to resolve host names with getent hosts, and that is why it is worth getting fond of: one interface for all the system's databases.

A script that uses jq and runs on a machine without jq fails halfway through, leaving temporary files and half-finished work. Checking the requirements at startup is the direct application of the guard clauses from 03-04:

command -v is the correct way: it is a builtin (no processes spawned), it is POSIX, and it also finds functions and aliases, unlike which, which is an external executable, is not everywhere and returns inconsistent codes. The array accumulates all the missing ones instead of dying on the first, which is far kinder to whoever installs it. Code 127 is the one Bash uses for "command not found" (05-03).

To find out which version of a package is installed, dpkg -l <package> on Debian/Ubuntu and rpm -q <package> on Red Hat/Fedora. Use them for diagnosis, not as a check: what matters to your script is that the command is available and works, not how it was installed.

To know whether veloz-api is alive there are two complementary routes. pgrep (05-02) asks whether the process exists; systemctl is-active asks whether the service manager considers it active, which is more reliable because it takes restarts and failures into account:

--quiet suppresses the output and leaves only the return code, which is what an if cares about. Without --quiet, is-active prints active, inactive, failed or activating, useful for the error message. To see why it failed, journalctl -u veloz-api -n 20 --no-pager shows the last 20 lines of its log; --no-pager is essential in a script, because otherwise journalctl tries to open less and hangs waiting. The detail of systemd —units, timers, dependencies— is lesson 07-05; here we only query it.

env (or printenv) lists the exported variables, and on srv-veloz-01 it is worth remembering that the environment of your interactive session is not the one the script will have when cron launches it (07-01): there the PATH is minimal and HOME may be different. That difference is cause number one of "it works in my terminal and fails in cron".

env -i goes to the opposite extreme: it runs a command with a completely empty environment, and it is the way to check what your script really depends on:

If the script works with that line, it will work in cron. It is a two-second test that saves an afternoon of debugging.

A script that interrogates the system has to report what it saw, and there are three destinations depending on who is listening. The first, already known from 02-04, is stderr for diagnostics and stdout only for the result, so that daily-report.sh > report.txt does not mix warnings with data. The second is the system log, with logger:

-t sets the tag it will appear under in the log and -p the facility and priority. The advantage over writing to your own file is that the message enters the same system as everything else —queryable with journalctl -t veloz-ops, rotated and forwardable to a central server—. The third is email: mail -s "subject" ops@veloz.example or sendmail read the body from standard input, and they are the traditional way for cron to warn you. All three are combined in the monitoring strategy of 07-04; here it is enough to know they exist and that a serious script does not limit itself to printing on screen.

Everything above is data. What turns a script into an operations tool is applying a threshold to it and acting. Three rules to keep thresholds from becoming a nuisance:

• Configurable, not embedded. The 85% disk figure goes in veloz-ops.conf (05-06), not inside the if.
• Relative when the absolute says nothing. Load is compared against nproc; free megabytes, against the disk size.
• With hysteresis. A threshold that fires at 84.9% and goes quiet at 85.1% generates intermittent alerts. It is better to warn once and not repeat until it crosses back with some margin.

Notice the trick with the load: since it is a decimal number, (( )) is no use (04-06), so the comparison is delegated to awk, whose BEGIN { exit !(c > n) } turns the result into an exit code —0 if it holds— usable directly in an &&. It is the natural bridge between the awk of 06-01 and Bash's logic. This battery of checks is exactly the skeleton of project 09-01.

The toolkit gains its second executable. daily-report.sh answers "what happened yesterday?"; service-status.sh answers "how is this right now?":

The export LC_ALL=C at the top fixes the behavior of everything the script invokes, not just of one command. The double check of systemctl and pgrep is not gratuitous redundancy: systemd can consider a unit active whose process has turned into a zombie. And the final exit code is what makes the script useful: service-status.sh || notify works from another script, from cron or from a timer. What it still cannot do is check whether port 8080 responds, which is different from the process existing: the next lesson closes that gap.

• Parsing df -h in a script. 1.9G does not compare against numbers and the line can be split. Use df -P and, if need be, -BM.
• Looking at free instead of available. Linux uses free RAM as a cache; low free is normal, not an alert.
• Using which. It is not POSIX, it is an external process and its codes vary. command -v always.
• Comparing the load with (( )). It is decimal: (( 1.5 > 1 )) gives a syntax error. Delegate to awk.
• grep on /etc/passwd. It only sees local users. getent passwd sees them all.
• A fixed load threshold. Always compare it against nproc; 4 does not mean the same on 2 cores as on 16.
• journalctl without --no-pager. The script hangs waiting for less.
• Tip: prefer /proc/loadavg and /proc/meminfo to parsing uptime or free: fewer processes and a format that does not change.
• Tip: test your script with env -i PATH=/usr/bin:/bin ... before putting it in cron; you will see the hidden dependencies on the environment.

Exercise 1. Write a function veloz_disk_usage for lib/common.sh that takes a mount point and returns its usage percentage as a whole number, without the %, working even when the device name is long and returning code 1 if the mount point does not exist.

Exercise 2. Write a snippet that checks whether the 15-minute load average exceeds the number of cores and, if so, records a warning in the system log with the tag veloz-ops and exits with code 1. The load is decimal.

Exercise 3. Add to service-status.sh a system summary with: distribution and version, uptime, number of open sessions, available memory in MB and the three fullest partitions. Use the most reliable source for each figure.

Solution 1.

-P guarantees a single line per filesystem, which is exactly the case that breaks the naive version. gsub(/%/,"",$5) strips the symbol and $5+0 forces the result to a number, so the caller can use it with (( )) without any further cleanup. LC_ALL=C in front of df —and not exported— applies only to that command. Careful: if the caller uses set -e, it is worth invoking it as usage=$(veloz_disk_usage /srv/veloz) || return 1.

Solution 2.

/proc/loadavg has five fields (1min 5min 15min processes last_pid), so the third is taken with two _ in front. The decimal comparison goes in awk: exit !(c > n) returns 0 —success for the if— when the condition is true, because in awk exit 0 means success and the negation ! turns the true (1) into 0. It is the standard idiom for using awk as an evaluator of numeric conditions from Bash.

Solution 3.

Each figure uses its correct source: /etc/os-release instead of lsb_release, /proc/meminfo instead of parsing free, df -P instead of df -h. -x tmpfs -x devtmpfs excludes the in-memory filesystems, which always show up at 0% or 100% and only clutter the listing. The sort -k2 -rn goes after the awk because sorting an already-clean percentage is trivial, whereas sorting df's raw output with the % attached is not.

A script decides well when it asks well. Identity comes from $EUID, id and whoami; context, from hostname -f, uname and above all /etc/os-release, which is the correct way to know which distribution you are on. Time is handled with date, uptime -s and TZ, logging in UTC, and LC_ALL=C fixes the behavior of sort, regexes and messages so the script does not depend on the machine's language. Resources come from df -P (never -h for processing), du -sh, free -m looking at available and nproc; but the truly reliable source is /proc —loadavg, meminfo, cpuinfo, <pid>/status—, a kernel contract that does not change format or get translated, and that on top of that is read without spawning processes. getent queries users and groups wherever they come from, command -v checks dependencies at startup instead of failing halfway, systemctl is-active --quiet and journalctl --no-pager interrogate services, and env -i reveals what environment your script depends on before cron discovers it. All of it culminates in configurable, relative thresholds with hysteresis that turn data into decisions, with awk as the evaluator when the number is decimal.

service-status.sh already knows how to say whether the machine is healthy and whether the veloz-api process exists. But "the process exists" and "the service responds" are not the same thing: a process can be alive with port 8080 closed, blocked or unreachable from another machine. For that you have to step outside the local system and look at the network: check connectivity and name resolution, see which ports are listening, test whether a port responds —with nc or with the /dev/tcp noted down in 05-05— and wait with retries for a service to come up. That is the next lesson (06-04).