In 05-01 you learned to pick precisely which files the toolkit acts on. But an operations script does not just read and write: it launches processes. daily-report.sh invokes grep, bc, tar and queries veloz-api; it walks four cities one after another when it could do them at once; and it has no defense at all if the 6:00 report is still running when cron starts the 6:05 one —two processes writing the same file, counters mixed together and a meaningless report—. In 02-06 you saw just enough not to lose control of the terminal: Ctrl-C, Ctrl-Z, jobs, fg, bg, &. Here we complete that picture and take it into scripting territory: viewing, searching, launching in parallel, waiting, signaling, limiting and locking.

1. The Unix process model
2. Seeing what runs: ps, pstree, top
3. Finding processes: pgrep
4. Foreground and background
5. Surviving logout: disown, nohup, setsid
6. $! and wait: controlled parallelism
7. Signals
8. kill, pkill, killall and timeout
9. Priorities with nice and renice
10. One instance at a time: flock
11. Zombies and orphans

A process is a running program with its own memory, its own environment (the exported variables from 01-02) and a number that identifies it: the PID. Every process also has a PPID, the PID of whoever created it. That turns the system into a tree whose root is init/systemd (PID 1).

When in 01-04 you saw that an external command runs in a child process, that is exactly what happens: Bash duplicates itself (fork) and the duplicate turns into the command (exec). Two consequences follow that you already know and that now fall into place: the child inherits the parent's environment, and nothing the child changes affects the parent —the reason cd is a builtin and the reason a pipeline's loop does not see its variables (Module 4)—.

The states a process can be in show up in the STAT column of ps: R running or ready, S sleeping while waiting for something (the normal case), D in uninterruptible disk wait, T stopped, Z zombie.

ps has two historical syntaxes that coexist; both are correct and you will see both:

The ps aux columns people actually read:

ps -ef shows PPID instead of %CPU, which makes it better for investigating who launched what. And -o selects only the fields you care about, which is what gets used in scripts:

--sort=-rss sorts by memory descending; etime is the elapsed time since startup, far more useful than TIME for finding out whether something has been hung since yesterday.

pstree -p 2210 draws the tree of a process and its children, ideal for understanding what a script has launched. top (and its more comfortable version htop) are interactive: they refresh live and are meant for watching, not for scripting. In a script you never use top; you use ps with -o.

The learned reflex is ps aux | grep veloz-api, and it has a flaw that causes real bugs:

The grep shows up in its own search, because ps lists it while it runs. An if ps aux | grep -q veloz-api is always true, even when the API is down. The traditional workaround is the [v]eloz-api character-class trick, but the right tool exists:

-f is essential for scripts: without it, pgrep daily-report.sh finds nothing, because the process name is bash and the script is just an argument. And pgrep returns code 0 if it found something and 1 if not, so it slots straight into a guard like the ones in 03-04:

The shell groups the processes it launches into jobs, and numbers them. Recapping and extending 02-06:

The references %1, %+ (the most recent), %- (the previous one) and %?tar (by command text) work with fg, bg, kill and wait. One detail that matters: jobs are a concept of the interactive shell; inside a script there is no job control, but &, $! and wait do work, and they are the ones you use.

When you close the terminal, the shell sends SIGHUP (hang up, inherited from the modem days) to its jobs, and they die. Three ways to prevent it —nohup cmd > output 2>&1 &, cmd & disown %1 and setsid cmd > /dev/null 2>&1 &—:

disown is the rescue when you already launched something long-running and realize you need to close the session. For genuinely unattended tasks, systemd (07-05) is the right answer; nohup is the quick fix.

$! holds the PID of the last process launched in the background, and wait waits for it to finish. With those two, daily-report.sh can compute the four cities at once:

Three things make this pattern work rather than being merely "launching stuff":

• Each child writes to its own file. If they all wrote to the same one, their outputs would interleave mid-line. A mktemp -d directory (05-01) is the natural place.
• wait PID returns that child's exit code, so parallelism does not give up error handling.
• wait with no arguments waits for all children, but then you lose the individual codes. wait -n returns as soon as any of them finishes, which is the basis of a queue with a maximum number of simultaneous tasks.

Remember that each child is a separate process: variables it modifies do not come back to the parent. That is why the results travel through files, exactly as with the pipeline subshell in Module 4.

A signal is an asynchronous notification the kernel delivers to a process. The process can handle it, ignore it or let the default behavior take over —except for two that nobody can intercept—.

The diagram contains the whole lesson: the route through CleaningUp is what lets a process close files, delete temporaries and leave the system consistent. SIGKILL cuts that route off. Catching signals in your own scripts is done with trap, and that is the central topic of 05-03; here we deal with sending them.

Why kill -9 is the last resort: SIGKILL never reaches the process, the kernel carries it out. The program does not close its files (leaving a half-written CSV), does not delete its temporaries, does not release its lock, and its children are left orphaned. The correct sequence is SIGTERM, wait a few seconds checking with kill -0 "$pid" —which sends no signal at all, it only asks "does it exist and may I signal it?"— and only if it is still alive, SIGKILL. You will build it that way in exercise 3.

timeout solves the other side of the problem: something that never finishes.

If it expires, timeout returns 124. That code is the way to tell "it failed" from "it took too long", and it is essential in any network call inside a scheduled task.

The value runs from -20 (highest priority) to 19 (lowest). Only root can lower the number; anyone can raise it, that is, be considerate. For Veloz Envíos' nightly archiving, nice -n 10 together with ionice -c 3 keeps the compression from competing with veloz-api for CPU and disk.

The problem that opened the lesson: two simultaneous daily-report.sh. The homemade fix is a PID file:

It is fragile for two reasons. Between the check and the write there is a window in which another process does the same thing (a race condition, the same family as the /tmp/file.$$ one from 05-01). And if the script dies suddenly, the file stays there and blocks every future run until someone deletes it by hand.

flock uses a kernel lock tied to an open descriptor, which the system releases on its own when the process dies, no matter what:

exec 9> opens the file on descriptor 9 (descriptors are the subject of 05-05; for now, read it as "reserve channel 9 for this file"). flock -n 9 tries to lock it without waiting. If you prefer to queue instead of aborting, flock -w 60 9 waits up to 60 seconds. The lock disappears when the script ends, even if that is via SIGKILL. When you take this to cron in 07-02 you will see the one-line variant, flock -n /path/lock command, designed precisely for the crontab.

When a process finishes, its entry stays in the system table until its parent collects the exit code with wait. During that interval it is a zombie (STAT = Z): it no longer consumes CPU or memory, just a table entry. They are normal and short-lived; they are only a symptom if hundreds pile up, which points to a parent that launches children and never calls wait —exactly the mistake the pattern in section 6 avoids—.

An orphan is the opposite: its parent died before it did. The system adopts it by assigning init/systemd as its parent, so it keeps running normally and someone will collect its code when it finishes. A process launched with nohup ... & ends up as an adopted orphan, and that is why it survives.

• ps aux | grep process in a condition. It finds itself and is always true. Use pgrep.
• pgrep daily-report.sh without -f. The process is called bash; without -f you will not find your script.
• Starting with kill -9. It leaves half-written files, undeleted temporaries and orphaned locks. SIGTERM first.
• Launching N tasks with & and not calling wait. The script finishes before its children and the results are not there.
• Expecting a child to modify the parent's variables. It is another process: communicate through files or through stdout.
• Homemade PID files as a lock. A race when creating them and a permanent lock if the script dies. Use flock.
• Tip: every call to an external service inside an unattended task goes wrapped in timeout. A curl with no limit can leave the 6:00 task hanging until the next day's.

Exercise 1. Write api_alive(), which returns 0 if the veloz-api process is running and 1 if not, without falling into the problem of the grep that finds itself, and which also reports to stderr the PID and how many minutes it has been running.

Exercise 2. Modify the city loop in daily-report.sh so the four are computed in parallel, each in its own file, waiting for them all and counting how many failed.

Exercise 3. Write a function stop_service() that sends SIGTERM to every veloz-api process, waits up to 15 seconds checking once a second and only then falls back to SIGKILL, reporting which route it took.

Solution 1.

pgrep -x demands an exact name and already returns 1 if there are no matches, so the || return 1 is enough as a guard (03-04). ps -o etime= with the trailing = prints the value without a header, and tr -d ' ' strips the alignment padding.

Solution 2.

Each child's errors go to its own .err so they do not interleave. With four cities the saving is three quarters of the time if the bottleneck is reading the CSV. Note that rm -rf "$tmpdir" only runs if we get there alive; in 05-03 you will turn it into a trap.

Solution 3.

$pids goes unquoted on purpose —the only case in the whole course where that is correct—: we want the list of PIDs to split into several arguments for kill. And the final return 1 lets the caller know that death had to be forced, which probably deserves an entry in the log.

Every external command is a process with its PID, its PPID and its place in a tree rooted at systemd. ps aux and ps -ef show it all —with RSS and etime as the most useful columns—, ps -eo selects fields for scripts, pstree draws the hierarchy and top/htop are for watching live, never for scripting. pgrep, with -f and -x, replaces the ps aux | grep that finds itself. &, jobs, fg, bg and %1 govern the shell's jobs; nohup, disown and setsid take them out of SIGHUP's reach. The pair $! + wait turns four sequential computations into four parallel ones without giving up their exit codes. Signals are the language between processes: SIGTERM asks for termination and allows cleanup, SIGKILL cannot be intercepted and that is why it is the last resort, timeout sets a limit and returns 124. And flock on a descriptor gives the mutual exclusion a homemade PID file will never give.

daily-report.sh now knows how to check whether the API is alive, split the work across four processes and refuse to run twice at once. But it still has a hole: if something fails inside —an undefined variable, a grep with no results, a disk that fills up mid-write—, the script carries on as if nothing happened and produces a silently false report, on top of leaving the mktemp temporary undeleted. In 05-03 we close that hole: set -euo pipefail with its honest critique, PIPESTATUS, trap with EXIT and ERR for the guaranteed cleanup we have been promising for two lessons, a die() function that reports the exact line of the failure, an exit-code convention and the debugging tools bash -x, PS4 and BASH_XTRACEFD.