Closing 05-02 we identified the hole daily-report.sh still has: if something fails inside, the script never finds out. Bash, by design, runs the next line no matter what happened on the previous one. If the grep over the CSV finds nothing, if a variable is misspelled and expands to the empty string, if the disk fills up mid-write, the script carries on and produces a silently false report —the worst thing an operations tool can do, because nobody suspects a wrong number that arrives punctually every morning—. And on top of that it leaves behind the mktemp temporary we promised in 05-01 to learn to clean up. This lesson gives Bash the discipline it does not ship with, and gives you the tools to see what is going on when something does not add up.

1. The silent failure
2. set -e: what it catches and what it does not
3. set -u: undefined variables
4. set -o pipefail and PIPESTATUS
5. set -euo pipefail and its honest critique
6. trap: signals and pseudo-signals
7. Guaranteed cleanup with EXIT
8. die() and trap ... ERR with context
9. Exit-code convention
10. Retries with backoff
11. Debugging: -n, -x, PS4, -v, DEBUG and bisection
12. daily-report.sh hardened

That example, in one variant or another, has destroyed real servers. Bash reports the error on stderr and carries on. Against that there are three switches.

set -e (or set -o errexit) makes the script terminate as soon as a command returns a non-zero code: with it at the top, the previous example dies at the cd and never reaches the rm. What matters is knowing what it does not catch, because the exceptions are many and deliberate: a command whose failure is part of a decision must not kill the script.

The last two rows are the ones that cause the most headaches. The local one has a fix: declare and assign on separate lines (local report_date and then report_date=$(date -d "$1" +%F)), and that way the failure is detected. And the second-to-last one —a function loses set -e inside it when called from an if or from an &&— is the main reason some people distrust set -e. It is not a Bash bug: it is POSIX. But it means you cannot delegate all your error handling to -e.

Under set -u, an echo "Report for $CITYY" with the variable misspelled aborts with bash: CITYY: unbound variable. Without -u, that typo expands to the empty string and produces rm -rf "$BASE_DIR/" with BASE_DIR empty, that is, rm -rf /. It coexists perfectly with the default values from 03-06, which are the explicit way of saying "this one may not exist":

Careful with arrays: under -u and in Bash older than 4.4, "${arr[@]}" on an empty array aborts. The safe idiom is "${arr[@]:-}" or checking beforehand with (( ${#arr[@]} > 0 )).

In 02-04 this was deferred: the exit code of a pipeline is that of its last command.

A failure like that in daily-report.sh produces "0 errors today" when the truth is "I could not read the file". pipefail fixes it: the pipeline returns the code of the last command that failed, or 0 if none failed. When you need to know which link failed, the PIPESTATUS array holds every code, in order:

PIPESTATUS is overwritten by every command, including an echo. If you are going to use it, copy it first: local -a codes=( "${PIPESTATUS[@]}" ).

The header you will see in almost every serious script is set -euo pipefail right below the shebang, sometimes accompanied by a restrictive IFS=$'\n\t' (03-06). It is a good default and you should use it. But it is worth stating clearly why it is not magic:

• set -e has the hole of functions called from if, so critical functions must check their own errors.
• It aborts with a message from the command that failed, without saying which line of your script it happened on. That is fixed by the trap ... ERR in section 8.
• It turns any non-zero code into a fatal one, and there are commands where that is normal: grep with no matches, diff with differences, pgrep with no processes. You have to mark them explicitly.

For those cases, three idioms:

The rule: || true for the trivial case, if ! when you want to react, and set +e/set -e confined to the fewest possible lines. A set +e at the top of a long script is worse than never having written -e, because it gives a false sense of safety.

trap 'commands' SIGNAL [SIGNAL...] registers code that runs when a signal arrives. Its variants: trap - SIGNAL restores the default behavior, trap '' SIGNAL ignores it and trap -p lists the active traps.

Besides the real signals from 05-02, Bash defines four pseudo-signals:

Use single quotes in the trap's code. trap 'rm -f "$tmp"' EXIT is correct because $tmp is resolved when it fires; with double quotes it would expand when registering the trap and you would capture the wrong value.

Here we close what was promised in 05-01 and 05-02:

EXIT fires on exit, when the last line finishes, when set -e aborts and when a trapped signal arrives. Two details people forget: the local code=$? must be the first line of the function, because any command before it overwrites the value; and the final exit "$code" stops the cleanup from turning a failure into an apparent success. So that Ctrl-C also goes through it, add the signals: trap cleanup EXIT INT TERM HUP.

Every serious script has a function for aborting with a message. The convention: message to stderr, meaningful exit code.

And for the failures you did not anticipate, an ERR trap that says exactly where it happened:

The three variables that supply the context are the ones that turn a useless message into a diagnosis:

FUNCNAME is an array: ${FUNCNAME[1]} is whoever called the current function, and by walking it alongside BASH_LINENO you can print a complete stack trace. One caveat: for the ERR trap to be inherited inside functions and subshells you need set -E (or set -o errtrace); without it, functions do not fire it.

In 03-01 you saw exit N. Now the convention, which matters when another script or cron reads your result:

The ones in the 64-78 range come from BSD's sysexits.h; not everybody uses them, but they are a better convention than inventing numbers. And the 128 + N rule for signals explains at a glance the 137 (128+9, SIGKILL) and the 143 (128+15, SIGTERM). Reserve the codes above 125 and do not use values greater than 255: they are truncated modulo 256, so exit 256 is exit 0.

Network failures are transient by nature: retrying is correct, but retrying immediately and without a limit is an attack on your own server. The pattern is exponential backoff:

The waits are 1, 2, 4, 8 seconds. Note the combination with timeout from 05-02: without it, one hung attempt keeps you from ever reaching the next. The real-world use with curl and APIs arrives in 06-04 and 06-05.

bash -n should be a reflex before saving any script destined for cron: it spots the missing fi without running the rm. And set -x / set +x confine the trace to the suspect part, which is what makes it usable in a long script.

The trace goes to stderr prefixed with +. That prefix is the PS4 variable, and enriching it turns an illegible trace into a diagnosis:

The resulting trace looks like this —+ daily-report.sh:91:accumulate_day(): [[ 2026-08-03 == 2026-08-03 ]]—: every line states file, line and function. And BASH_XTRACEFD diverts it to a file so it does not mix with the report's output (descriptor 8 is from the family you will see in depth in 05-05). For its part, trap 'printf "line %d: total=%s\n" "$LINENO" "${total:-}" >&2' DEBUG inspects the state before every command, useful for hunting down where a variable changes.

When none of this is enough, there is still bisection: put an exit 99 halfway through the script, see whether the problem shows up, move the exit to the middle of the relevant half and repeat. With five or six runs you locate the guilty line in a thousand-line script. And to prevent rather than cure, shellcheck statically detects most of these faults before running anything: that is the subject of 08-05.

Four header lines and two traps: the script no longer lies. If the CSV cannot be read, it dies with 66 and a message; if something unexpected fails, it states file, line and function; and the temporary is deleted along all four possible exit routes.

• Believing set -e covers everything. Functions called from if lose it. Check by hand as well.
• trap with double quotes. The variables expand when the trap is registered, not when it fires.
• Not capturing $? on the first line of the EXIT handler, or forgetting the final exit "$code". A failure becomes an apparent success and cron never warns.
• trap ... ERR without set -E. Functions do not fire it and the diagnosis never appears.
• Using PIPESTATUS after another command. It is overwritten by each one; copy it immediately.
• exit 300. It is truncated modulo 256 and exits with 44. Stay between 0 and 125.
• Tip: write the cleanup trap on the same line where you create the temporary. If you leave it for "later", the early return you add a month from now will take the deletion down with it.

Exercise 1. Write a complete header for a script archive-history.sh that uses strict mode, creates a temporary directory, guarantees its deletion even on Ctrl-C, preserves the original exit code and reports the exact line on any unforeseen failure.

Exercise 2. The function count_errors() uses zgrep -c ERROR "$LOG" | tr -d ' '. Explain why under set -e without pipefail it can return a false number, and rewrite it so it distinguishes "zero errors" from "I could not read the file", exiting with 66 in the second case.

Exercise 3. Add to daily-report.sh a --debug option that enables an enriched trace (file, line and function) dumped to ~/veloz-ops/logs/trace-YYYY-MM-DD.log, without polluting either the report's output or its errors.

Solution 1.

-E is essential for the ERR trap to work inside functions. INT TERM HUP alongside EXIT may look redundant —a trapped signal ends up firing EXIT— but spelling them out guarantees the deletion even if someone redefines the default behavior.

Solution 2. Without pipefail, the pipeline's code is tr's, which is always 0: if zgrep could not open the file, the function returns the empty string as though it were zero errors. On top of that, zgrep -c returns 1 when there are zero matches, which under set -e would kill the script without there being a real error.

The [[ -r ]] guard separates the real failure (unreadable file → code 66) from the legitimate case (zero errors → prints 0). It is clearer than any acrobatics with PIPESTATUS, and the lesson is general: check the preconditions yourself instead of deducing them from ambiguous codes.

Solution 3.

BASH_XTRACEFD=8 diverts the trace to descriptor 8, so neither stdout (the report) nor stderr (the warnings) get contaminated: you can keep piping the report to a file while you read the trace in another terminal with tail -f. The >> preserves the traces from several runs on the same day.

Bash does not abort on its own, so robustness has to be requested explicitly. set -e kills the script at the first failure, with deliberate exceptions —conditionals, &&/||, negations, functions called from if— that you must know in order not to get complacent; set -u turns a typo into a visible error instead of a dangerous empty string; set -o pipefail fixes the exit code of pipelines, and PIPESTATUS says which link failed. The line set -Eeuo pipefail is the right starting point, but not a guarantee: you still have to mark with || true or if ! the commands whose failure is normal. trap completes the picture: EXIT gives the guaranteed cleanup that makes the mktemp + deletion pair safe, and ERR with $LINENO, BASH_SOURCE and FUNCNAME turns a mute error into a diagnosis with file, line and function. A die() with codes from the sysexits convention and a retry-with-backoff pattern round out the arsenal. And for investigating: bash -n validates syntax, set -x with an enriched PS4 and BASH_XTRACEFD produces a readable, isolated trace, and bisection with a temporary exit locates whatever resists.

There is a debt outstanding since Module 2. The toolkit validates paths, processes and codes, but when it has to validate text —a date --date 2026-08-03, an IP, an HTTP code, the structure of an app.log line— it still falls back on fragile comparisons with globs. In 05-04 real regular expressions finally arrive: the three flavors and which tool uses each one, the syntax from scratch, and Bash's =~ operator with BASH_REMATCH, which validates and extracts in a single step.