We closed Module 4 by saying that daily-report.sh still lacks robustness, and we promised to start right here: find, xargs, tar and safe temporary files. It is no accident that robustness begins with files. Almost everything an operations script gets wrong, it gets wrong when choosing which files to act on —it deletes too much, it skips a directory, it chokes on a name containing a space— or when writing to a temporary file that another process can clobber. In this lesson you will learn to select files with surgical precision, to act on them without a strange name breaking anything, and to create temporary and compressed files that do not turn into a security hole.

1. Why globbing is not enough
2. Anatomy of find
3. Selection criteria
4. Combining criteria: -a, -o, ! and parentheses
5. -maxdepth and -mindepth: why they come first
6. Actions: -print, -delete, -exec and -execdir
7. Names with spaces: -print0 and xargs -0
8. xargs in depth
9. Safe temporary files with mktemp
10. Archiving and compressing: tar, gzip, zcat, zgrep
11. Hard and symbolic links
12. rsync in two lines

In 02-05 you saw that *.log is expanded by the shell before the command starts, and that with shopt -s globstar even **/*.log walks subdirectories. So why do we need find at all?

That last point is what kills scripts in production: globbing builds a command line containing every name, and the kernel rejects it beyond roughly two megabytes. find processes files one at a time. The rule of thumb: globbing for a handful of known files in one directory; find to search by criteria or walk trees.

Three parts, always in this order: where to search, what to select and what to do with what it finds. If you omit the action, find assumes -print. If you omit the criteria, it shows everything under the path.

Two details that cause surprises from day one:

• The pattern '*.log' is quoted. Without quotes, the shell would expand it before launching find, using the files in the current directory, and find would receive something it did not expect. It is the same quoting principle as 03-06: here the pattern is for find, not for the shell.
• find walks recursively from the given path. find . can take minutes on a large tree.

Watch out for the difference between -name and -path: -name compares only the final name (the basename from 04-04), whereas -path compares the whole path exactly as find generates it, so its * do cross slashes.

The size and time criteria are what make find irreplaceable:

The sign matters a great deal: + is "more than", - is "less than", and no sign means exactly. A -mtime 30 means "modified on day 30 counting backwards", almost never what you want. And -mtime counts in 24-hour blocks, truncating: for fine-grained windows, use -mmin.

There is the purge of old reports the toolkit needs: every report older than a month.

When you put two criteria one after another, find joins them with an implicit AND. For everything else there are operators:

The parentheses are escaped (\\( and \\)) or quoted ('('), because to the shell unescaped parentheses mean a subshell. And they are essential as soon as you mix -o with another criterion, because -a binds tighter than -o: without parentheses, that last example would mean "the .csv files (of any date) or the .tsv files older than 90 days".

This mistake is especially serious combined with -delete. Always check with -print before replacing it with a destructive action.

-maxdepth 0 is the starting path itself; 1 is its direct children. -mindepth 1 excludes the starting path, handy for emptying a directory without deleting it.

These two options must come before any other criterion. Technically they are not criteria but global options: they affect the whole traversal, not one particular file. If you write find . -name '*.log' -maxdepth 1, find warns with warning: -maxdepth is a global option and applies it anyway, but it is a sign that you have not understood the ordering, and one day it will bite you with options that are less forgiving.

-delete is safer and faster than -exec rm {} \; because it does not spawn processes and has no trouble with strange names, but it must go last: find . -delete -name '*.txt' deletes the whole tree before looking at the name.

The difference between \; and + is the one that costs the most performance to ignore. The {} is the placeholder where find inserts the file name, and the terminator decides how they are grouped:

The \; is escaped for the same reason as the parentheses: unescaped, the ; ends the command as far as the shell is concerned.

-execdir is the safe variant —find /srv/veloz/data -name '*.tmp' -execdir rm -- {} \;—: it runs the command inside the directory where each file lives and passes it ./name instead of the full path. That prevents attacks in which someone renames a directory halfway through the traversal, and stops a file called -rf from being interpreted as an option.

This is the classic that separates people who copy recipes from people who understand the shell: find ~/veloz-ops/logs -name '*.txt' | xargs rm is broken.

find prints one name per line and xargs splits on whitespace. A file called report july.txt arrives as two arguments: xargs tries to delete report and july.txt. A name containing a newline is even worse. The canonical solution uses the one byte that cannot appear in a file name, the null:

-print0 separates with \0 instead of \n; -0 tells xargs to expect that separator. Never pipe find into xargs without that pair. If the command allows it, -exec ... + does the same thing without a pipe and without the risk.

xargs reads from standard input and builds command lines out of what it reads. Its useful options:

Without -r, if the find matches nothing, xargs runs gzip with no arguments and it sits waiting on standard input: the script hangs with no explanation. It is a real and frequent failure in scheduled tasks.

-I{} is for when the name does not go at the end of the command —xargs -0 -I{} cp -- {} /srv/veloz/backup/—, but it does not batch: one invocation per file, with the cost you already know. Use it only when the argument position demands it.

daily-report.sh needs an intermediate file to sort the table before printing it. The temptation is tmp=/tmp/report.$$, and it is wrong: $$ is the PID, which is neither secret nor unpredictable. /tmp is writable by anyone. Two classic attacks: the race condition (an attacker creates the file between your check and your write) and the symlink attack (they create /tmp/report.12345 pointing at ~/.ssh/authorized_keys, and your script writes there with your permissions). The solution is to delegate to the system:

mktemp creates the file atomically, with a random name and permissions 600 (700 for directories). There is no race window and nobody else can read it. The XXXXXX (six minimum) is where the random part is inserted.

The other half is missing: deleting it no matter what, including if the script dies halfway through. That is done with trap, and it is the opening line of 05-03: tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT. Remember the pair: whoever creates a temporary file schedules its cleanup on the very next line.

tar groups many files into one (.tar); gzip compresses a single file. Together they give the good old .tar.gz:

The letters: c create, x extract, t list, z pipe through gzip, f target file, -C change directory before acting. j uses bzip2 and J uses xz (slower, compresses more).

Always use relative paths. If you archive /srv/veloz/data/archive, tar warns with Removing leading '/' and, in old versions or with the -P option, extraction would write directly over /srv/veloz/data on any machine, trampling whatever was there. With relative paths you decide where to restore using -C. The correct pattern is the preceding cd (or tar -C /srv/veloz/data -czf ... archive/2026/07).

--exclude accepts globs and can be repeated. And for the rotated logs in /var/log/veloz you do not need to decompress anything:

zgrep over the rotated files is what lets daily-report.sh answer "how many 500 errors were there last week?" without using extra disk space.

In practice: symbolic for 95% of cases (pointing at the "latest report", switching the active version), hard for cheap copies within the same filesystem. readlink -f resolves the whole chain down to the real path, and it is the reliable way to know what something points at.

rsync -a --delete source/ target/ synchronizes trees by copying only what has changed, works over SSH and is the right tool for incremental backups. Its natural home is 07-03; for now just remember that it exists and that the trailing slash on the source changes the meaning.

• Forgetting the quotes in -name '*.log'. The shell expands the pattern first and find searches for something else.
• -delete before the criteria. It deletes the entire tree. Always put it last and test first with -print.
• find ... | xargs without -print0/-0. One space in a name and you delete what you should not have.
• xargs without -r. With empty input it runs the command with no arguments: a hang or a disaster.
• -o without parentheses. The precedence of -a silently changes the meaning of the expression.
• -mtime 30 with no sign. It means "exactly day 30", not "more than 30 days".
• tar with absolute paths. Restoring tramples the original system. Use cd or -C and relative paths.
• Tip: build destructive find commands in two steps. First with -print, you review the list, and only then do you change the action. In a script, save the list into a mktemp temporary file and act on that.

Exercise 1. Write a function purge_reports() that deletes from ~/veloz-ops/logs the report-*.txt files older than 30 days and the empty files of any age, reporting how many it deleted. It must work with names containing spaces.

Exercise 2. Archive last month's history (/srv/veloz/data/archive/YYYY/MM/) into ~/veloz-ops/logs/archive-YYYY-MM.tar.gz excluding the .tmp files, verify that the archive can be listed and only then delete the originals.

Exercise 3. Count how many requests with code 500 there are in all the access.log* files in /var/log/veloz, including the compressed ones, using a safe temporary file for the intermediate result.

Solution 1.

read -d '' reads up to the null byte, which is exactly what -print0 emits: the precise counterpart of the pattern from section 7. The parentheses group the condition "(old AND with that name) OR empty", and -maxdepth 1 stops the purge from sneaking into subdirectories. The -- before "$f" protects against names starting with a dash.

Solution 2.

The date -d 'last month' comes from 04-06. The order is non-negotiable: verify before deleting. The tar -tzf to /dev/null shows nothing but returns a non-zero code if the archive is corrupt, and the if exploits that exactly as in 03-04.

Solution 3.

zgrep handles plain files and .gz files alike, and -h suppresses the file-name prefix so that cut always sees the same column structure. The final rm is correct, but it only runs if the script makes it there alive: in 05-03 you will replace it with a trap that guarantees cleanup even in the face of a failure.

find is the search tool that globbing cannot replace: it selects by name, path, type, size, date, owner, permissions or emptiness, combines criteria with -a, -o, ! and escaped parentheses, bounds the traversal with -maxdepth/-mindepth —which come first because they are global options— and acts with -print, -delete, -exec ... \;, -exec ... + (thousands of times cheaper) or -execdir. When you do need a pipe, the pair -print0 with xargs -0 is mandatory, and -r, -n, -I{}, -P and -t round out the toolkit. mktemp and mktemp -d create atomic, private and unpredictable temporary files, instead of the /tmp/file.$$ that invites others to clobber you. tar with relative paths archives, and zcat/zgrep read the rotated logs without decompressing them. And links, hard or symbolic, give alternative names to what already exists.

With this, daily-report.sh now knows how to choose which files it works on and where it leaves its intermediates. But an operations script does not only touch files: it launches processes, and sometimes several at once. Is veloz-api alive before we ask it for data? Can I compute the four cities in parallel instead of one after another? What happens if the 6:00 report is still running when the 6:05 one starts? In 05-02 we get into process management: PIDs, ps, pgrep, background jobs, wait, signals, timeout and the locking with flock that stops two reports from stepping on each other.