At the end of 08-01 we replaced a while read loop with a single awk, claiming it was "faster". That is a comfortable and much repeated claim, but until there is a number next to it, it is not engineering, it is folklore. This lesson is about numbers: how you really measure a script's timings, what is actually expensive in Bash — and it is not what most people think — which optimizations have a measurable effect and which are superstition, and above all when to stop optimizing. At the end we will profile daily-report.sh over a CSV of 500,000 shipments and cut its run time from minutes to seconds, measuring before and after.

1. Measure first, optimize afterwards
2. The three ways to measure: time, SECONDS and date +%s%N
3. The dominant cost in Bash: creating processes
4. Builtins versus external commands
5. Useless pipelines and reading the file only once
6. The hidden cost of subshells
7. Accumulate the output and write once
8. while read, mapfile or awk: when to use each
9. Parallelizing for real, and when it does not help
10. LC_ALL=C, memory and the final criterion
11. Application: profiling daily-report.sh

There are three reasons not to optimize without measuring. The first is that intuition fails: almost everybody believes the loop is slow because it is a loop, when in reality it is slow because of the two cut calls inside it. The second is that optimizing costs readability, and you have just spent an entire lesson earning it; paying it back for a 2% improvement is a bad deal. The third is that 90% of the time is usually in 10% of the code: if you do not know which 10% that is, you will optimize the rest.

The method is always the same, four steps:

1. Establish the baseline. Measure the script as it stands, over representative data.
2. Find the hot spot. Measure by parts until you know which block takes the time.
3. Change one single thing and measure again under the same conditions.
4. Stop when it is enough. If the daily report already takes 4 seconds and runs once a day, getting it down to 2 helps nobody.

And a warning about "representative data": measuring with the 50-line test CSV tells you absolutely nothing. Many optimizations only show up beyond a certain size, and some — loading the file into memory — are good with 1,000 lines and catastrophic with ten million.

The three figures tell different stories and knowing how to read them is half the lesson:

The diagnostic combination is this: if user + sys is far below real, the script is waiting (disk, network, a sleep) and optimizing the code will not help at all. And if sys is high and comparable to user, as in the example, the culprit is almost always process creation: 24 seconds inside the kernel are not spent by a computation, they are spent by one fork per line.

Watch out for one detail: time is both a shell keyword and an external command (/usr/bin/time), with different outputs. The shell one is what you saw; the external one, with -v, adds peak memory used, which is very handy. To measure parts of a script, two already familiar tools:

SECONDS is free (it is internal) but only gives whole seconds: it works for long blocks. date +%s%N gives nanoseconds and works for short blocks, though it creates a process per measurement, so do not put it inside the loop you are measuring. And a lab rule: repeat the measurement three times and keep the median. The first run fills the disk cache and is always the slowest; comparing a first run against a third one is the most common way of "proving" a nonexistent improvement.

This is the idea of the lesson. In Bash, running an external command means the kernel duplicates the process (fork), loads the binary (exec), resolves its libraries and destroys it on exit. That costs on the order of 1 to 3 milliseconds. It looks like nothing, until you multiply it by the number of lines in a file.

Let's compare three ways of extracting the city (field 3) from each line of a CSV of 500,000 shipments:

The times measured over the same file and the same machine:

A million processes to read half a million lines. Version A is not "a bit slower": it is unusable. And notice that the difference between A and B is not the loop, which is the same one, but having taken the two processes out of it. This is the first operational rule: never run an external command inside a loop that iterates over lines. If you need one, the whole loop should probably be an awk.

A corollary of the above: when the shell can do something by itself, do it with the shell. type -a (01-04) tells you which is which.

A dose of honesty is in order here: outside a loop, this is irrelevant. Swapping a lone basename for ${f##*/} saves two milliseconds across the whole script and may hurt readability for anyone who is not fluent in expansions. The table matters inside loops, where the saving is multiplied by the number of iterations. Optimize the inside of loops and leave the outside readable.

Every | in a pipeline is one more process. Most of the long pipelines you see in production have stages that are unnecessary:

The useless cat case even has a name (useless use of cat). But the most expensive mistake is not counting processes: it is reading the same file several times. This pattern shows up in every reporting script:

The first version reads 500,000 lines four times; the second, once. In the real toolkit this brought the statistics block down from 3.2 s to 0.8 s. And the single-pass version is also more consistent: if the file changes while it runs, the four figures above may not add up with each other.

$( ) creates a subshell, which is a child process with all its memory copied. It costs the same as launching an external command, even if there is none inside:

The same happens with pipelines: cmd | while read ... runs the while in a subshell, which is why the variables you modify inside do not exist on exit — the classic problem we solved in 05-05 with done < <(cmd). There, process substitution is not just a matter of correctness: it also saves a process.

A useful trick for blocks that write a lot: instead of grouping with ( ... ) > output.txt, which creates a subshell, group with braces, which does not:

Every >> inside a loop opens the file, writes and closes it. With 500,000 iterations that is 500,000 opens:

All three produce the same file; the third is the fastest, the shortest and the one that uses the least memory, because the descriptor is opened once and the loop writes into the stream (05-05). Measured with 200,000 lines: 4.1 s for the first, 0.9 s for the second, 0.7 s for the third.

The practical rule, in one sentence: if inside the while you only filter, count or add up, your loop is a badly written awk. while read is justified when each line triggers actions awk cannot do well — firing a curl per shipment, calling shell functions, writing to different files — and even then it is worth asking whether awk can generate the list and the loop just act on it.

When the work is independent per item and each one really takes time — querying the API for each city, compressing each rotated log — parallelization is the biggest improvement available. With xargs -P (05-01) or with &/wait (05-02):

Two essential warnings. The first: if the bottleneck is the disk, parallelizing makes things worse. Eight processes reading at once from a mechanical disk generate random seeks and can take longer than a single one; on SSD the effect is smaller but the limit exists just the same. Look at time: if real is high but user + sys is low, you are waiting on I/O and more processes will not help. The second: parallel processes write interleaved; have each one write to its own file and consolidate at the end, as fleet.sh did in 07-06. A reasonable starting point for the degree of parallelism is -P "$(nproc)" for CPU-bound tasks (06-03), and considerably less for disk-bound ones.

An accelerator hardly anyone knows about: the locale. With es_ES.UTF-8, sort and grep apply collation rules and multibyte character classes; with LC_ALL=C they compare byte by byte.

Four times faster for one variable. The condition is that byte-by-byte ordering is good enough for you: for internal keys, identifiers or paths, yes; for a list of cities a human will read, no, because Ávila will end up in the wrong place. In the toolkit it is applied to intermediate sorts and the locale is left in place for the final presentation.

On memory, a simple rule: do not load into memory what you can traverse as a stream. mapfile -t lines < /var/log/veloz/access.log with a one-gigabyte log tries to allocate more than a gigabyte of RAM — Bash arrays have quite a lot of overhead per element — and can end with the process killed by the kernel's memory manager, in the small hours and with no explanation. Streams (while read, pipelines, awk) use constant memory regardless of the file size.

And the final criterion, the most important of all: knowing when the problem is no longer a Bash problem. Concrete signs that you have hit the limit:

• After applying everything above, the script still takes minutes.
• The bulk of the logic already lives inside a forty-line awk with its own data structure.
• You need structures Bash does not have: serious floating-point numbers, two-dimensional arrays, nested JSON that jq no longer covers comfortably.
• You need to join two large files by a key, sort by several criteria or query history: that is a database, even if it is SQLite.
• The script goes past 500 lines and its business logic matters more than its system logic.

Bash is excellent at gluing system commands together; it is a bad language for intensive computation. Recognizing that in time is a technical decision, not a defeat: daily-report.sh is still Bash, but if tomorrow we have to join shipments with billing and three years of history, that piece gets written in Python or SQL and Bash orchestrates it.

We generate a representative CSV and measure the baseline:

sys almost equal to user: the diagnosis from section 3 is immediate, this is processes. We instrument the script with SECONDS around each block to locate the hot spot:

Inside the 188-second block there was a while IFS=, read with a cut and a date -d per line (a million processes) plus four later grep -c calls over the whole file. The three changes applied, in this order and measuring each one:

From 192 seconds to 3.9: forty-nine times faster without changing a single line of the business logic, only where each thing runs. And here step 4 of the method applies: we stop. There are 3.9 seconds left, 2 of which are the email sending, which depends on the network. Further optimizing a script that runs once a day at 06:30 brings nothing to anybody.

• Optimizing without a baseline. Without the starting number you cannot prove the improvement or detect that you have made things worse. Write down the initial time before touching anything.
• Measuring only once. The first run pays for the cold disk cache. Run three times and use the median; for serious comparisons, hyperfine automates exactly this.
• Micro-optimizing outside loops. Swapping basename for ${f##*/} on a line that runs once saves nothing and can subtract clarity. The inside of loops is where you win.
• Confusing "fewer lines" with "faster". A six-stage pipeline is short to write and creates six processes; a three-line awk creates one.
• Parallelizing disk-bound work. If real is high but the CPU is idle, more processes only add contention.
• Tip: set -x with PS4 works as a poor man's profiler. With PS4='+ $(date +%s.%N) ' each trace line (05-03) is stamped with its instant and you can see the time jumps.
• Tip: the most profitable optimization is usually not doing the work. Before speeding up the processing of the whole history, ask yourself whether you need to process more than the current day.

Exercise 1. This block takes 2 minutes with 300,000 lines. Identify the three performance problems and rewrite it in a single pass.

Exercise 2. Write a veloz_measure function that runs a command three times, prints the real time of each run in milliseconds and the median, using date +%s%N.

Exercise 3. A script takes real 5m10s, user 0m12s, sys 0m8s. Where is the time and what optimization makes sense? And if it were real 5m10s, user 2m40s, sys 2m20s, what would you look for?

Solution 1.

The three problems. (1) Quadratic complexity: for each of the 300,000 identifiers a grep scans the entire file; that is 9×10¹⁰ line comparisons. This is the serious defect, and it disappears when you traverse the file once instead of searching inside it. (2) Processes inside the loop: grep, echo, cut, expr and bc are five processes per iteration, a million and a half in total. (3) for over a command substitution, which besides creating the subshell builds in memory a string with the 300,000 identifiers and splits it on spaces. The awk version does one pass, one process and adds up in floating point natively, with no bc (06-01).

Solution 2.

We use "$@" with exactly the right quoting so the command and its arguments arrive intact (03-05), the output is discarded so we do not measure the terminal, and the median of three is the middle element of the sorted array, which is more robust than the mean against an anomalous run.

Solution 3. In the first case the process only consumes 20 seconds of CPU out of the 310 on the clock: it is waiting, almost certainly on network or disk. Optimizing the code will change nothing; you have to find the wait (a curl with no --max-time, an inherited sleep, an ssh without ControlMaster, a read over a network mount) and attack it with parallelism, caching or time limits. In the second case the CPU is indeed busy, and the split almost evenly between user and sys gives away massive process creation: you have to look for external commands inside loops and turn them into expansions or into a single-pass awk.

Optimizing in Bash starts and ends in the same place: measuring. time gives the three figures that guide all the work — real is what the user suffers, user is computation and sys is kernel, so a high sys means processes and a real far above user + sys means waiting; SECONDS brackets long blocks inside the script and date +%s%N short ones, always repeating the measurement and keeping the median over representative data. The dominant cost is creating processes: 1-3 ms each, insignificant on their own and devastating multiplied by half a million lines, as the jump from 21 minutes to 12 seconds shows by merely taking echo and cut out of the loop, and from there to 0.9 s with a single awk. From that the practical rules follow: use builtins inside loops ($(( )) instead of expr, ${f##*/} instead of basename, ${#s} instead of wc -c) and do not bother outside; eliminate the useless stages of pipelines (grep -c instead of grep | wc -l, an awk that filters instead of grep | awk); read the file only once, computing all the aggregates in a single pass; avoid unnecessary subshells, grouping with { } instead of ( ) and using done < <(cmd); and redirect the whole loop instead of accumulating >>. Between while read, mapfile and awk, the rule is that if inside the loop you only filter, count or add up, that loop should be an awk; mapfile only for small files that have to be traversed several times. Parallelizing with xargs -P or &/wait is the biggest win when the work is independent and heavy, but it does not help if the bottleneck is the disk. LC_ALL=C multiplies sort's speed by four when byte-by-byte ordering is good enough, and no one-gigabyte log should ever go into an array. The real case closes the argument: daily-report.sh went from 3 min 12 s to 3.9 s — forty-nine times — without touching the business logic, and then we stopped, because optimizing further a script that runs once a day helps nobody.

The toolkit is already readable and fast. The most uncomfortable of the three questions that opened the module remains: is it secure? These scripts run privileged tasks, store SSH keys, read configuration files with credentials and process customers' personal data. Lesson 08-03 audits them from top to bottom: command injection and why eval is a bomb, input validation with an allowlist, secret management, safe temporary files, least privilege, race conditions and handling personal data, with a checklist applied to the complete toolkit.