Module 5 closed with the first gap in the toolkit written down: every time column arithmetic has to be done over shipments.csv, daily-report.sh chains cut, sort and uniq, and rereads the file once for every figure it wants. Summing amounts by city, counting by status and computing the average per courier are today three separate pipelines and three reads from disk. awk solves all three in a single pass, because it is not just another command: it is a small language designed for text organized into records and fields. This lesson introduces it as such.

1. Why awk is not just another command
2. The execution model: pattern { action }, BEGIN and END
3. Records, fields and built-in variables
4. Input and output separators
5. Patterns
6. Actions: print, printf, conditionals and loops
7. User variables and their automatic initialization
8. Associative arrays: aggregating in a single pass
9. Walking and sorting the result
10. Built-in functions
11. Free floating point and Bash values with -v
12. Long programs, .awk files and flavors
13. Application: the new core of daily-report.sh

grep decides whether a line is of interest, cut extracts a column, sort and uniq -c count. Each does one thing, which is why they get chained. awk does all four at once because it has what they lack: memory between lines (variables and arrays that survive from one record to the next) and arithmetic.

That is the underlying difference: a pipeline is a stateless flow; an awk program is a loop with state. As soon as the question stops being "which lines?" and becomes "how much does this add up to, grouped by that?", the pipeline grows and awk stays just as short.

An awk program is a list of rules. For each line of the input, awk walks all the rules in order: if the pattern holds, it runs the action. In awk '/ERROR/ { print $0 }' /var/log/veloz/app.log, the pattern is /ERROR/ and the action { print $0 }: awk reads a line, checks, prints if appropriate and moves on to the next. There is no loop written down, the loop is implicit, and that is the first thing to internalize. Both parts are optional:

Two special patterns do not depend on any line: BEGIN { } runs once before reading anything (set up separators, print headers) and END { } once after the last line, which is where the totals are dumped.

There you already have the pattern that dominates the lesson: accumulate during the walk, print in END. The program always goes inside single quotes: it contains $1, $NF… and with double quotes Bash would expand them to the script's arguments (03-06) before awk ever saw them.

awk splits the input into records (lines, by default) and each record into fields (spaces or tabs, by default). $0 is the whole record, $1, $2… the fields counting from 1, NF the number of fields and therefore $NF the last one and $(NF-1) the second to last. The parentheses are mandatory: $NF-1 means "the last field, minus one", a subtraction.

$NF is gold when a line has a variable number of fields: in access.log the byte count is always last, however many words the request has, so awk '{ s += $NF } END { printf "%.2f MB\n", s/1048576 }' /var/log/veloz/access.log works without counting columns.

NR and FNR differ only with several files: on starting the second one, FNR goes back to 1 and NR keeps climbing. Hence the idiom for skipping CSV headers: awk -F, 'FNR>1' shipments.csv skips every header, whereas NR>1 would skip only the first file's. With archive/2026/*/shipments.csv, that difference is a silent data error.

For the CSV you have to tell awk the separator is a comma, and there are two equivalent ways: the option awk -F, '{ print $3 }' shipments.csv and the assignment awk 'BEGIN { FS="," } { print $3 }' shipments.csv. -F, is the usual one; FS in BEGIN is used when the separator is complex, because it accepts a regular expression: FS="[,;]" accepts a comma or a semicolon, and FS="[[:space:]]+" collapses multiple spaces.

The comma in print $3, $6 means "separate them with OFS". Without the comma, print $3 $6 concatenates: Valencia34.50. It is confusion number one with awk. Besides, OFS only applies when awk rebuilds the record; if you print $0 untouched it comes out as it was, and the trick to force it is to assign a field to itself ({ $1=$1; print }).

The pattern can be any expression awk evaluates as true:

That line replaces a grep | grep | cut or a seven-line while read. awk decides on its own whether to compare as a number or as text: $6 > 50 is numeric because 50 is; $3 == "Madrid" is textual.

Inside { } there is room for a complete language with C-like syntax: assignments, if/else, for and while.

printf works like the Bash one you saw in 04-04 (%-10s aligns text to the left in 10 columns, %8.2f a number with 2 decimals), with one important difference: you always have to write the \n yourself, because it does not add one; print does. The C-style for is what you use to walk a record's fields: for (i=1; i<=NF; i++) if ($i == "") print FILENAME": line "FNR" field "i detects empty fields in the CSV.

In awk variables are not declared: they are used. And what saves the most code is that a new variable is worth 0 if you treat it as a number and the empty string if you treat it as text.

In awk -F, 'FNR>1 { total += $6; n++ } END { print n, total, total/n }' shipments.csv, total and n do not exist before the first line and yet total += $6 works: awk creates them holding 0, whereas in Bash you would have to write total=0; n=0 before the loop. The flip side: a misspelled name is not an error, it is worth 0. If you accumulate into totl and print total, the result will be 0 with no complaint at all.

This is the reason for the lesson. awk's arrays are always associative —the index is a string, like those in 04-03— and they also create themselves:

Read it slowly: for each record, take field 3 (the city), use it as the index and add field 6 to it. There is no need to know in advance which cities there are, nor to sort, nor to walk twice. The structure is always { accumulator[key] += value } during the walk and END { for (k in accumulator) ... } at the end. And since the rules are independent, the three questions from the opening fit in one program and one single read:

An array can also be indexed by the combination of two fields, and that gives you cross tables for free: cell[$3, $5] += $6 sums amounts by city and status at the same time.

for (k in array) guarantees no order at all: awk walks its hash table as it sees fit. If you need an order, the portable way is to print and sort through a pipe with the sort from 02-02:

Notice that the number is printed first, so that sort -rn sorts by it with no odd options. GNU awk has PROCINFO["sorted_in"] and asorti(), but they are not portable; the pipe is.

Combined with arrays they give aggregations that in Bash would cost a whole loop: awk -F, 'FNR>1 { month[substr($2,1,7)] += $6 } END { for (m in month) print m, month[m] }' shipments.csv | sort bills by month by extracting the YYYY-MM out of the date.

In 04-06 it became clear that $(( )) only does integers and that for decimals you had to go out to bc -l. In awk all arithmetic is floating point, with nothing to install:

This makes awk a script's natural calculator: pct=$(awk -v a="$ok" -v b="$tot" 'BEGIN { printf "%.1f", 100*a/b }') replaces bc with no pipe. Watch out for %%: in printf, a literal percent sign is written doubled.

That -v is the correct way to get a Bash value into the program, since single quotes prevent any expansion. The temptation is to write awk -F, "\$3 == \"$city\" { n++ }", interpolating the variable. Do not do it, for two reasons. First, it breaks on anything: a value with spaces, quotes or backslashes wrecks the syntax. Second, it is code injection: whatever is in the variable becomes program, so if it comes from an argument or a file, whoever controls that value controls what awk runs —the same risk as eval (05-06), covered in depth in 08-03—. With -v c="$city" the value goes in as data. Two nuances: -v processes escape sequences, and its variables are already available in BEGIN, which is not the case with the awk '...' var=value file syntax.

A program can take up several lines inside the same single quotes, with normal indentation. When it goes beyond fifteen lines or so, or gets reused, it is pulled out into a file and invoked with awk -f ~/veloz-ops/lib/summary.awk shipments.csv:

An .awk file accepts comments with #, is versioned in Git and is far more readable than a lump between quotes: it is to awk what lib/common.sh is to Bash.

awk is a POSIX standard with several implementations: on Ubuntu 24.04 /usr/bin/awk is usually mawk (fast, pure POSIX), on Fedora gawk (with extensions such as gensub(), asorti(), RS as a regex or --csv) and on macOS/BSD the original awk. Everything in this lesson is POSIX and works on all three; if you use a gawk extension, invoke gawk explicitly so that the failure on another machine is "gawk is not installed" and not silently different output. One last warning: awk splits on plain commas, so a quoted field containing a comma ("Madrid, downtown") breaks the count. shipments.csv has no such case; if it did, the answer is gawk --csv, not a more complicated regex.

At the close of Module 5, the calculation was three pipelines, three reads of the file and no decimals: tail -n +2 "$CSV" | wc -l for the total, cut -d, -f5 "$CSV" | grep -c '^delivered$' for the delivered ones and cut -d, -f3 "$CSV" | tail -n +2 | sort | uniq -c | sort -rn for the breakdown by city. This is how it looks now as a function in lib/report.sh, with a single pass and a city-by-status cross table that was not viable before:

Three details deserve an explanation. SUBSEP is the separator awk uses internally for compound indexes —cell[$3, $5] and cell[$3 SUBSEP $5] are the same thing— and it is a character that will never show up in the data, safer than inventing a "|". The + 0 forces a non-existent cell to print as 0 and not as an empty string. And exit 65 inside END terminates awk with that code, which set -euo pipefail (05-03) turns into a script failure; 65 is EX_DATAERR, the conventional one for incorrect input data.

The Bash consumer reads that output with the loop from 04-01, without touching the CSV again:

The process substitution < <( ) is the one from 05-05: it keeps the loop in the current shell so the variables it assigns survive. The division of responsibilities is clear: awk computes, Bash orchestrates and presents.

• Double quotes around the program. awk "{ print $1 }" prints empty lines because Bash expanded $1 first. Program in single quotes; values go in with -v.
• Forgetting the comma in print. print $3, $6 separates with OFS; print $3 $6 concatenates.
• Confusing $NF with NF. NF is the number of fields; $NF is the content of the last one. The second to last needs parentheses: $(NF-1).
• Using NR>1 with several files. It skips only the very first header; with wildcards or archive/, use FNR>1.
• Expecting order from for (k in array). There is none: if it matters, sort with a pipe to sort.
• printf without \n. Unlike print, it does not add one and all the output comes out glued together.
• Comparing text that looks like a number. A field 007 compares as 7 against a number and as text against a string; force the type with $1+0 or $1 "" when it matters.
• Tip: awk is not always the answer. To just filter lines, grep is faster and clearer; awk wins when there are fields, arithmetic or memory between lines. And LC_ALL=C awk ... speeds up processing large ASCII files (we will see this in 06-03).

Exercise 1. With a single awk command over /srv/veloz/data/shipments.csv, print for each courier their name, number of shipments, total amount and average amount, sorted from highest to lowest total amount and in aligned columns.

Exercise 2. Over /var/log/veloz/access.log (the HTTP code is the second-to-last field and the bytes the last one), obtain with a single awk the number of requests and the megabytes served per response code, flagging the 5xx ones with <-- ATTENTION.

Exercise 3. Write a function veloz_average_amount for lib/common.sh that takes the CSV and a city, returns that city's average amount with two decimals passing the city in safely, and exits with code 1 if that city has no shipments.

Solution 1.

Two arrays with the same index (n counts, s sums) are the pattern for computing averages, and the division is done in END, never during the walk. The ordering is resolved outside with sort -k3 -rn on the total-amount column, because for (r in n) guarantees none.

Solution 2.

$(NF-1) and $NF avoid counting fields by hand in a format where the request has a variable length. The ternary operator condition ? a : b exists in awk just as in C and saves a four-line if/else; c >= 500 works because awk treats c as a number when comparing it against one.

Solution 3.

The city goes in with -v c="$city", so a value with quotes or spaces is harmless data and not part of the program. The exit 1 inside END propagates the failure to the function's exit code —awk is just another command and $? works as with any other (03-01)—, so the caller can write average=$(veloz_average_amount "$CSV" Madrid) || veloz_log_error "no data". Without the if (n == 0), the division would give an error or a nan depending on the implementation.

awk is a language with an implicit loop: for each record it walks its pattern { action } rules, with BEGIN and END as the two moments that depend on no line. It slices each record into fields reachable as $1, $NF or $(NF-1) and describes its position with NR, FNR, NF and FILENAME; -F, or FS set how it splits the input and OFS how it joins it on the way out. Patterns range from a regex to comparisons on fields, ranges and logical combinations, and actions include print, printf, conditionals and loops. But what justifies the lesson are two traits no pipeline has: variables that initialize themselves to 0 and associative arrays, which turn accumulator[key] += value followed by for (k in accumulator) in END into the universal aggregation pattern —sum by city, count by status, average per courier and even cross tables with SUBSEP, all in one read and with free floating point—. Bash values go in with -v, never interpolated, and long programs are pulled out into an .awk file with -f.

daily-report.sh no longer rereads the CSV three times: awk computes and Bash orchestrates. But awk reads and summarizes; it is not the tool for rewriting text while preserving its shape. When the IPs in access.log have to be anonymized before sharing it, when the separators of a badly exported file have to be normalized or when a key in veloz-ops.conf has to change without opening an editor, what you need is a stream editor. That is sed, and it is the next lesson (06-02): the same regular expressions you already command, but applied to transforming instead of filtering.