Closing 05-04 we set out the problem: daily-report.sh already validates and extracts text precisely, but it does not know how to direct its own output. It prints with loose printf calls, and to save the report to a file you have to redirect the whole script from outside —which takes the error messages down with it—. It cannot write to the screen and to a file at once, nor keep a log open while it works, nor compose a twenty-line template without twenty printf calls. In 02-04 you saw >, >>, 2>&1 and pipes as recipes that work. This lesson teaches you the machinery underneath, and with it redirection stops being a set of memorized tricks and becomes something you can design.

1. Descriptors are numbers
2. Opening, reading and closing your own descriptors
3. Reading two files at once
4. Duplicating descriptors and the order of redirection
5. Saving and restoring standard output
6. Redirecting blocks, functions and loops
7. Here-documents
8. Here-strings
9. Process substitution
10. tee with several destinations
11. The special files in /dev
12. The report with a template

A file descriptor is an integer a process uses to refer to an open channel. There is nothing magical about it: it is an index into a table the kernel keeps per process. Every process starts with three: 0 (stdin, the keyboard), 1 (stdout, the screen) and 2 (stderr, also the screen).

What people rarely mention is that 3 through 9 are free for you to use, and that all the redirection syntax you already know consists of operations on that table:

The number goes attached to the operator: in command 2> errors.log the 2 glued to the > means "descriptor 2 goes to this file", whereas command 2 > f (with a space) passes 2 as an argument and redirects stdout, a silent bug. To see it with your own eyes, ls -l /proc/$$/fd lists your shell's open descriptors right now, all three pointing at your terminal (/dev/pts/3 or similar).

exec, with no command after it, applies the redirections to the current shell instead of to a specific command. That is the mechanism for opening long-lived channels:

You read with read -u N and write by redirecting to >&N:

Closing (exec 3<&- for reading, exec 4>&- for writing) is not optional in a long script: every process has a limit on open descriptors, and one left open for writing can hold data in the buffer.

Why bother, if done < file already works? For two reasons. First, efficiency: opening a file has a cost, and if on each iteration of an outer loop you reopen the same file, you pay it every time. With exec 3< you open it once and the descriptor keeps the read position between reads —that is why the header read and the loop that follows do not read the same line—. Second, and more importantly: it lets you read two files at once.

A normal while read loop consumes stdin. If you need to walk two files in parallel —comparing today's report with yesterday's, line by line—, with a single channel it is impossible. With your own descriptors it is straightforward:

read -r yesterday <&3 redirects that particular read's input to descriptor 3; it is the alternative form to read -u 3, and both are equivalent. The && makes the loop end when either of the two files runs out.

N>&M means "make descriptor N point where M points now". It is a copy of the destination, not a permanent link, and that distinction explains the classic everybody memorizes without understanding:

Redirections are processed left to right, and each one copies the destination the other descriptor has at that instant:

In the second case, when 2>&1 runs, descriptor 1 still points at the screen; the fact that 1 changes afterwards does not drag 2 along. The mnemonic rule: 2>&1 must always come after the stdout redirection. And the &> file shortcut from 02-04 is exactly > file 2>&1, written in a way that does not allow this mistake. Duplication in the other direction is just as useful: printf 'Warning\n' >&2 writes to stderr, and exec 3>&1 saves where stdout points now.

Combining the above yields a very powerful pattern: redirect everything the script prints from a certain point onward, and then go back.

Without the preceding exec 3>&1, there would be no way back: the original reference to the screen would have been lost. This pattern is what lets a function write to a file without its author having to know it, and it is the basis of the logging systems you will see in 07-04.

You do not have to go all the way to a global exec. Any grouping accepts a redirection at the end, which applies to all its contents:

The difference between { } and ( ) is the one from Module 1: braces run in the current shell (the variables you assign persist) and parentheses create a subshell (they do not persist, but the cd does not affect you). Syntax: { needs a space after it and a ; or newline before }.

A here-document feeds a command's standard input with a block of text written in the script itself:

The delimiter (EOF by convention, it can be any word) marks the end, and it must appear alone on its line, with no spaces in front. That is failure number one with here-documents: an indented delimiter does not close the block and the script dies with unexpected end of file.

There are three variants and the difference matters:

Two decisions in that example. The delimiter in single quotes makes the text literal: without them, a $1 or a $HOME in the help would be expanded. And <<- lets you indent the block to the level of the code, stripping the leading tabs when printing —only tabs, not spaces, which is why the example is written with real tabs—.

A here-document works with any command that reads from stdin, not just with cat: bc -l << EOF followed by scale=2 and the expression is a perfectly normal use too.

When the text fits on one line, <<< avoids the whole block. You have already used it in 04-06 with bc:

A here-string turns a variable into a command's standard input, and that solves the pipeline subshell problem you saw in Module 4: echo "$x" | read var does not work because the read runs in a subshell, whereas read var <<< "$x" does assign, because there is no pipe and therefore no subshell.

Careful: <<< always adds a trailing newline, something you notice when comparing lengths (wc -c <<< "abc" gives 4).

<(command) and >(command) are the most powerful construct in the lesson: they turn a command's output (or input) into a file name. The canonical example is diff <(sort yesterday.txt) <(sort today.txt).

This cannot be done with pipes, and the reason is structural: diff needs two files, and a pipe only feeds one stdin. Bash solves it by creating a special file for each substitution and passing its path to the command —echo <(echo hi) prints /dev/fd/63, which is literally what the command receives—.

The most frequent use in scripts is the one that already appeared in Module 4, and now you understand why it works:

In the first case, the pipe puts the loop in a subshell and its variables die with it. In the second, < <(...) is a redirection from a file that happens to be a command's output: the loop runs in the main shell and its variables survive. The space between < and <( is mandatory; without it, Bash reads << and waits for a here-document.

>(command) works the other way around, feeding a command's input: tar -czf - /srv/veloz/data | tee >(sha256sum > backup.sha256) > backup.tar.gz archives and computes the checksum in a single pass.

tee (02-04) duplicates standard input towards stdout and towards the files you name. With process substitution, those destinations can be commands:

The report is saved to a file, counted in parallel and shown formatted on the screen, all in one pass. And if you need a whole block to be seen and saved, piping it is enough: { printf 'REPORT %s\n' "$(date +%F)"; cities_table; } | tee ~/veloz-ops/logs/report.txt.

With tee -a it appends instead of truncating, and something important for the strict mode of 05-03: under pipefail, that pipeline's exit code is the worst of the three links, so a failure writing the file is indeed detected.

/dev/stdin is useful for commands that demand a file name: awk -f program.awk /dev/stdin. And /dev/tty solves a real case: asking the user something even when the script's output is redirected to a file, writing with printf 'Continue? [y/N] ' > /dev/tty and reading with read -r answer < /dev/tty, which takes from the keyboard even when stdin comes from a pipe.

/dev/tcp/host/port does not exist on disk: it is a Bash invention that lets you speak TCP without nc or curl. You will see it in depth in 06-04; for now just remember that (echo > /dev/tcp/localhost/8080) 2>/dev/null is a one-line port check.

Everything together, applied to the backbone script:

The template lives in a here-document with expansion, so the report's text reads exactly as it will come out, without the fog of twenty printf calls. The { } block groups the header, the table and the footer into a single stream, and tee delivers it simultaneously to the file and to the screen —what the script could not do at the start of the lesson—. The log_info/log_error messages keep going to stderr and do not contaminate the report.

• 2>&1 before > file. stderr stays on the screen. Always after, or use &>.
• An indented here-document delimiter. It must sit flush against the margin, except with <<- and tabs.
• Using << EOF for help text containing $. It gets expanded and comes out empty. Use << 'EOF'.
• Writing <<(command) with no space. Bash reads it as a here-document. It is < <(command).
• Forgetting to close descriptors, or using echo "$x" | read var: the subshell takes the variable with it. Use read var <<< "$x".
• Redirecting to a file that is also read on the same line. sort f > f empties the file: the shell truncates it before sort opens it. Use an mktemp temporary (05-01) or sort -o f f.
• Tip: exec 3>&1 before any global redirection. It is the only way back, and it costs one line.

Exercise 1. Write a function compare_reports() that takes two files and shows the lines that differ, giving the line number, reading both at once with your own descriptors and without using diff.

Exercise 2. Write report_header() that generates with a here-document a header with date, server and user, and a function usage() with a literal indented here-document that expands nothing.

Exercise 3. Make daily-report.sh write its report simultaneously to the screen and to ~/veloz-ops/logs/report-YYYY-MM-DD.txt, save separately only the lines with issues into issues-YYYY-MM-DD.txt and keep the log messages out of both files.

Solution 1.

A single loop walks two files because each read has its own channel. The last line takes advantage of (( )) setting $? (04-06) so the function returns a useful code without an explicit if.

Solution 2.

The first one uses << END without quotes because it needs $(date), $(hostname) and $USER to be expanded. The second uses <<- 'END' because it wants the literal text —if VELOZ_CSV were not in single quotes, the help would show its value instead of its name— and the dash lets you indent the block with tabs to the level of the code.

Solution 3.

The { } block produces a single stream; the first tee saves it whole and keeps passing it on; the second diverts it to a grep through process substitution, which filters into the second file while the main stream carries on to the screen. Since log_info writes to stderr (a decision we made in 04-02), its messages do not enter the pipeline and therefore do not appear in either file: that is the real reason for that convention.

Descriptors are numbers in a process's table of open channels: 0, 1 and 2 come given and 3 through 9 are yours. exec 3< f, exec 4> f and exec 6<> f open them, read -u 3 or read <&3 read from them, exec 3<&- closes them, and their great advantage is opening once what is read many times —and being able to walk two files at once—. N>&M duplicates a destination at the instant it runs, and from that comes the definitive explanation of why > f 2>&1 sends everything to the file while 2>&1 > f leaves stderr on the screen. Saving stdout in 3 and restoring it later lets you redirect a whole stretch; { ...; } > f and done < f redirect blocks, functions and loops without touching the global shell. Here-documents give readable templates with expansion (<<EOF), literal text (<<'EOF') and tab indentation (<<-EOF); here-strings <<< push a variable into stdin without a subshell. And process substitution turns commands into files: diff <(a) <(b) has no equivalent with pipes, and done < <(command) is the correct way for a loop to keep its variables.

With this, daily-report.sh is a complete program: robust, verbose when asked, precise when validating and able to direct its output exactly. And it is also a file of several hundred lines where log_info, die, validate_env, percentage and half a dozen more functions coexist that have nothing to do with reports and that any other script in the toolkit would need. Copying and pasting them into the next script would be the beginning of the end. In 05-06, the module's milestone, we extract those functions into ~/veloz-ops/lib/common.sh: source versus executing, how to locate the library without relative paths betraying you, include guards, naming conventions, configuration files with their precedence order and the definitive structure of the toolkit.