In 07-03 backup.sh was born: it copied /srv/veloz/data with rsync --link-dest, compressed, verified with sha256sum and applied a simple retention. It was enough not to lose the day's data, but not enough to sleep soundly. This project takes it to its definitive, trustworthy version: configurable profiles without touching code, a verifiable manifest, retention with promotion to weekly and monthly, remote copy, optional encryption, safe deletion and —most important— an automatic restore test that runs on its own every week. The principle that orders the whole project: an unverified backup does not exist.

1. Inventory: what gets backed up, with what RPO and RTO
2. The 3-2-1 rule applied to Veloz Envíos
3. Design: profiles in files, not in the code
4. Phase 1: the incremental copy that restores as a full one
5. Phase 2: manifest and verification
6. Phase 3: retention with promotion
7. Phase 4: restore and its automatic test
8. Remote copy and encryption
9. Disk space: check before starting
10. Observability and exit codes
11. Going to production and the runbook

The inventory is written before the code, and reviewed every quarter:

RPO is how much data we accept losing; RTO, how long we take to come back. Those two columns are what decide the frequency and the design, and the last row is the most important: backing up the whole system multiplies the cost and improves neither number.

Three copies, two media, one off-site (07-03):

Copy 2 on the same disk as the data does not satisfy the rule on its own —a disk that dies takes both with it— but it is the one that gives a one-hour RTO, because restoring from local is instantaneous. Copy 3 covers the death of the server; the off-site one covers fire and malicious deletion.

The mistake in 07-03 was having the paths inside the script: adding a set meant editing code, and editing code demands review and deployment. The definitive version reads profiles from etc/backup.d/*.conf:

The numeric prefix fixes the order. They are loaded in a loop, each one in a subshell so that one profile's variables do not contaminate the next (01-04):

set -a marks everything defined afterwards for export, and the subshell isolates. The alternative —loading everything into the same scope— causes the classic bug: profile 20 inherits KEEP_MONTHLY from profile 10 because it forgot to define it. With the subshell, each profile starts from the script's default values, which is the 05-06 precedence: default < file < environment < options.

Four decisions that are worth the whole project. --link-dest (07-03) hard-links whatever has not changed: today's backup takes up what the new files take up, but it contains the complete tree, so restoring is copying a directory and not rebuilding a chain —with the requirement that the source and destination of the links live on the same filesystem—. And link=() is an array (04-03), not a string: on the first day there is no previous copy, the array stays empty and the option disappears from the command; if it were an empty quoted string, rsync would receive an empty argument and fail.

${DRY_RUN:+--dry-run} (03-06) adds the option only if the variable has a value. Dry-run mode is mandatory here because there is a --delete involved: before the first real run of a new profile it is always run with --dry-run. And the .partial suffix with a final mv is the atomic write of 08-03: while rsync works, the directory has a name that last_good ignores; only the mv —atomic within the same filesystem— turns it into a valid backup. If the server shuts down halfway, what is left is identifiable garbage, never an incomplete backup that looks good.

A directory with files proves nothing. The manifest is the proof:

Three things are verified, from the cheapest to the most expensive: that the manifest exists, that the count matches and that the checksums of the key files add up. Checksums are only computed for VERIFY_GLOB (the CSVs) and not for the 40 GB of archive: verifying everything every night would cost half an hour and adds nothing, because the files linked with --link-dest are the very same data already verified yesterday. sha256sum -c --quiet returns a non-zero code if something does not add up, and that is enough to mark the profile as failed.

Retention is not "delete the old stuff", it is deciding which copy becomes the weekly one and which the monthly one before it expires as a daily.

Promotion uses hard links (cp -al, inside link_if_missing), not copies: the weekly of August 3rd shares data with the daily of the same day, so promoting costs zero bytes, and when the daily is deleted, the weekly will still contain the complete data. It is the same property of --link-dest applied to retention. There are also two safeguards from 08-03. The first, ${dir:?} and ${copies[i]:?}: if a configuration error left dir empty, rm -rf "/${copies[i]}" would delete starting from the root; with :? the script dies before running anything. The second, the warning when there are many more copies than expected: it means the script has gone days without pruning and something is wrong; deleting fifteen directories at once without warning is exactly what we do not want it doing on its own.

The restore subcommand never writes over the original:

Verification happens before restoring: restoring a corrupt copy over good data turns an incident into a disaster. And the default destination is a temporary directory, so a deliberate act is needed to write into /srv. The automatic restore test is the piece that separates this project from a simple copy script:

It runs every Sunday with its own timer and compares the recovered CSV with the original using cmp -s. It can give a false negative if the file changed between the backup and the test; that is why it is compared against the file as it was —or the false positive is accepted and investigated— but the test is never disabled just to make it stop warning. An unverified backup does not exist; one that is verified but never restored does not either.

BatchMode=yes (07-06) makes SSH fail instead of waiting for a password nobody is going to type at three in the morning. On the remote end a restricted key with command= in authorized_keys is used, the least privilege of 08-03: that key can only receive an rsync, not open a session. The encryption key, in turn, lives in a file with 600 permissions outside the repository and outside the backup itself. Encrypting the backup and storing the key inside it is the oldest joke in the trade; and a key that only exists on the burned-down server is no use either: the copy of the passphrase goes into the company password manager, and the runbook says who can recover it.

A backup that fills the disk brings down the very service it meant to protect.

A 20% margin is required because --link-dest makes the real consumption much smaller than the size of the source, but the margin covers the day somebody rewrites the whole CSV. If it does not fit: the profile is not run, an error is logged, a warning is sent and the next profile continues. What is never done is deleting old backups to make room —that turns a space problem into a loss of history—.

Each profile produces its own code, and the script aggregates:

The profile loop does not abort on the first failure: it accumulates into the failures array and carries on, because the log profile failing is no reason to end up without a copy of the data. At the end, a summary (07-04):

SECONDS (04-06) gives the duration without subtracting timestamps, and its trend is a valuable datum: if the backup goes from three to thirty minutes, something changed even if the exit code is 0.

Persistent=true (07-05) recovers the run if the server was off at 03:15, which is exactly when the copy is needed most. The RandomizedDelaySec prevents the three servers from hitting srv-veloz-03 in the same second. The service runs backup.sh --all with flock (07-02) so that two runs never overlap.

The runbook (docs/runbook.md) contains the only sequence that matters at four in the morning, with copy-pasteable commands: check the last good backup, verify it, restore to a temporary directory, compare, and only then move to production. A runbook that has never been executed is fiction: it is exercised in the quarterly drill.

• Confusing a copy with a backup. An rsync --delete to a disk mounted on the same server replicates the accidental rm -rf in seconds. Retention is needed, and retention is what turns the copy into a backup.
• --link-dest across different filesystems. It does not fail: it silently makes full copies until the disk fills up. And without .partial + mv, a power cut leaves an incomplete directory that retention counts as good and that you discover empty on restore day.
• rm -rf "$dir/$copy" without :?. The day a variable ends up empty, the script deletes from the root. It is the most dangerous line in the whole toolkit.
• Verifying the backup with the backup itself. Keeping the manifest only inside the copy detects data corruption, not the deletion of the whole set: that is why the count and the size are also recorded in the central log.
• Tip: put a real restore on the calendar every quarter, with a stopwatch. If the measured RTO exceeds the promised RTO, the design is wrong, not the operator.

1. Database backup. Add a postgres profile that uses pg_dump -Fc into a file before the copy, with the password in ~/.pgpass (600) and verification with pg_restore -l.
2. Weekly report on the state of the copies. A report subcommand that walks every profile and produces a table with: last good copy, age in days, size, number of copies per level and the result of the last verification, in text and JSON.
3. Partial restore. Extend restore with --file PATH to recover a single file, finding which copies it exists in and in which one it last changed.

1. The key is that the dump is taken before the rsync and verified on the spot:

pg_dump -Fc gives a consistent copy even while the database is in use; copying its files hot would give an inconsistent snapshot (07-03). pg_restore -l lists the contents without restoring: it is the cheapest possible verification that the dump is not truncated. It is hooked in with a PRE_COMMAND in the profile's .conf, invoked if it is defined.

2. The report leans on the manifests already written, so nothing has to be recomputed:

The days column is the one people really look at: a "good" backup from nine days ago is an alert, even if it verifies fine.

3. Walk the copies from newest to oldest with find "$base" -path "*/$FILE", and since unmodified files share an inode thanks to --link-dest, comparing the inode number (stat -c %i) between consecutive copies identifies exactly which one it changed in, without reading a single byte of the content.

backup.sh is no longer a copy script: it is a system. The profiles in etc/backup.d/*.conf let you add a set without touching code; rsync --link-dest gives copies that are generated as incrementals and restored as full ones; the .partial + mv pattern guarantees that a half-finished backup that looks good never exists; the manifest with count, size and checksums turns "there are files" into "there are these files and this is what they are"; retention promotes with hard links before pruning and deletes with ${var:?} as a safety net; and the weekly restore test answers on its own, every Sunday, the only question that matters: can it be recovered? Around it, what you learned before: subshells to isolate profiles (01-04), arrays for optional options (04-03), mktemp and trap (05-02), atomic writing and least privilege (08-03), BatchMode in SSH (07-06), a timer with Persistent=true (07-05) and warnings only when there are failures (07-04).

In 09-04 we move from protecting data to watching the service. We will build network-monitor.sh: defining in a measurable way what "the network is fine" means, a catalog of targets in a file, checks that return 0/1/2 in the style of the classic plugins, parallel execution because everything is network waiting, retries with backoff, alerts only on transitions and a TSV history from which the availability percentage comes.