The three previous projects look inside the server: its state, its logs, its data. This one looks outward. When somebody at Veloz Envíos says "the network is bad", they usually mean the application is slow, and the job of operations is to turn that sentence into a measurement. network-monitor.sh does exactly that: it defines in a measurable way what it means for the network and the service to be fine, checks it in parallel every few minutes, alerts only when something changes and keeps a history from which the month's availability comes.

Contents

  1. What "the network is fine" means, measurably
  2. Point-in-time check versus trend
  3. Design: a catalog of targets and one function per type
  4. The checks, one by one
  5. The dispatcher and the normalized result
  6. Parallelism: why it pays off here
  7. Retries with backoff
  8. State and alerts only on transitions
  9. Maintenance windows
  10. History and the summary subcommand
  11. Live mode, timer and going to production

  1. What "the network is fine" means, measurably

"It is fine" is not monitorable. These five statements are:

Measurable statement How it is checked Threshold at Veloz Envíos
The host responds ping -c1 -W2 0% loss, RTT < 50 ms
The name resolves dig +short Returns at least one A, < 500 ms
The port is open nc -z -w2 or /dev/tcp Connection in < 2 s
The API responds correctly curl -w '%{http_code} %{time_total}' 200 and < 800 ms
The certificate has not expired openssl s_client More than 15 days of margin

Each row is an independent check, with its threshold and its severity. The distinction matters: a ping that responds says nothing about whether the API works, and an API that returns 200 in four seconds is broken even though the code is correct. A service is declared healthy by its useful response, not by its reachability.

  1. Point-in-time check versus trend

They are two different products that share data:

  • The point-in-time check answers right now: is it down? Its output is a status and its possible consequence is an alert.
  • The trend answers over the month: how available has it been, how is latency evolving? Its output is a number in a report and its consequence is an architecture decision.

Confusing them produces the two classic failures: alerting about a latency that has been rising for three weeks (too late) or drawing graphs while nobody notices an outage (useless). The design separates them: each run alerts if appropriate and appends a line to the history; the summary subcommand only reads the history.

  1. Design: a catalog of targets and one function per type

Just like the backup profiles (09-03), the targets live in a file:

# etc/network-monitor.d/targets.conf
# name|type|target|threshold|severity
api-health|http|http://localhost:8080/salud|800|critical
api-shipments|http|http://localhost:8080/envios?ciudad=Valencia|1500|high
db-port|port|10.0.0.20:5432|2000|critical
dns-internal|dns|api.veloz.local|500|medium
srv-02|ping|srv-veloz-02|50|high
gateway|ping|10.0.0.1|30|critical

The separator is | and not a comma because URLs easily carry commas and vertical bars do not. It is read with IFS='|' read (03-06), ignoring comments and empty lines:

read_targets() {
  local name type target threshold sev
  while IFS='|' read -r name type target threshold sev; do
    [[ $name == \#* || -z $name ]] && continue
    printf '%s|%s|%s|%s|%s\n' "$name" "$type" "$target" "$threshold" "${sev:-medium}"
  done < "$TARGETS"
}

Each type has its check_<type> function and returns the convention of the classic plugins (07-04): 0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN. That convention is what allows the rest of the script to know nothing about ping or curl.

  1. The checks, one by one

check_ping() {  # $1 target, $2 threshold ms -> prints "ms message"
  local output rtt
  output=$(ping -c1 -W2 -n "$1" 2>/dev/null) || { printf '0 no response\n'; return 2; }
  rtt=${output#*time=}; rtt=${rtt%% *}
  printf '%s rtt=%sms\n' "$rtt" "$rtt"
  awk -v r="$rtt" -v u="$2" 'BEGIN {exit !(r+0 > u)}' && return 1 || return 0
}

check_dns() {
  local start end ip
  start=$(date +%s%3N); ip=$(timeout 3 dig +short +time=2 +tries=1 "$1" A | head -1)
  end=$(date +%s%3N)
  [[ -n $ip ]] || { printf '0 does not resolve\n'; return 2; }
  printf '%s resolves to %s\n' "$((end - start))" "$ip"
  (( end - start > $2 )) && return 1 || return 0
}

check_port() {
  local host=${1%:*} port=${1##*:} start end
  start=$(date +%s%3N)
  timeout 2 bash -c "exec 3<>/dev/tcp/$host/$port" 2>/dev/null ||
    { printf '0 closed or filtered\n'; return 2; }
  end=$(date +%s%3N); printf '%s open\n' "$((end - start))"
}

check_http() {
  local resp code ms
  resp=$(curl -sS -o /dev/null --max-time 5 -w '%{http_code} %{time_total}' "$1" 2>/dev/null) \
    || { printf '0 no connection\n'; return 2; }
  read -r code ms <<< "$resp"
  ms=$(awk -v t="$ms" 'BEGIN {printf "%d", t * 1000}')
  [[ $code == 200 ]] || { printf '%s code %s\n' "$ms" "$code"; return 2; }
  printf '%s 200 in %sms\n' "$ms" "$ms"
  (( ms > $2 )) && return 1 || return 0
}

Details that come from 06-04 and 06-05. ping -n avoids the reverse lookup, which can add seconds of waiting because of a slow DNS and distort the measurement. /dev/tcp with timeout does not need nc installed, but it is a pure bashism (08-07): if the script had to run under dash, nc -z -w2 would go here. curl -w gives the code and the time in a single invocation, instead of measuring from outside with date; --max-time is mandatory, because a check with no limit can hang the whole cycle. And HTTP 200 is demanded explicitly: a 302 to a maintenance page is not service. Each function prints <milliseconds> <message> and returns the status through the exit code: separating the datum (stdout) from the verdict (code) is the same split from 09-01 between collecting and deciding.

  1. The dispatcher and the normalized result

run_target() {  # name|type|target|threshold|severity -> TSV line
  local name type target threshold sev
  IFS='|' read -r name type target threshold sev <<< "$1"
  local fn=check_$type output status
  declare -F "$fn" >/dev/null || { veloz_log_error "unknown type: $type"; return 3; }
  output=$(with_retries "$fn" "$target" "$threshold"); status=$?
  printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
    "$(date -Is)" "$name" "$type" "$status" "${output%% *}" "${output#* }"
}

declare -F as an allowlist (08-03): the type from the file becomes a function name only if that function exists. An explicit case would be just as valid (04-05) and more readable for anyone who does not know the trick; the advantage of declare -F is that adding a new type means writing a function and nothing else. The normalized result —ISO date, name, type, status, milliseconds, message— is the internal format consumed by the report, the alerts and the history: once again the same architecture as the three previous projects.

  1. Parallelism: why it pays off here

With twelve targets averaging 700 ms each, in series that is eight and a half seconds. And they are not eight seconds of CPU: they are eight seconds of waiting. That is the 08-02 criterion for parallelizing: when the process is blocked waiting on the network, launching twenty at once costs practically nothing.

run_all() {
  local tmp; tmp=$(mktemp -d); trap 'rm -rf "$tmp"' RETURN
  local i=0 tgt
  while IFS= read -r tgt; do
    run_target "$tgt" > "$tmp/$(printf '%03d' "$i")" &
    ((i++))
    (( $(jobs -rp | wc -l) >= PARALLELISM )) && wait -n
  done < <(read_targets)
  wait
  cat "$tmp"/*
}

Three things make this correct and not chaos. Each job writes into its own numbered temporary file (05-02): if they all wrote into the same one, the lines would interleave. The name carries a zero-padded index (%03d) so that the final cat returns the catalog's order, not the completion order —a report whose rows dance on every run is unreadable and makes comparing two outputs impossible—. And wait -n limits the simultaneous jobs: it waits for any one to finish before launching the next, which is the equivalent of xargs -P with background processes. With xargs -0 -P "$PARALLELISM" the code would be shorter, at the price of having to export the functions; with twenty targets, either one works.

  1. Retries with backoff

A lost packet is not an outage. Alerting on the first failed check generates the noise that makes people ignore alerts, which is the worst possible breakdown of a monitoring system.

with_retries() {  # $1 = function, rest = arguments
  local fn=$1 attempt delay=1 output status; shift
  for attempt in 1 2 3; do
    output=$("$fn" "$@"); status=$?
    (( status == 0 )) && { printf '%s' "$output"; return 0; }
    (( attempt < 3 )) && { sleep "$delay"; delay=$((delay * 2)); }
  done
  printf '%s' "$output"; return "$status"
}

Exponential backoff (05-03): 1 s, 2 s. Only the failure is retried and it exits on the first success, so the normal case costs nothing. Three attempts and a maximum of three seconds of waiting are enough to filter the noise without delaying the cycle. Watch out for multiplication: three retries times twelve sequential targets would be a minute-and-a-half cycle in a general blackout; with the parallelism of the previous section, it is still four seconds.

  1. State and alerts only on transitions

The state file keeps the last known situation of each target, and the alert fires only when it changes (07-04):

process_states() {
  declare -A previous
  [[ -r $STATE ]] && while IFS=$'\t' read -r n s; do previous[$n]=$s; done < "$STATE"
  local new=$STATE.partial prev; : > "$new"
  local stamp name type status ms msg
  while IFS=$'\t' read -r stamp name type status ms msg; do
    printf '%s\t%s\n' "$name" "$status" >> "$new"
    prev=${previous[$name]:-0}
    (( prev == 0 && status != 0 )) && alert FAIL "$name" "$msg" "$ms"
    (( prev != 0 && status == 0 )) && alert RECOVERED "$name" "$msg" "$ms"
  done
  mv "$new" "$STATE"
}

Only two transitions generate a notification: OK→FAIL and FAIL→OK. A target that has been down for six hours does not warn again every five minutes, and the recovery notice is as important as the failure one, because it closes the incident without anyone having to go and look. Writing the state uses .partial + mv again (08-03): if the script dies halfway, the previous state file is left intact, and a corrupt state would cause a storm of false transitions on the next run.

Sending uses jq -n to build the body (06-05), never printf with the message interpolated:

alert() {
  local class=$1 target=$2 detail=$3 ms=$4
  veloz_log_info "alert $class $target: $detail"
  in_maintenance "$target" && { veloz_log_info "silenced (maintenance)"; return 0; }
  [[ -n ${WEBHOOK:-} ]] || return 0
  jq -n --arg c "$class" --arg o "$target" --arg d "$detail" --arg h "$(hostname -s)" \
     '{text: "[\($c)] \($o) on \($h): \($d)"}' |
    curl -sS --max-time 10 -X POST -H 'Content-Type: application/json' -d @- "$WEBHOOK" >/dev/null ||
    veloz_log_error "could not send the alert"
}

A webhook failure is logged but does not abort the monitor: a downed notification system must not take monitoring down with it. And the message is always written to the local log first, so that there is a record even if it never goes out.

  1. Maintenance windows

# etc/network-monitor.d/maintenance.conf -> target|start ISO|end ISO
in_maintenance() {
  local tgt=$1 now o start end; now=$(date +%s)
  [[ -r $MAINTENANCE ]] || return 1
  while IFS='|' read -r o start end; do
    [[ $o == "$tgt" || $o == '*' ]] || continue
    (( now >= $(date -d "$start" +%s) && now <= $(date -d "$end" +%s) )) && return 0
  done < "$MAINTENANCE"
  return 1
}

Silencing suppresses the alert, not the check: the history keeps recording the real status, so the month's availability is not falsified. It is a distinction many commercial systems get wrong, and it is the difference between "do not bother me right now" and "let us pretend it did not happen".

  1. History and the summary subcommand

Each run appends its lines to logs/network-YYYY-MM.tsv. One file per month keeps the size bounded without needing logrotate and makes the monthly report trivial.

summary() {  # $1 = monthly file
  awk -F'\t' '
    { n[$2]++; if ($4 == 0) ok[$2]++; if ($5 + 0 > 0) { sum[$2] += $5; m[$2]++ }
      if ($4 != 0) fails[$2]++ }
    END {
      printf "%-14s %8s %8s %10s %8s\n", "TARGET", "SAMPLES", "AVAIL.", "AVG.LAT", "FAILURES"
      for (k in n)
        printf "%-14s %8d %7.2f%% %8.0fms %8d\n", k, n[k], ok[k]*100/n[k],
               (m[k] ? sum[k]/m[k] : 0), fails[k]+0
    }' "$1" | (read -r hdr; printf '%s\n' "$hdr"; sort -k3 -n)
}

Availability is a ratio between samples, not between minutes: with a five-minute cycle, each failed sample represents five minutes of unavailability, and it is worth saying so in the report so nobody confuses precision with accuracy. The average latency excludes the samples with 0 ms, which are failures and not measurements —including them would artificially lower the average exactly when the service is at its worst, the most common interpretation error in monitoring—. The (read -r hdr; ...) keeps the header at the top while sorting the rest by ascending availability, that is, the worst ones first.

  1. Live mode, timer and going to production

For a diagnostic session a loop of your own works well, and it is preferable to watch because it keeps the history and the alerts:

live_mode() {  # \033[H\033[J clears the screen on each round
  while true; do printf '\033[H\033[J'; run_all | format_text; sleep "${INTERVAL:-30}"; done
}

But in production a systemd timer (07-05) every five minutes rules, for three concrete reasons: it survives the SSH session closing, journalctl -u keeps the trace of every run, and systemctl list-timers shows whether it really is running. A loop in a forgotten terminal looks like monitoring until the day somebody closes the laptop.

Common Mistakes and Tips

  • Monitoring the ping and calling it the service. The host responds and the API returns 500: the check that matters is the one that does what the user does.
  • Alerting on every run for as long as the failure lasts. It is the fast lane to alerts being ignored. Transitions only, and always a recovery notice.
  • Checks without timeout. A target that never responds blocks the whole cycle; with parallelism, it exhausts the processes. --max-time, -W, timeout: always.
  • Parallelizing with everyone writing to the same file. The lines interleave: one temporary per job and an ordered consolidation. And during maintenance, silence the alert, never the check, or you will falsify the availability in the monthly report.
  • Tip: monitor the monitor as well. If veloz-monitor.timer does not run, nobody notices; a check on the "age of the last line in the history" inside watchdog.sh closes that hole.

Exercises

  1. TLS certificate expiry. Add the tls type, warning when fewer than 15 days remain until expiry and going critical below 5.
  2. Route trace on failure. When a ping target goes to FAIL, automatically attach a route trace toward the target to the log, without delaying the normal cycle.
  3. Trend report. Extend summary with --compare MONTH, showing the variation in availability and latency against the previous month, marking the regressions.

Solutions

1. openssl s_client needs -servername for SNI and input on stdin that closes the connection:

check_tls() {
  local host=${1%:*} port=${1##*:} end days
  end=$(echo | timeout 5 openssl s_client -connect "$host:$port" -servername "$host" 2>/dev/null |
        openssl x509 -noout -enddate 2>/dev/null) || { printf '0 no certificate\n'; return 2; }
  end=${end#notAfter=}; days=$(( ( $(date -d "$end" +%s) - $(date +%s) ) / 86400 ))
  printf '%s expires in %s days (%s)\n' "$days" "$days" "$end"
  (( days < 5 )) && return 2; (( days < 15 )) && return 1; return 0
}

The echo | at the start is essential: without it, s_client leaves the connection open waiting for input and timeout ends up killing it. The date arithmetic is that of 04-06, subtracting epoch seconds.

2. The trace must not block the cycle, so it is launched in the background and with a limit:

extra_diagnostics() { { timeout 20 traceroute -n -w1 -q1 "$1" 2>&1 | veloz_log_info_stdin; } & disown; }

It is invoked from alert only on the transition to FAIL —never while it lasts— so as not to launch a trace every five minutes during a two-hour outage.

3. Run summary over the two monthly files, load them into two associative arrays (04-03) indexed by target and walk them comparing. The presentation rule: show the difference with a sign and mark only what gets worse by more than a threshold (for example, 0.5 points of availability or 20% of latency), because a report that marks everything marks nothing.

Conclusion

network-monitor.sh turns "the network is fine" into five measurable statements with their thresholds, and checks them from a catalog in a file that grows without touching code. The transferable ideas are four. The 0/1/2/3 convention of the classic plugins, which lets the dispatcher know nothing about ping or curl and makes adding a type a matter of writing a function. Parallelize what waits: twelve targets in four seconds instead of eight and a half, with one temporary per job and consolidation in a fixed order so the report is comparable. Alert on transitions, with a recovery notice and windows that silence the notification but never the measurement. And separate the point-in-time check from the trend: the first alerts, the second informs, and both come out of the same TSV line. Along the way: retries with backoff (05-03), curl -w and /dev/tcp with timeout (06-04), jq -n for the webhook body (06-05), wait -n to limit concurrency (05-02), atomic writing of the state (08-03) and awk for the monthly summary (06-01).

Five working scripts are left, each with its options, its configuration and its own way of being invoked. In 09-05, the close of the course, they stop being a collection and become a product: veloz-ops, a single command with subcommands, unified configuration, an idempotent installer, versioning with Git, a test battery, fleet deployment, rollback and documentation.

Bash Programming Course

Module 1: Introduction to Bash

Module 2: Basic Bash Commands

Module 3: Scripting Fundamentals

Module 4: Intermediate Scripting

Module 5: Advanced Scripting Techniques

Module 6: Working with External Tools

Module 7: Automation and Scheduling

Module 8: Best Practices and Optimization

Module 9: Real-World Projects

© Copyright 2026. All rights reserved