service-status.sh already knows how to say whether the machine is healthy and whether the veloz-api process exists. But "the process exists" and "the service responds" are not the same thing: a process can be alive with port 8080 closed, listening only on 127.0.0.1 while the rest of the network cannot reach it, or blocked without accepting connections. To tell those cases apart you have to step outside the local system and look at the network. This lesson covers network diagnosis and automation from a script: checking connectivity, resolving names, seeing which ports are listening, testing whether one responds —including the /dev/tcp noted down in 05-05— and waiting with retries for a service to come up.

1. The four layers that can fail
2. Connectivity: ping and its exit code
3. Name resolution
4. Ports and sockets: ss
5. Checking whether a port responds: nc and /dev/tcp
6. Interfaces, routes and your own IP
7. curl versus wget
8. Checking availability and measuring with curl
9. Active waiting with retries and backoff
10. Transfers and remote sessions
11. Firewalls, in broad strokes
12. Security: credentials and certificates
13. Application: veloz_port_open in the toolkit

When "the API is down", the useful diagnosis is knowing where it breaks, and for that it is worth checking in order:

Skipping a layer is the cause of wrong diagnoses: blaming the network when what is failing is DNS, or writing off an API that is in fact returning a perfectly delivered 500. A serious monitoring script distinguishes the four cases and says so in the error message.

Interactive ping is launched bare and cut off with Ctrl-C; in a script that is unacceptable, so it always carries -c (number of packets) and it is worth adding -W (maximum wait per reply, in seconds):

The key is that what matters is not ping's output but its return code: 0 if it got any reply, non-zero if not. That is why -q (quiet mode) and the redirection to /dev/null lose nothing: all the information the script needs is in $?. It is exactly the same principle as 03-01 applied to the network.

Two warnings. First, ping uses ICMP, and many firewalls block it even though the service works perfectly: a failed ping does not prove the machine is down, only that it does not answer ICMP. Second, to see where the path is lost there are traceroute (or tracepath) and mtr, which combines a trace with continuous statistics; they are manual diagnostic tools, not script tools.

Before blaming the network you have to check that the name translates to an IP. The correct option in a script is getent hosts, the same command from 06-03: it queries the sources the system has configured in /etc/nsswitch.conf —including /etc/hosts—, which is exactly what the application will do when it connects.

getent hosts api.veloz.example returns 10.20.0.15 api.veloz.example, whereas dig +short api.veloz.example gives the same IP but asking DNS only, ignoring /etc/hosts. The difference matters: if someone put an entry in /etc/hosts, dig will say one thing and the application will go somewhere else. dig +short, on the other hand, is unbeatable for asking a specific server (dig @8.8.8.8 +short api.veloz.example) and thereby telling an internal DNS problem apart from a general one. host is an abbreviated version of dig, and nslookup is the veteran that still shows up in old manuals: it works, but its output is the most awkward to parse and its exit code is not very reliable. Order of preference: getent hosts in scripts, dig +short to diagnose.

ss shows the system's sockets and has replaced netstat, which on many distributions is not even installed any more. The combination to memorize is ss -tulpn:

Reading that whole line is the diagnosis that was missing in 06-03. LISTEN confirms it accepts connections; 127.0.0.1:8080 is the revealing part: the API listens on the local interface only, so no other machine will be able to connect even if the process is perfect. If it said 0.0.0.0:8080, it would be listening on every interface. And users:(("veloz-api",pid=2114)) says which process holds it, which is the answer to "something is occupying my port". For the same thing from the open-files angle there is lsof -i :8080 (06-03), useful when ss is not available.

ss looks from inside the machine. To check from outside —or simply to verify that a connection really does get established— there are two ways.

The first is nc (netcat) with -z, which tries to connect without sending data, and -w with a maximum time: nc -z -w2 localhost 8080 && echo open. The second needs nothing installed, because it is a Bash feature: the pseudo-files /dev/tcp/host/port and /dev/udp/host/port. Opening them with the descriptor redirection of 05-05 attempts a TCP connection:

Three details make this work. exec 3<> opens descriptor 3 for reading and writing over the connection; if the port is closed, the redirection fails and the bash -c returns a non-zero code. The timeout 2 from 05-02 is mandatory: without it, a machine that silently drops packets would leave the script hanging for the system's timeout, which can be over a minute. And the 2>/dev/null silences the Connection refused message that Bash prints.

The portability warning is a serious one: /dev/tcp is not a real file nor a system feature, it is a Bash facility. It does not exist in dash, which is Ubuntu's /bin/sh, so a script with #!/bin/sh that uses it will fail with No such file or directory. Besides, some distributions compile Bash without this facility. The comparison comes out like this:

The ip family replaced ifconfig and route. ip a (short for ip address show) lists the interfaces with their addresses, ip -4 a only the IPv4 ones, and ip r shows the routing table, whose first line default via ... is the gateway.

Getting your own IP in a script has a catch: a machine can have several interfaces, and hostname -I returns all of them separated by spaces, without saying which one will be used to go out. The reliable way is to ask the routing table:

ip route get sends no packet at all: it asks which route and source IP the kernel would use to reach that destination, which is exactly the right answer. The awk of 06-01 looks for the word src and takes the following field, instead of trusting a fixed position that changes with the configuration —the same robustness principle as $NF—.

Both download over HTTP, but they are designed for different things:

For an operations script, the rule is simple: curl to talk to services, wget to fetch a large file unattended. Downloading with both:

And a hygiene warning: curl ... | bash —the "one-line installer" many projects promote— runs as you a piece of code you have not seen and that can change between two runs. Download first, review, run afterward.

To know whether an HTTP service is alive without downloading the body, -I asks for the headers only and -f makes curl fail with code 22 on an HTTP error instead of cheerfully returning 0:

That -sfI with --max-time is the standard availability check: quiet, failing on errors and unable to hang. The fine breakdown of -s, -S and -f and of the HTTP codes belongs to 06-05; here what matters is the exit code.

To measure, -w prints variables on completion, with -o /dev/null to throw the body away:

%{time_total} is the complete latency and %{time_connect} only the connection setup: if the total is high but the connection is fast, the problem is in the application, not in the network. Recording that number on every run gives you a time series with which to spot degradations before they turn into outages —the basis of project 09-04—.

After restarting veloz-api, the port takes a few seconds to accept connections. Checking immediately gives a false negative; sleeping for a fixed time is guesswork. The solution is the active wait with exponential backoff from 05-03, applied to the network:

Backoff (1, 2, 4, 8…) is preferable to a fixed interval for two reasons: it reacts fast if the service comes up right away and it does not hammer a machine that is already in trouble. The six default attempts cover about 63 seconds, enough for a normal restart. And the final return 1 is what lets you chain veloz_wait_service localhost 8080 || veloz_die 1 "the API did not come up".

Copying files between machines is done with scp source user@destination:/path for one-off copies or with rsync -az --delete source/ user@destination:/path/ when directories have to be synchronized, since it only transfers the differences. Running commands on another machine is ssh user@destination 'command'. All three are the basis of remote automation and have their own lesson: passwordless keys, ssh-agent, known_hosts, non-interactive options and the dangers of running blind on several machines are covered in 07-06. The only thing worth retaining now: in a script, ssh needs -o BatchMode=yes so it fails instead of sitting there waiting for a password nobody is going to type.

When the port is in LISTEN on 0.0.0.0 but cannot be reached from another machine, the usual suspect is the firewall. On Ubuntu, ufw status verbose (needs privileges) lists the active rules; underneath are iptables -L -n or nft list ruleset. In the cloud there is also a second firewall outside the machine —security groups— that no local command can see, and which explains many impossible diagnoses. Mental rule: if ss says it is listening on 0.0.0.0, nc works from the machine itself and not from outside, the problem is on the path, not in the service.

Three non-negotiable rules, developed further in 08-03:

• Never embed credentials in the URL. curl https://user:password@api.veloz.example/ leaves the password in the history (02-06), in ps for as long as the command lasts and in the server's logs. Use --netrc, or an environment variable read from a file with 600 permissions such as veloz-ops.conf.
• Do not disable certificate verification. curl -k (or --insecure) accepts any certificate, including that of whoever interposes themselves in the connection: it turns HTTPS into HTTP with a fake padlock. If the certificate is internal, the right thing is to install it or pass it with --cacert /path/ca.pem.
• Always use HTTPS, even on the internal network. A "trusted" network stops being one the day somebody plugs in a compromised laptop.

The function that was missing in lib/common.sh, with both implementations and automatic selection:

Validating the port with =~ (05-04) is not paranoia: the number ends up inside a string that bash -c is going to execute, so an unvalidated value would be code injection in the fullest sense. command -v (06-03) picks the available implementation, and both branches return their exit code directly, which becomes the function's.

With it, service-status.sh finally tells apart the cases it used to confuse:

Those three messages are the whole lesson in summary: process, port and application are three different things, and saying which one failed saves half an hour of diagnosis for whoever reads the alert at three in the morning. This function is the seed of project 09-04.

• ping without -c. In a script it hangs forever. And a failed ping does not prove the machine is down: many firewalls block ICMP.
• Trusting pgrep alone. The process existing does not mean the port accepts connections or that the application answers.
• Using /dev/tcp without timeout. Against a machine that drops packets, the script hangs for over a minute.
• /dev/tcp with #!/bin/sh. It is a bashism: it fails in dash (08-07). If you need portability, use nc.
• curl without -f. It returns 0 even if the server answers 500: your check will always say everything is fine.
• dig to find out which IP the application will go to. dig ignores /etc/hosts; use getent hosts.
• Misreading 127.0.0.1:8080 in ss. It means only the machine itself is served, however much it is in LISTEN.
• Tip: put --max-time (or timeout) on every network command. A monitoring script that hangs is worse than one that fails, because nobody finds out.
• Tip: always diagnose in the order network → name → port → application, and make the error message say which layer broke.

Exercise 1. Write a function veloz_dns_ok that checks that a name resolves and that the IP obtained matches the expected one, logging different messages for "does not resolve" and "resolves to an unexpected IP".

Exercise 2. Write a snippet that, after restarting veloz-api, waits up to 30 seconds for /salud to answer correctly, measuring how long it took, and exits with an error if it does not make it.

Exercise 3. Write a check that detects the specific case of "the API listens only on 127.0.0.1" and reports it as a configuration problem, not as an outage.

Solution 1.

getent hosts can return several lines if there are several addresses, hence the NR==1. The || true prevents set -e (05-03) from killing the script when the name does not resolve: here the failure is an expected result we want to handle, not an error. The different codes 1 and 2 let the caller react differently to "there is no DNS" and "DNS points to the wrong place", which are very different incidents.

Solution 2.

Five attempts with 1+2+4+8+16 backoff cover 31 seconds. The two checks are chained because they are different: first that the port accepts connections, then that the application answers properly —a service can open the port before it finishes starting up—. The measurement with date +%s is the epoch arithmetic of 04-06.

Solution 3.

ss -tuln without -p needs no privileges, which is what you want in a checking script. The awk filters the local address column ending in :8080, and the glob comparisons from 04-05 catch both 127.0.0.1 and the IPv6 variant [::1]. Telling this case apart matters a lot: it is a configuration fault fixed in a file, not an outage fixed by restarting.

Diagnosing the network from a script means walking four layers in order —network, names, port, application— and saying which one broke. ping -c -W answers for the first, and what you use from it is its exit code, not its output; remember that ICMP is often blocked. Names are resolved with getent hosts in scripts (it sees the same as the application, including /etc/hosts) and with dig +short to diagnose against a specific server. ss -tulpn shows who is listening and on which address, and that address tells an accessible 0.0.0.0 apart from a 127.0.0.1 that condemns the service to talking only to itself. To test a port from outside there are nc -z -w2 and the bashism /dev/tcp/host/port, which needs nothing installed but demands timeout and does not exist in sh. ip route get gives your own IP reliably, curl -sfI --max-time checks HTTP availability while genuinely failing on an error, curl -w measures latencies, and the active wait with exponential backoff replaces eyeballed sleeps when you have to wait for a service to come up. All of it with a timeout, never without; and with no credentials in the URL and no curl -k.

service-status.sh now tells process, port and application apart. But it stops at the door: it knows /salud returns a 200, not what it says. The veloz-api answers JSON —database state, queued shipments, deployed version, metrics—, and for a script that is text you must not touch with grep or sed. The next lesson (06-05) closes the module with curl in depth for APIs and with jq, the tool that turns JSON into manageable data and also into output: daily-report.sh will stop writing plain text only and start publishing its summary as JSON.