Reservalia now deploys on its own, with reproducible environments, a deliberately chosen handover strategy and a rehearsed go-back button. And yet it still has a hole right in the middle: it finds out that production is unwell because a customer calls. The 3 minutes of detection promised by the table in the previous lesson do not exist yet, check-metrics.sh invokes thresholds nobody has defined, and the Friday freeze is still standing because nobody has an objective argument for lifting it. With no production signal, continuous deployment is simply deploying blind faster. This lesson closes the loop — and with it, the module: what to observe, how to turn those observations into objectives with consequences, when it is worth waking somebody up, and how the pipeline itself feeds on what happens in production.

1. The open loop: why CD without monitoring is a fast bet
2. The three pillars of observability
3. The four golden signals applied to Reservalia's API
4. Reservalia's minimal dashboard
5. SLI, SLO and error budget
6. The budget policy: goodbye to the Friday freeze
7. Marking deployments on the dashboards
8. Alerts: symptoms, not causes
9. Feedback into the pipeline: the DORA metrics feed themselves
10. Reservalia's dashboard after modules 2 and 3
11. Common Mistakes and Tips
12. Exercises
13. Conclusion

Everything built so far forms a chain that pushes changes into production faster and faster. What is missing is the return arrow.

Without that arrow three specific things happen, all of them visible at Reservalia today: the canary from 03-04 cannot decide whether to promote, because check-metrics.sh has nothing to compare against; the rollback from 03-05 arrives late, because detection depends on a customer calling; and the team keeps the Friday freeze because its only alternative to blind caution would be blind recklessness.

It is worth distinguishing two words that get used as synonyms. Monitoring is watching things you already know can fail: CPU, memory, the number of errors. Observability is the property of a system that lets you answer questions you had not anticipated: "why do bookings for businesses with split opening hours take four seconds only on Mondays?". The first is configured; the second is designed, by instrumenting the code so it emits enough context.

The order matters and it is the order of an investigation: the metric warns you, the trace locates it, the log explains it. A team with only logs finds out late and searches blind; one with only metrics knows something is wrong but not why.

Reservalia makes a small investment that multiplies the value of its logs: emitting them as structured JSON with fixed fields.

1. Including the commit on every line is what lets you, during an incident, filter errors by version and confirm in ten seconds whether they started with the 09:08 deployment. It is the same idea as the /version endpoint, applied to logs.
2. Structured fields rather than interpolated text. log('error', 'booking failed', { businessId: 412, durationMs: 3140 }) can be aggregated and filtered; console.log('Booking error for business 412') can only be read. The rule: never put a variable value inside the message text.

A cost warning that surprises many teams: logs are the pillar that grows most and bills most. Reservalia logs business operations at info (booking created, cancelled) and failures at error, but not one line per HTTP request: that is what metrics are for, and they cost a fraction.

The four golden signals are the minimal set that answers "is my service healthy?" for almost any request-driven system.

Three points that separate a useful dashboard from a decorative one:

Latency is measured in percentiles, never as a mean. If 99 requests take 100 ms and one takes 10 seconds, the mean is 199 ms — a reassuring number that hides a user who has walked away. The p95 says "95% of users wait less than this", and the p99 is where the rare cases that end up in the support chat live. On top of that, the latency of errors must be measured separately: a service that starts returning 500s in two milliseconds improves its mean latency while it is falling over.

Traffic is almost never an alarm, but it is essential as context. A 5% error rate with 20,000 requests and with 40 requests are incidents of very different severity, and traffic dropping to zero may be the only signal that the front end is broken even though the API responds perfectly.

Saturation is the only signal that warns you beforehand. Latency, errors and traffic describe what is already happening; a connection pool at 85% says there will be an incident in twenty minutes. It is the signal that lets you act rather than react.

Nuria creates a single dashboard, reservalia-api-prod, with the rule that it must fit on one screen and answer the question "is it healthy?" in ten seconds.

1. Dimensions are the key and also the trap. They let you filter by route, method and response class, which is exactly what you need to tell whether the problem is general or confined to one endpoint. But each distinct combination of dimensions is a billable metric: using req.path instead of req.route.path would create a metric for every /api/businesses/412/slots, that is, one per business, with a three-figure bill and unusable dashboards. Never use identifiers as a dimension.

A dashboard tells you how the system is; it does not tell you whether that is acceptable. For that you need three linked concepts.

Reservalia defines two SLOs, and only two. The temptation to define fifteen is strong and it is a mistake: an SLO nobody looks at disciplines nothing.

Three design decisions deserve an explanation. It is measured from the user's point of view, not the server's: the SLI counts requests that reached the ALB, including those that failed because there was no healthy task; measuring only what the application managed to process is marking your own homework. The objective is not 100%, and this is the hardest part to accept: 100% is impossible (the network fails, AWS fails, the client fails) and undesirable, because the cost of each additional nine multiplies and that money comes out of building product. 99.9% over 30 days is 43 minutes of budget, which is exactly the material with which you buy the ability to deploy fast.

Because that is the central idea: the error budget is not an evil to be avoided, it is a resource to be spent. A team that ends the month with 100% of its budget intact has not been excellent: it has been too conservative, it has deployed less than it could and it has delivered less value than it could. A team that exhausts it on the 12th has a reliability problem. The goal is to reach the end of the month having spent almost all of it.

In lesson 03-01 we left the Friday freeze labelled as technical debt, with a retirement date and conditions. The error budget is the objective rule that replaces it, because it answers the same worry — "I do not want to break production at a bad moment" — with data instead of a calendar.

The three properties that make this work better than a freeze:

• It is objective. It does not depend on somebody feeling nervous today. The rule is agreed when nobody is in an incident, which is the only moment anyone thinks clearly.
• It is symmetrical. If reliability is good, the team gains freedom: you deploy at five o'clock on a Friday. The freeze only restricted; the budget also rewards.
• It aligns everybody. Product stops asking for "more features and also more stability" as if they were independent: the budget makes it explicit that they are the same currency.

Reservalia lifts the Friday freeze the day the three conditions written down in 03-01 are met — a rollback demonstrated in a timed drill, automatic detection of serious incidents, and an approval history that was filtering nothing — and replaces it with this table. In the first week without a freeze, Diego merges a change at 16:40 on a Friday; the canary promotes it at 17:10 and absolutely nothing happens. That non-event is the result of the entire module.

The question a team asks itself when facing a degradation is always the same: did this start with the last deployment? Answering it by looking at two tabs and comparing timestamps is slow and error-prone. The solution is cheap: annotate every deployment onto the graphs.

With that metric overlaid as a vertical line on the latency and error graphs, the correlation is visible at a glance: if the curve bends right at the line, the suspect is identified; if it bent twenty minutes earlier, the deployment is innocent and you need to look at the dependencies. It is the same reasoning we applied to logs with the commit field, carried over to the graphs, and it is what makes the first question of an incident answerable in seconds rather than in ten minutes.

An honest caveat: correlation is not causation. A coincident deployment may be a coincidence — at 09:00 you deploy and the morning traffic peak begins. The annotation does not close the investigation; it starts it in the right place.

An alert must meet three conditions: it is real (not noise), it is urgent (it cannot wait until tomorrow) and it is actionable (whoever receives it can do something). If any of them fails, it is not an alert: it is an email.

The fundamental distinction is between alerting on causes and alerting on symptoms.

Alerting on causes produces the worst of both worlds: lots of notifications that mean nothing to the user and, at the same time, gaps — the day the system fails for a reason nobody foresaw, there is no alert. Reservalia defines three alerts and not one more:

The third deserves a comment, because it is the most sophisticated. A monthly SLO has a problem: if the system goes down completely on the 2nd, the accumulated monthly percentage still looks fine for hours and nothing fires. The budget's burn rate measures how much is being spent right now, and detects in minutes an outage that the monthly percentage would take a day to reflect. It is the alert that turns an SLO into something operational rather than just a report.

On alert fatigue: it is the most common and the most expensive failure, because it does not manifest as an incident but as a slow loss of trust. A team receiving thirty notifications a day stops reading them — not out of indiscipline, but because it is humanly inevitable — and the day the important one arrives, it is lost among the rest. Three rules of hygiene: if an alert has required no action the last five times, delete it or turn it into a ticket; every alert that wakes someone links to a written procedure with the first three steps; and the number of alerts that wake people is reviewed monthly and must trend downwards.

The definitive test before creating an alert that will go off in the small hours: what would a person do at 3:40 on receiving it? If the answer is "look at it and go back to sleep", that alert must not exist.

In lesson 01-05 we built the DORA dashboard with the deployments and incidents tables, filled in by hand. A dashboard that depends on somebody remembering to write a row is out of date within three weeks. Now that the pipeline knows about every deployment and the alerts know about every incident, both tables can fill themselves.

1. if: always() also records failed deployments. Without them, the change failure rate would measure only successes and would always come out at 0%, which is the most common way of having a beautiful, useless metric.
2. run_id links the row to the GitHub Actions run: from the dashboard you reach the logs in one click.
3. first_commit_date is what allows the real lead time to be computed, measured from when the code was written rather than from when the deployment started. It is obtained with git log -1 --format=%cI on the oldest commit in the PR.
4. ON CONFLICT DO NOTHING makes the insert idempotent, as 03-02 required: re-running the workflow does not duplicate the row or pollute the metrics.

Incidents are recorded the same way, from the alert and from rollback.yml itself in 03-05, which already took a mandatory reason precisely for this. With both tables up to date, the four metrics are queries:

A methodological point about the change failure rate: it does not count the deployments that failed in the pipeline — those are good news, the system did its job — but those that reached production and had to be fixed or reverted. The practical way to measure it is to count deployments followed by a rollback or an incident in the next 24 hours, which is exactly what cross-referencing deployments with incidents allows.

This is the result of two modules' work, measured with the same yardstick as the baseline:

Three of the four targets met, and the fourth halfway there. It is worth looking at why the change failure rate has stalled, because the answer is not in this module: the failures that remain are not deployment failures, they are content failures — a dependency that changed behaviour in a minor update, a schema migration that locked a table during working hours, a business case the tests did not cover. That is exactly what the next module tackles.

And there is one improvement no table captures. The Friday ritual from lesson 01-04 — three hours of SFTP, migrations by hand in psql, Diego watching logs — has disappeared. It has not been optimised: it has ceased to exist as a category of work. Marta, Diego and Nuria spend that time on something else, and that is the real benefit the four metrics only hint at.

What remains outstanding, and what shapes module 4:

• The pipeline is slow and it is putting on weight. CI takes 4 minutes today; with more tests it will be 15, and Diego will stop watching it. Cache, parallelism and selective execution are needed.
• Dependencies are not under control. Nobody knows how many transitive packages get into reservalia/api, nor who maintains them, nor which exact version was used in a3f9c21.
• The pipeline has permissions over production and has never been audited. Since 03-01 we have known it stopped being an internal tool and became a critical system, and it still has no security analysis, artifact signing or component inventory.
• Database migrations are still the fragile point. They are the only reason the rollback from 03-05 might not work.

Diego: "If CI takes longer than going for a coffee, I stop watching it. And now that the deployment is mine all the way to production, I watch it far more."

Mistake 1: measuring the mean instead of percentiles. It hides precisely the users having the worst time. Mistake 2: alerting on causes. It produces noise and, at the same time, gaps in the face of unanticipated failures.

Mistake 3: defining fifteen SLOs. An SLO nobody consults disciplines nothing; two that are looked at every week change the team's behaviour. Mistake 4: setting the objective at 100%, which makes the budget zero and turns any deployment into a violation.

Mistake 5: using identifiers as a metric dimension. A businessId as a dimension multiplies the cost by 340 and makes the dashboard illegible. Mistake 6: dashboards nobody looks at, built out of everything the tool offered instead of out of what answers a question.

Mistake 7: recording only successful deployments. The change failure rate will always look wonderful and will mean nothing.

Tip 1: define the SLOs with product, not just with engineering. The reliability objective is a business decision dressed up as a technical one. Tip 2: agree the budget policy in writing and in the cold, before the first incident that puts it to the test. Tip 3: review the alerts once a month and mercilessly delete the ones that have prompted no action.

Reservalia's dashboard shows a stable p95 of 180 ms all week, but support reports three businesses complaining about slowness when opening the schedule. The team replies that "the metrics are fine". Explain what may be happening and which three changes you would make to the instrumentation.

It is the 18th of the month. The availability error budget is 78% consumed because of an RDS outage on the 6th. Product asks to deploy a large feature this week. Apply the policy from section 6, argue the decision and explain what you would do if product insists with a legitimate business reason.

Design the alert that would detect, in under five minutes, a deployment that causes 100% of the bookings of a single large business to fail, while all the other businesses work normally. Discuss why Reservalia's three current alerts would probably not detect it.

Solution 1. The most likely explanation is that the metrics are well measured and badly aggregated. A global p95 across all routes mixes thousands of cheap requests (/health, cached listings) with expensive ones: if 92% of the traffic responds in 30 ms, there is plenty of room for one specific endpoint to take several seconds without moving the global p95. On top of that, three businesses out of 340 are less than 1% of the traffic: by definition they live in the p99, invisible in the p95. And there is a third possibility: that the slowness is in the front end — resource downloads, rendering — and the API does not see it at all.

The three changes: (1) measure percentiles per route, not just globally, and bring the p95 and p99 of the critical-path routes onto the dashboard; (2) add the p99 next to the p95, because the problems support reports live there, and complement it with structured logs that record requests above a threshold (1 s, say) together with their businessId, which would let you discover in a minute that the three affected businesses are the ones with split opening hours and 400 weekly appointments; (3) instrument the client (Real User Monitoring) to measure what the browser experiences, which is the only thing the user perceives. The underlying lesson: an SLI that does not represent the user's experience is a number that reassures without informing.

Solution 2. With 78% consumed, 22% remains, so the policy places Reservalia in the 10-25% band: only fixes, reliability changes and low-risk work. The default decision is not to deploy the large feature this week, and the conversation with product should not be a negotiation of wills but the application of a previously agreed rule.

That said, there is an important and honest nuance: the budget was exhausted by an RDS outage unrelated to deployments, not by the team's changes. Blocking delivery because of an infrastructure incident punishes the wrong behaviour, and a policy perceived as unfair eventually gets ignored. Two legitimate ways out: review whether that outage should be excluded from the calculation (many teams define in advance which events are exceptional and how they are documented), or keep the restriction but accompany it with the work that resolves it — reliability work on RDS: Multi-AZ is already there, so what remains is reviewing retries, timeouts and graceful degradation.

If product insists with a legitimate reason (a dated commercial commitment), the right answer is not to skip the policy quietly, but to use the module's tools: deploy the code behind a switched-off feature flag, enable it for one business, then for 10% and only afterwards for everybody, with an immediate rollback available. That decouples the delivery date from the risk, which is precisely what all of this was built for. And the exception is recorded: who authorised it and why, so it can be reviewed in the retrospective.

Solution 3. The three current alerts would not detect it because of a dilution problem. A large business may account for 2% of total traffic; if all its bookings fail, the global error rate rises from 0.1% to 2.1%, which probably crosses the 1% threshold of Api5xxErrors… or probably does not, if the business is smaller or if the failure affects only part of its requests. With a business making up 0.5% of the traffic, the global signal is indistinguishable from noise, and the same goes for latency and for budget consumption.

The design that does detect it: an alert on the maximum error rate per business, not on the aggregate. In practice it is implemented by emitting an error metric grouped by a dimension of controlled cardinality — not businessId, which would be 340 metrics and violates the rule from section 4, but something like the business's size segment, or a counter of "distinct businesses with more than 50% failed requests in the last 5 minutes". That last formulation is the good one: a single number, cardinality one, that fires when any business is completely broken even though the aggregate looks impeccable. Condition: ≥ 1 business with more than 50% failures for 5 minutes, over a minimum of 20 requests to avoid false positives from low-traffic businesses.

An essential complement: when that alert fires, you need to be able to answer "which business?", and metrics do not give you that — the structured logs with businessId from section 2 do. It is the perfect illustration of the order of the three pillars: the metric warns, the log explains.

The loop is closed. Reservalia no longer deploys blind: it has metrics, structured logs and traces with distinct roles — the metric warns, the trace locates, the log explains; it watches the four golden signals on a dashboard that fits on one screen; it has translated "is it healthy?" into two SLOs with an error budget of 43 minutes a month that decides, through a table agreed in the cold, when to deploy freely and when to slow down; it marks every deployment on the graphs so the first question of an incident is answered in seconds; it alerts on symptoms and not on causes, with three alerts that wake people and no more; and it feeds its DORA metrics automatically from the pipeline itself, failed deployments included.

The Friday freeze has gone, and not out of bravery but the opposite: because there is now an objective rule saying when you can deploy and when you cannot, and because a deployment that is undone in four minutes and detected in two no longer deserves a ceremony. The numbers back it up: from 1.1 to 12 deployments a week, from 6.2 days to 3.5 hours of lead time, from 68 to 9 minutes of recovery. The change failure rate has stalled at 6.5% against a target of 5%, and that is the honest clue as to where the work goes next.

Because the failures that remain are no longer deployment failures. They come from a pipeline that is starting to put on weight and slow down, from dependencies that enter the artifact without anybody knowing how many there are or whose they are, from a system with permissions over production that has never had a security review, and from schema migrations that remain the only reason the go-back button might not work. Module 4, Advanced CI/CD Practices, tackles those four fronts, and it starts with the one that orders them all: its first lesson, CI/CD Pipelines, dissects the full anatomy of a pipeline — its stages, how they are orchestrated, what is parallelised, what can be skipped and how to design one that stays fast once the team has tripled in size.