The Project | About Us | Contribute | Donations | License

HOME

Your pipeline deploys and knows how to go back, but it is blind. The smoke test from 07-03 checks that the service responded correctly for the thirty seconds after the deployment; about what happens twenty minutes later, when real traffic arrives, it says nothing. And there is an even more uncomfortable question you cannot answer either: is the pipeline making things better? How many times have you deployed this week? How long does a change take from commit to production? What percentage of deployments end in a rollback? Without those numbers, the pipeline is an act of faith.

In this lab you build both directions of the feedback loop. Inwards, into the system: you will instrument Mini-Reservalia so it exposes metrics, bring up Prometheus and Grafana with docker compose, define an SLO with its error budget worked out by hand, write a symptom-based alert and fire it on purpose. Outwards, into the pipeline: you will mark deployments on the dashboard so you can correlate "we deployed" with "it got worse", calculate the four DORA metrics for your own repository with a scheduled job, and —closing the loop— make cd.yml watch the metrics after deploying and trigger the rollback on its own if things go wrong.

Objective, prerequisites and starting point
Instrumenting the server: counters and histograms
The /metrics endpoint in Prometheus format
The four golden signals on top of these metrics
Prometheus and Grafana with docker compose
The dashboard as code
The SLO and its error budget, with the arithmetic
Symptom-based alert rules
Firing the alert on purpose
Marking deployments on the dashboard
Closing the loop: the four DORA metrics of your repository
Automatic rollback driven by metrics
Final verification
Common Mistakes and Tips
Exercises
Conclusion

Objective, prerequisites and starting point

Objective. By the end you will have Mini-Reservalia exposing metrics in Prometheus format, a Grafana dashboard versioned in the repository, an SLO with an error budget, an alert you have watched fire, an automatic weekly report with the four DORA metrics of your repository, and a cd.yml that reverts on its own when the metrics get worse after a deployment.

Prerequisites. Lessons 07-01 to 07-03. Docker and docker compose working. cd.yml deploying and rollback.yml operational.

Starting point. Mini-Reservalia deployed by the pipeline to staging (port 3001) and production (port 3002).

git checkout main && git pull
git checkout -b observability

Instrumenting the server: counters and histograms

We need two kinds of metric, and it is worth understanding the difference before writing any code:

Type	What it is	Example here	How it is queried
Counter	A number that only goes up; it resets when the process restarts	Total requests by route and code	`rate()` over a window
Gauge	A number that goes up and down	Requests in flight, seconds of uptime	Direct value
Histogram	Cumulative counters per value "bucket", plus a sum and a count	Request duration	`histogram_quantile()` for percentiles

An average latency counter would be useless: the mean hides exactly what matters. If 99 requests take 10 ms and one takes 5 seconds, the mean is 60 ms and looks splendid, while one user in every hundred walks away. A histogram lets you ask about the 95th or 99th percentile, which is where the real pain lives. Lesson 03-06 explained it; now you are going to implement it.

src/metrics.js:

// src/metrics.js
// A Prometheus-format metrics registry, with no dependencies.
//
// Prometheus is plain text: each line is
//   name{label="value",...} number
// preceded by # HELP and # TYPE comments. That is all Prometheus needs to
// scrape it and store it as a time series.

/** Histogram bucket boundaries, in seconds. */
const BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10];

export class Metrics {
  /** Counters: key = "name|serialised labels" -> number */
  #counters = new Map();
  /** Histograms: key -> { buckets: number[], sum: number, count: number } */
  #histograms = new Map();
  #startedAt = Date.now();
  #inFlight = 0;

  #key(name, labels) {
    const parts = Object.entries(labels)
      .sort(([a], [b]) => a.localeCompare(b))
      .map(([k, v]) => `${k}="${String(v).replace(/["\\\n]/g, '_')}"`);
    return `${name}|${parts.join(',')}`;
  }

  increment(name, labels = {}, amount = 1) {
    const key = this.#key(name, labels);
    this.#counters.set(key, (this.#counters.get(key) ?? 0) + amount);
  }

  observe(name, labels = {}, valueSec = 0) {
    const key = this.#key(name, labels);
    let h = this.#histograms.get(key);
    if (!h) {
      h = { buckets: new Array(BUCKETS.length).fill(0), sum: 0, count: 0 };
      this.#histograms.set(key, h);
    }
    // Prometheus buckets are CUMULATIVE: the le="0.1" bucket counts
    // every observation <= 0.1, not just the ones in that interval.
    for (let i = 0; i < BUCKETS.length; i++) {
      if (valueSec <= BUCKETS[i]) h.buckets[i]++;
    }
    h.sum += valueSec;
    h.count++;
  }

  enter() { this.#inFlight++; }
  exit() { this.#inFlight--; }

  /** Serialises the whole registry in the Prometheus exposition format. */
  expose({ version = 'dev', environment = 'unknown' } = {}) {
    const lines = [];

    lines.push(
      '# HELP mini_reservalia_info Instance information (value is always 1)',
      '# TYPE mini_reservalia_info gauge',
      `mini_reservalia_info{version="${version}",environment="${environment}"} 1`,
      '',
      '# HELP mini_reservalia_uptime_seconds Seconds since the process started',
      '# TYPE mini_reservalia_uptime_seconds gauge',
      `mini_reservalia_uptime_seconds ${((Date.now() - this.#startedAt) / 1000).toFixed(0)}`,
      '',
      '# HELP http_requests_in_flight HTTP requests being served right now',
      '# TYPE http_requests_in_flight gauge',
      `http_requests_in_flight ${this.#inFlight}`,
      '',
    );

    // Counters grouped by metric name.
    const byName = new Map();
    for (const [key, value] of this.#counters) {
      const [name, labels] = key.split('|');
      if (!byName.has(name)) byName.set(name, []);
      byName.get(name).push([labels, value]);
    }
    for (const [name, series] of byName) {
      lines.push(`# HELP ${name} Cumulative counter`, `# TYPE ${name} counter`);
      for (const [labels, value] of series.sort()) {
        lines.push(labels ? `${name}{${labels}} ${value}` : `${name} ${value}`);
      }
      lines.push('');
    }

    // Histograms: _bucket (cumulative, with le="+Inf"), _sum and _count.
    const histByName = new Map();
    for (const [key, h] of this.#histograms) {
      const [name, labels] = key.split('|');
      if (!histByName.has(name)) histByName.set(name, []);
      histByName.get(name).push([labels, h]);
    }
    for (const [name, series] of histByName) {
      lines.push(`# HELP ${name} Distribution of durations in seconds`, `# TYPE ${name} histogram`);
      for (const [labels, h] of series.sort()) {
        const suffix = labels ? `,${labels}` : '';
        for (let i = 0; i < BUCKETS.length; i++) {
          lines.push(`${name}_bucket{le="${BUCKETS[i]}"${suffix}} ${h.buckets[i]}`);
        }
        lines.push(`${name}_bucket{le="+Inf"${suffix}} ${h.count}`);
        lines.push(labels ? `${name}_sum{${labels}} ${h.sum.toFixed(6)}` : `${name}_sum ${h.sum.toFixed(6)}`);
        lines.push(labels ? `${name}_count{${labels}} ${h.count}` : `${name}_count ${h.count}`);
      }
      lines.push('');
    }

    return `${lines.join('\n')}\n`;
  }

  reset() {
    this.#counters.clear();
    this.#histograms.clear();
    this.#inFlight = 0;
  }
}

export const metrics = new Metrics();

And now the middleware. In src/server.js, wrap the handler:

// src/server.js  (additions from 07-04)
import { metrics } from './metrics.js';

export const ENVIRONMENT = process.env.ENVIRONMENT ?? 'local';

/**
 * Normalises the path so it can be used as a LABEL.
 *
 * CRITICAL: never use the raw URL as a label. Every distinct value creates a new
 * TIME SERIES in Prometheus. With `/api/slots?date=...` you would end up with one
 * series per date queried: thousands of series, memory through the roof and
 * queries that never finish. This is the mistake known as "cardinality
 * explosion" and it takes down entire Prometheus installations.
 */
function normaliseRoute(pathname) {
  const known = ['/health', '/metrics', '/api/slots', '/api/appointments'];
  return known.includes(pathname) ? pathname : '/other';
}

function instrument(handler) {
  return async (req, res) => {
    const startedAt = process.hrtime.bigint();
    metrics.enter();

    // 'finish' is emitted once the response has been fully sent, which is
    // the correct moment to measure end-to-end latency.
    res.once('finish', () => {
      const durationSec = Number(process.hrtime.bigint() - startedAt) / 1e9;
      const url = new URL(req.url, 'http://internal');
      const labels = {
        method: req.method,
        route: normaliseRoute(url.pathname),
        code: String(res.statusCode),
      };
      metrics.increment('http_requests_total', labels);
      metrics.observe('http_duration_seconds', {
        method: labels.method,
        route: labels.route,
      }, durationSec);
      if (res.statusCode >= 500) {
        metrics.increment('http_errors_total', { route: labels.route, type: 'server' });
      } else if (res.statusCode >= 400) {
        metrics.increment('http_errors_total', { route: labels.route, type: 'client' });
      }
      metrics.exit();
    });

    return handler(req, res);
  };
}

The `/metrics` endpoint in Prometheus format

Wrap the handler when the server is created and add the exposition route. Prometheus receives nothing: it goes and fetches it (the pull model), issuing a GET to /metrics every few seconds. That inversion is what means the application needs to know nothing about the monitoring system: it only has to publish some text.

export function createServer({ repository, openingHours = DEFAULT_OPENING_HOURS } = {}) {
  if (!repository) throw new Error('createServer requires a repository');

  const handler = async (req, res) => {
    const url = new URL(req.url, `http://${req.headers.host ?? 'localhost'}`);

    // --- Metrics endpoint ---
    if (req.method === 'GET' && url.pathname === '/metrics') {
      const body = metrics.expose({ version: VERSION, environment: ENVIRONMENT });
      res.writeHead(200, {
        // This exact content-type is the one Prometheus expects.
        'content-type': 'text/plain; version=0.0.4; charset=utf-8',
        'content-length': Buffer.byteLength(body),
      });
      return res.end(body);
    }

    // --- Artificial delay for the exercise in section 9 ---
    // Controlled by an environment variable: it can never be switched on by accident.
    const delay = Number(process.env.ARTIFICIAL_DELAY_MS ?? 0);
    if (delay > 0 && url.pathname === '/api/slots') {
      await new Promise((r) => setTimeout(r, delay));
    }

    /* ... the rest of the routes, unchanged ... */
  };

  return http.createServer(instrument(handler));
}

Add its test too, because coverage still has a threshold:

// test/metrics.test.js
import test, { describe } from 'node:test';
import assert from 'node:assert/strict';
import { Metrics } from '../src/metrics.js';

describe('metrics registry', () => {
  test('a counter accumulates per label combination', () => {
    const m = new Metrics();
    m.increment('http_requests_total', { route: '/health', code: '200' });
    m.increment('http_requests_total', { route: '/health', code: '200' });
    m.increment('http_requests_total', { route: '/health', code: '500' });
    const text = m.expose();
    assert.match(text, /http_requests_total\{code="200",route="\/health"\} 2/);
    assert.match(text, /http_requests_total\{code="500",route="\/health"\} 1/);
  });

  test('histogram buckets are CUMULATIVE', () => {
    const m = new Metrics();
    m.observe('http_duration_seconds', { route: '/api/slots' }, 0.03);
    const text = m.expose();
    // 0.03 s does NOT fall in le=0.025 but it DOES in le=0.05 and every larger one.
    assert.match(text, /_bucket\{le="0.025",route="\/api\/slots"\} 0/);
    assert.match(text, /_bucket\{le="0.05",route="\/api\/slots"\} 1/);
    assert.match(text, /_bucket\{le="\+Inf",route="\/api\/slots"\} 1/);
  });

  test('the histogram exposes _sum and _count', () => {
    const m = new Metrics();
    m.observe('http_duration_seconds', {}, 0.1);
    m.observe('http_duration_seconds', {}, 0.3);
    const text = m.expose();
    assert.match(text, /http_duration_seconds_count 2/);
    assert.match(text, /http_duration_seconds_sum 0\.400000/);
  });

  test('labels are sanitised so the format does not break', () => {
    const m = new Metrics();
    m.increment('test_total', { route: 'with"quotes' });
    assert.doesNotMatch(m.expose(), /route="with"quotes"/);
  });
});

Check it locally:

npm test                      # the new tests green
DATABASE_URL='sqlite:/tmp/m.db' ENVIRONMENT=local npm start &
for i in $(seq 1 20); do curl -s "localhost:3000/api/slots?date=2026-03-02" > /dev/null; done
curl -s localhost:3000/api/slots > /dev/null      # a 400
curl -s localhost:3000/metrics | head -30

What you should see:

# HELP mini_reservalia_info Instance information (value is always 1)
# TYPE mini_reservalia_info gauge
mini_reservalia_info{version="dev",environment="local"} 1

# HELP mini_reservalia_uptime_seconds Seconds since the process started
# TYPE mini_reservalia_uptime_seconds gauge
mini_reservalia_uptime_seconds 34

# HELP http_requests_in_flight HTTP requests being served right now
# TYPE http_requests_in_flight gauge
http_requests_in_flight 1

# HELP http_requests_total Cumulative counter
# TYPE http_requests_total counter
http_requests_total{code="200",method="GET",route="/api/slots"} 20
http_requests_total{code="400",method="GET",route="/api/slots"} 1

# HELP http_duration_seconds Distribution of durations in seconds
# TYPE http_duration_seconds histogram
http_duration_seconds_bucket{le="0.005",method="GET",route="/api/slots"} 19
...

The four golden signals on top of these metrics

Lesson 03-06 introduced the four golden signals of SRE. Here they are, mapped to concrete queries you can copy and paste:

Signal	Question it answers	PromQL query
Latency	How long do the requests that do work take?	`histogram_quantile(0.95, sum by (le, route) (rate(http_duration_seconds_bucket[5m])))`
Traffic	How much demand is there?	`sum by (route) (rate(http_requests_total[5m]))`
Errors	What fraction fails?	`sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))`
Saturation	How full is the system?	`http_requests_in_flight`

Two nuances that separate a useful dashboard from a decorative one:

Latency for successful requests only. A 500 returned in 2 ms improves your 95th percentile and makes you believe everything is fast. Filter it: http_duration_seconds_bucket{code=~"2.."} if you add the code as a histogram label (at the cost of cardinality).
5xx errors, not 4xx. A 400 because the client sent a malformed date is not your failure: it is your validation working. Mixing them makes your error rate go up whenever somebody scans your API, and gets you used to ignoring it. That is why http_errors_total separates type="client" from type="server".

Prometheus and Grafana with `docker compose`

observability/docker-compose.yml:

# observability/docker-compose.yml
# Local observability stack: Prometheus (collects and stores) + Grafana (draws).
services:
  prometheus:
    image: prom/prometheus:v2.53.0
    container_name: prometheus
    restart: unless-stopped
    ports: ['9090:9090']
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
      # Required to reload the configuration without restarting:
      #   curl -X POST http://localhost:9090/-/reload
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro
      - prometheus-data:/prometheus
    # Lets Prometheus reach the app containers published on the host
    # (Linux; on Docker Desktop host.docker.internal already exists).
    extra_hosts:
      - 'host.docker.internal:host-gateway'

  grafana:
    image: grafana/grafana:11.1.0
    container_name: grafana
    restart: unless-stopped
    ports: ['3000:3000']
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_USERS_ALLOW_SIGN_UP: 'false'
      GF_AUTH_ANONYMOUS_ENABLED: 'true'
      GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
    volumes:
      # Provisioning as code: Grafana reads these files at startup.
      # NOTHING is configured by clicking around the interface: if it is not
      # in the repo, it does not exist (the same principle as infrastructure as code, 03-03).
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    depends_on: [prometheus]

volumes:
  prometheus-data:
  grafana-data:

observability/prometheus.yml:

# observability/prometheus.yml
global:
  scrape_interval: 15s       # how often targets are scraped
  evaluation_interval: 15s   # how often alert rules are evaluated
  external_labels:
    project: mini-reservalia

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  # 1. Prometheus itself (always useful for knowing whether it has gone down)
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  # 2. Mini-Reservalia, one target per environment.
  #    The `environment` label lets you compare staging and production
  #    on the same dashboard, which is how you spot a regression before it hurts.
  - job_name: mini-reservalia
    metrics_path: /metrics
    scrape_interval: 10s
    scrape_timeout: 5s
    static_configs:
      - targets: ['host.docker.internal:3001']
        labels: { environment: staging }
      - targets: ['host.docker.internal:3002']
        labels: { environment: production }
    relabel_configs:
      # `instance` defaults to "host:port", which is unreadable on dashboards.
      - source_labels: [environment]
        target_label: instance

observability/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    uid: prometheus-mini

observability/grafana/provisioning/dashboards/dashboards.yml:

apiVersion: 1
providers:
  - name: 'mini-reservalia'
    folder: 'Mini-Reservalia'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false      # changes are made in the repo, not in the UI
    options:
      path: /var/lib/grafana/dashboards

Bring the stack up:

docker compose -f observability/docker-compose.yml up -d
docker compose -f observability/docker-compose.yml ps

What you should see. At http://localhost:9090/targets, the table of targets:

Endpoint	State	Labels
`http://host.docker.internal:3001/metrics`	UP	`environment="staging"`
`http://host.docker.internal:3002/metrics`	UP	`environment="production"`

If either is DOWN with connection refused, that environment's container is not running: deploy it with ./scripts/deploy.sh as in 07-03.

Generate some traffic and try a query at http://localhost:9090/graph:

for i in $(seq 1 200); do
  curl -s "localhost:3002/api/slots?date=2026-03-02&duration=60" > /dev/null
  sleep 0.2
done

sum by (environment) (rate(http_requests_total[1m]))

What you should see: a line with a value close to 5 requests per second for as long as the loop runs.

The dashboard as code

A Grafana dashboard is JSON. Having that JSON in the repository and provisioned automatically means the dashboard is reviewed in a PR, versioned and restored by itself if somebody breaks it. A dashboard built by hand in the interface is knowledge that exists in a database nobody backs up.

We are not going to paste the 500 lines of a complete dashboard here. We are going to look at one entire panel, which is where the lesson lives, and describe the rest.

observability/grafana/dashboards/mini-reservalia.json (fragment with one complete panel):

{
  "uid": "mini-reservalia",
  "title": "Mini-Reservalia · Golden signals",
  "tags": ["mini-reservalia", "slo"],
  "timezone": "browser",
  "refresh": "10s",
  "time": { "from": "now-1h", "to": "now" },
  "templating": {
    "list": [
      {
        "name": "environment",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prometheus-mini" },
        "query": "label_values(http_requests_total, environment)",
        "current": { "text": "production", "value": "production" },
        "includeAll": false
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Latency of /api/slots (p50 · p95 · p99)",
      "description": "Percentiles computed over the histogram buckets. The red line is the SLO target (300 ms).",
      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus-mini" },
      "targets": [
        {
          "refId": "A",
          "expr": "histogram_quantile(0.50, sum by (le) (rate(http_duration_seconds_bucket{route=\"/api/slots\", environment=\"$environment\"}[5m])))",
          "legendFormat": "p50"
        },
        {
          "refId": "B",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_duration_seconds_bucket{route=\"/api/slots\", environment=\"$environment\"}[5m])))",
          "legendFormat": "p95"
        },
        {
          "refId": "C",
          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_duration_seconds_bucket{route=\"/api/slots\", environment=\"$environment\"}[5m])))",
          "legendFormat": "p99"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "min": 0,
          "custom": { "lineWidth": 2, "fillOpacity": 8, "showPoints": "never" },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "red", "value": 0.3 }
            ]
          }
        }
      },
      "options": {
        "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] },
        "tooltip": { "mode": "multi", "sort": "desc" }
      }
    }
  ],
  "annotations": {
    "list": [
      {
        "name": "Deployments",
        "datasource": { "type": "prometheus", "uid": "prometheus-mini" },
        "enable": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "expr": "changes(mini_reservalia_uptime_seconds{environment=\"$environment\"}[2m]) > 0",
        "titleFormat": "Deployment",
        "textFormat": "New version in {{environment}}"
      }
    ]
  },
  "schemaVersion": 39,
  "version": 1
}

Read it carefully, because that single panel contains every decision that matters:

Element	Why it is there
`histogram_quantile(0.95, ...)` over `rate(..._bucket[5m])`	The only correct way to get percentiles out of a Prometheus histogram. `rate` before `histogram_quantile`, never the other way round
`sum by (le)`	Aggregates the instances while keeping the `le` label. Lose it and `histogram_quantile` returns `NaN`; it is the number one mistake
The three percentiles together	p50 tells you how the typical case is doing; p99 tells you how the worst 1 % is doing. The gap between them is the signature of a queueing problem
`thresholds` at 0.3	The SLO target drawn on the panel. A panel without the target line forces you to remember the number
The `$environment` variable	The same dashboard serves staging and production
`annotations` with `changes(...uptime_seconds...)`	Marks deployments: `uptime_seconds` resets when the process starts
`unit: "s"`	Without a unit, Grafana shows `0.087` and you have to translate it in your head every time

The dashboard's other panels, with their queries (build them as an exercise or copy them from the same pattern):

Panel	Type	Query
Traffic by route	timeseries	`sum by (route) (rate(http_requests_total{environment="$environment"}[5m]))`
5xx error rate	stat	`sum(rate(http_requests_total{code=~"5..",environment="$environment"}[5m])) / sum(rate(http_requests_total{environment="$environment"}[5m]))`
Requests in flight	timeseries	`http_requests_in_flight{environment="$environment"}`
SLO compliance (7 d)	gauge	`sum(rate(http_duration_seconds_bucket{le="0.25",route="/api/slots",environment="$environment"}[7d])) / sum(rate(http_duration_seconds_count{route="/api/slots",environment="$environment"}[7d]))`
Error budget remaining	gauge	`1 - ((1 - <the previous one>) / 0.005)`
Deployed version	stat	`mini_reservalia_info{environment="$environment"}` with legend `{{version}}`

Open http://localhost:3000 (admin/admin) and go to Dashboards → Mini-Reservalia. What you should see: the dashboard already exists without you importing anything. Modify it from the interface: it will not let you save (allowUiUpdates: false). That is deliberate, and it is the difference between a dashboard that is code and one that is a memory.

The SLO and its error budget, with the arithmetic

An SLO without a calculated error budget is a wish. Let us make the numbers explicit.

Mini-Reservalia's SLO:

99.5 % of requests to /api/slots must respond correctly in under 300 ms, measured over a rolling window of 7 days.

Four elements, and all four must be there: the indicator (/api/slots latency), the threshold (300 ms), the target (99.5 %) and the window (7 days). An SLO missing any of them cannot be evaluated.

Why 99.5 % and not 99.99 %. Every additional nine multiplies the cost by something close to ten: redundancy, on-call rotas, complexity. Mini-Reservalia is a booking tool for small businesses; one slow request in every two hundred is perfectly tolerable and nobody cancels a subscription over it. Choosing the lowest achievable target that keeps users happy is an engineering decision, not laziness.

The error budget, with the full arithmetic:

Observed traffic:       20 requests/minute to /api/slots
Window:                 7 days

Requests in the window:
    20 req/min × 60 min × 24 h × 7 d = 201,600 requests

Target: 99.5 % correct and fast
    Error budget = 100 % − 99.5 % = 0.5 %
    0.005 × 201,600 = 1,008 requests

  → We can afford 1,008 slow or failed requests in 7 days.

Translated into time, which is how it really sinks in:

If ALL requests fail during a total outage:
    1,008 requests ÷ 20 req/min = 50.4 minutes

  → The entire budget is worth roughly 50 minutes of complete outage
    every 7 days. Or, spread out: about 6 slow requests per hour.

And now the part that turns the budget into a decision-making tool:

Budget consumed	What it means	What you do
< 50 %	Plenty of margin	Deploy as normal. If consumption is chronically low, the SLO is too lax: raise it
50-75 %	Pay attention	Keep deploying, but reliability improvements move up the priority list
75-100 %	Alert	Low-risk changes only. Anything touching the affected route goes out as a canary
> 100 % (exhausted)	The SLO has been missed	Feature freeze: the team works on reliability until margin is recovered

That is the real value of an error budget: it turns "shall we deploy on Friday?" into a question with a numeric answer instead of a clash of opinions. And it works in both directions: with 90 % of the budget untouched, the answer is "yes, go ahead", and that needs saying too.

Record it in the repository, observability/SLO.md:

# Mini-Reservalia SLO

| Field | Value |
|---|---|
| Service | Mini-Reservalia · API |
| Indicator (SLI) | Proportion of requests to `/api/slots` with a 2xx code and latency < 300 ms |
| Target (SLO) | 99.5 % |
| Window | 7 rolling days |
| Error budget | 0.5 % ≈ 1,008 requests ≈ 50 min of total outage |
| Owner | Platform team |
| Review | Quarterly |

## SLI query

sum(rate(http_duration_seconds_bucket{le="0.25", route="/api/slots", code=~"2.."}[7d])) / sum(rate(http_duration_seconds_count{route="/api/slots"}[7d]))

> Note: the `le="0.25"` bucket is used because it is the closest bucket boundary
> below 300 ms. Histograms can only answer about the boundaries that exist. If
> the SLO were exactly 300 ms, we would have to **add a 0.3 bucket** to
> `src/metrics.js`. This is the real trade-off of histograms: the buckets have
> to be chosen before you know what you are going to ask.

## Budget policy

- < 50 % consumed: normal deployment.
- 50-75 %: reliability moves up the backlog.
- 75-100 %: low-risk changes only, as canaries.
- \> 100 %: feature freeze until margin is recovered.

That warning about the 0.25 bucket is not a minor detail: it is the sort of thing you discover three months after defining the SLO, when there is already historical data that cannot be recomputed. Add the 0.3 bucket to BUCKETS in src/metrics.js now, while you still can.

Symptom-based alert rules

The golden rule from 03-06: alert on symptoms, not on causes. "The CPU is at 90 %" is not a problem if nobody notices; "the 95th percentile of latency has been above 300 ms for five minutes" is. Alerting on causes produces noise; alerting on symptoms produces calls that are worth taking.

observability/alerts.yml:

# observability/alerts.yml
groups:
  - name: mini-reservalia-symptoms
    interval: 30s
    rules:
      # ---------------------------------------------------------------
      # SYMPTOM 1: users are waiting too long.
      # ---------------------------------------------------------------
      - alert: HighSlotsLatency
        expr: |
          histogram_quantile(0.95,
            sum by (le, environment) (
              rate(http_duration_seconds_bucket{route="/api/slots"}[5m])
            )
          ) > 0.3
        # `for` is what separates a useful alert from a noise generator:
        # the condition must hold for 5 minutes straight. A 20 s spike
        # during a deployment does not wake anybody up.
        for: 5m
        labels:
          severity: warning
          team: platform
          slo: slots-latency
        annotations:
          summary: 'p95 of /api/slots above 300 ms in {{ $labels.environment }}'
          description: >-
            The 95th percentile has been at {{ $value | humanizeDuration }} for 5 minutes,
            above the SLO target of 300 ms.
            The error budget is at risk.
          runbook: 'https://github.com/OWNER/mini-reservalia/blob/main/observability/RUNBOOK.md#high-latency'

      # ---------------------------------------------------------------
      # SYMPTOM 2: the service is returning errors of its own.
      # ---------------------------------------------------------------
      - alert: HighErrorRate
        expr: |
          (
            sum by (environment) (rate(http_requests_total{code=~"5.."}[5m]))
            /
            sum by (environment) (rate(http_requests_total[5m]))
          ) > 0.01
        for: 3m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: 'More than 1 % of 5xx errors in {{ $labels.environment }}'
          description: 'Current rate: {{ $value | humanizePercentage }}. Threshold: 1 %.'
          runbook: 'https://github.com/OWNER/mini-reservalia/blob/main/observability/RUNBOOK.md#5xx-errors'

      # ---------------------------------------------------------------
      # SYMPTOM 3: there is no service at all.
      # ---------------------------------------------------------------
      - alert: ServiceDown
        expr: up{job="mini-reservalia"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Mini-Reservalia is not responding in {{ $labels.environment }}'
          description: 'Prometheus has been unable to scrape /metrics for 1 minute.'

      # ---------------------------------------------------------------
      # SYMPTOM 4 (predictive): the error budget is burning down fast.
      # This is "burn rate": it does not alert because things are bad, it alerts
      # because they are heading that way. That is what lets you act in time.
      # ---------------------------------------------------------------
      - alert: ErrorBudgetBurningFast
        expr: |
          (
            1 - (
              sum by (environment) (rate(http_duration_seconds_bucket{le="0.3", route="/api/slots"}[1h]))
              /
              sum by (environment) (rate(http_duration_seconds_count{route="/api/slots"}[1h]))
            )
          ) > (14.4 * 0.005)
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'Accelerated error budget consumption in {{ $labels.environment }}'
          description: >-
            At the rate of the last hour, the 7-day budget would be exhausted
            in about 12 hours (burn rate 14.4x). Google SRE standard threshold.

The 14.4 in that last alert deserves an explanation, because it looks like a magic number: it is the factor that exhausts a 30-day budget in 2 days, and it is the SRE Workbook's standard multiplier for the fast-page alert. With our 7-day window, a burn rate of 14.4× exhausts the budget in about 12 hours. The idea is that it warns you while you can still do something, not once you have already missed the target.

Reload Prometheus and check:

docker compose -f observability/docker-compose.yml restart prometheus
# or, without restarting:
curl -X POST http://localhost:9090/-/reload

# Validate the syntax BEFORE reloading (always do this):
docker run --rm -v "$PWD/observability:/o" --entrypoint promtool \
  prom/prometheus:v2.53.0 check rules /o/alerts.yml
# Checking /o/alerts.yml
#   SUCCESS: 4 rules found

That promtool check rules belongs in your ci.yml: alert rules are code and they are validated like code. A rule with a syntax error makes Prometheus ignore the entire file, and you find out on the day you needed the alert.

What you should see at http://localhost:9090/alerts: all four rules in the Inactive state (green).

Firing the alert on purpose

An alert you have never watched fire is a hypothesis. Let us test it.

# 1. Redeploy staging with a 500 ms artificial delay
docker rm -f mini-reservalia-staging
docker run -d --name mini-reservalia-staging \
  -p 3001:3000 \
  -e ENVIRONMENT=staging \
  -e ARTIFICIAL_DELAY_MS=500 \
  -e APP_VERSION=slow \
  ghcr.io/YOUR_USERNAME/mini-reservalia@sha256:YOUR_DIGEST

# 2. Generate sustained traffic for 7 minutes
END=$((SECONDS+420))
while [ $SECONDS -lt $END ]; do
  curl -s "localhost:3001/api/slots?date=2026-03-02" > /dev/null
  sleep 0.5
done

What you should see, in chronological order:

Moment	`http://localhost:9090/alerts`	Grafana
t = 0	`HighSlotsLatency` Inactive	p95 jumps to ~0.5 s
t ≈ 40 s	PENDING (amber) — the condition holds but not yet for 5 min	The line crosses the red 0.3 threshold
t ≈ 5 min 40 s	FIRING (red)	The line is still above
After removing the delay, t + ~1 min	Back to Inactive	The line comes down

That intermediate PENDING state is the for: 5m. It is worth watching: it is the difference between an alerting system people attend to and one people mute. Without for, a three-second spike during a routine deployment would have fired the alert.

Try the error alert too:

docker rm -f mini-reservalia-staging   # ServiceDown goes to FIRING within 1 min

And restore the good version:

IMAGE=ghcr.io/YOUR_USERNAME/mini-reservalia@sha256:GOOD \
ENVIRONMENT=staging PORT=3001 ./scripts/deploy.sh

Write the runbook the annotations point at as well, observability/RUNBOOK.md, because an alert with no runbook forces you to improvise at 3 in the morning:

# Mini-Reservalia runbook

## HighSlotsLatency

**Symptom:** p95 of `/api/slots` > 300 ms for 5 minutes.
**Impact:** users see the slot list with a noticeable delay. It consumes error budget.

**Diagnosis, in order:**
1. Has there been a recent deployment? Check the dashboard annotations.
   `gh run list --workflow=cd.yml --limit 5`
2. Has traffic gone up? "Traffic by route" panel. If so, it is capacity, not a regression.
3. Are requests in flight high? "Saturation" panel. If so, there is queueing.
4. `docker logs --tail 200 mini-reservalia-production`

**Mitigation:**
- If it coincides with a deployment: **roll back first, investigate afterwards**.
  `gh workflow run rollback.yml -f environment=production -f digest=<previous> -f reason="HighSlotsLatency"`
- If it is load: scale up (more replicas / more resources).

**Escalation:** if it has not been mitigated within 30 minutes, notify the service owner.

## 5xx errors

**Symptom:** more than 1 % of 5xx responses for 3 minutes.
**First action:** `docker logs --tail 200` and look for `Unhandled error`.
**Most frequent cause:** the database is unreachable (volume with wrong permissions after a deployment).

Marking deployments on the dashboard

The most frequent question in an incident is: "did we change anything?". A dashboard that overlays deployments on top of the metrics answers it at a glance.

We already have an implicit marker —changes(mini_reservalia_uptime_seconds[2m]), because the counter resets when the process starts— but it is indirect and carries no metadata. Let us send an explicit annotation from cd.yml.

Add this at the end of the production job:

      - name: Annotate the deployment in Grafana
        if: always() && vars.GRAFANA_URL != ''
        continue-on-error: true    # a failed annotation must NOT bring the deployment down
        env:
          GRAFANA_URL: ${{ vars.GRAFANA_URL }}
          GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}
        run: |
          set -Eeuo pipefail
          NOW_MS=$(( $(date +%s) * 1000 ))
          STATUS="${{ job.status }}"
          COLOUR=$([ "$STATUS" = "success" ] && echo "green" || echo "red")

          curl -sS -X POST "${GRAFANA_URL}/api/annotations" \
            -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
            -H 'Content-Type: application/json' \
            -d @- <<JSON
          {
            "dashboardUID": "mini-reservalia",
            "time": ${NOW_MS},
            "timeEnd": ${NOW_MS},
            "tags": ["deployment", "production", "${STATUS}", "${COLOUR}"],
            "text": "<b>Deployment ${STATUS}</b><br/>Commit: ${{ needs.prepare.outputs.commit }}<br/>Digest: <code>${{ needs.prepare.outputs.digest }}</code><br/>By: @${{ github.actor }}<br/><a href='${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'>View the run</a>"
          }
          JSON
          echo "Annotation sent to Grafana."

Set it up:

# In Grafana: Administration > Service accounts > Add > Editor role > Add token
gh variable set GRAFANA_URL --body "http://localhost:3000"
gh secret set GRAFANA_TOKEN --body "glsa_xxxxx"

Add the annotation layer to the dashboard:

{
  "name": "Deployments (pipeline)",
  "datasource": { "type": "grafana", "uid": "-- Grafana --" },
  "enable": true,
  "iconColor": "rgba(0, 211, 255, 1)",
  "target": { "type": "tags", "matchAny": false, "tags": ["deployment", "production"] }
}

What you should see: a vertical line with a triangle at the base of the graph at the exact moment of each deployment; hover over it and you get the commit, the digest, who approved it and a link to the run.

And now for the part that really teaches something. Redeploy the version with ARTIFICIAL_DELAY_MS=500 and look at the dashboard: you will see the deployment's vertical line and, exactly from that point onwards, the p95 curve climbing. That image —the moment of the change and the moment of the degradation lining up— is what turns a half-hour argument ("I doubt it's ours, must be the network") into a thirty-second decision. It is probably the best value-for-effort ratio in the whole lesson: twenty lines of curl in cd.yml.

Real equivalent in Reservalia. Reservalia's cd.yml sends the annotation to Grafana Cloud and, on top of that, a deployment event to the APM tool, so traces end up tagged with the version. With that, "since when has it been slow?" is answered by filtering by version instead of by time.

Closing the loop: the four DORA metrics of your repository

So far you have measured the system. Now we are going to measure the process, which is what 01-05 introduced with Reservalia's baseline. The four metrics and how they are calculated from the GitHub API:

DORA metric	Definition	Source in GitHub
Deployment frequency	Production deployments per unit of time	Deployments with `environment=production` and state `success`
Lead time for changes	From commit to production	`commit.author.date` → `deployment_status.created_at`
Change failure rate	% of deployments that need remediation	Failed deployments + `rollback.yml` runs ÷ total
Time to restore	From failure to recovery	From the failed deployment to the next successful one

scripts/dora.js:

#!/usr/bin/env node
// scripts/dora.js
// Computes the four DORA metrics of THIS repository using the GitHub API.
//
// Usage:
//   GH_TOKEN=... REPO=owner/repo DAYS=30 node scripts/dora.js
//
// It relies on the Deployments GitHub creates automatically when a job uses
// `environment:`. That is why the cd.yml from 07-03 generates them without a
// single extra line: using Environments gives you traceability for free.

const TOKEN = process.env.GH_TOKEN ?? process.env.GITHUB_TOKEN;
const REPO = process.env.REPO ?? process.env.GITHUB_REPOSITORY;
const DAYS = Number(process.env.DAYS ?? 30);
const ENVIRONMENT = process.env.PRODUCTION_ENVIRONMENT ?? 'production';

if (!TOKEN || !REPO) {
  console.error('GH_TOKEN and/or REPO (owner/repo) are missing');
  process.exit(2);
}

const SINCE = new Date(Date.now() - DAYS * 24 * 3600 * 1000);

async function api(path) {
  const response = await fetch(`https://api.github.com${path}`, {
    headers: {
      authorization: `Bearer ${TOKEN}`,
      accept: 'application/vnd.github+json',
      'x-github-api-version': '2022-11-28',
    },
  });
  if (!response.ok) {
    throw new Error(`GitHub API ${response.status} at ${path}: ${await response.text()}`);
  }
  return response.json();
}

// ---------------------------------------------------------------------------
// 1. Collect the deployments to the production environment
// ---------------------------------------------------------------------------
const deployments = [];
for (let page = 1; page <= 5; page++) {
  const batch = await api(`/repos/${REPO}/deployments?environment=${ENVIRONMENT}&per_page=100&page=${page}`);
  if (batch.length === 0) break;
  for (const d of batch) {
    if (new Date(d.created_at) < SINCE) continue;
    const statuses = await api(`/repos/${REPO}/deployments/${d.id}/statuses?per_page=100`);
    const finals = statuses.filter((s) => ['success', 'failure', 'error'].includes(s.state));
    if (finals.length === 0) continue;
    const final = finals[0]; // the API returns the most recent first
    deployments.push({
      id: d.id,
      sha: d.sha,
      createdAt: new Date(d.created_at),
      finishedAt: new Date(final.created_at),
      success: final.state === 'success',
    });
  }
  if (batch.length < 100) break;
}
deployments.sort((a, b) => a.finishedAt - b.finishedAt);

if (deployments.length === 0) {
  console.log(`No deployments to "${ENVIRONMENT}" in the last ${DAYS} days.`);
  process.exit(0);
}

// ---------------------------------------------------------------------------
// 2. Metric 1: deployment frequency
// ---------------------------------------------------------------------------
const successful = deployments.filter((d) => d.success);
const perWeek = (successful.length / DAYS) * 7;

// ---------------------------------------------------------------------------
// 3. Metric 2: lead time (commit -> production)
// ---------------------------------------------------------------------------
const leadTimes = [];
for (const d of successful) {
  try {
    const commit = await api(`/repos/${REPO}/commits/${d.sha}`);
    const commitDate = new Date(commit.commit.author.date);
    const hours = (d.finishedAt - commitDate) / 3_600_000;
    if (hours >= 0 && hours < 24 * 90) leadTimes.push(hours);
  } catch {
    /* commit removed by a force-push or similar: ignored */
  }
}
const median = (xs) => {
  if (xs.length === 0) return 0;
  const s = [...xs].sort((a, b) => a - b);
  const m = Math.floor(s.length / 2);
  return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2;
};
const medianLead = median(leadTimes);

// ---------------------------------------------------------------------------
// 4. Metric 3: change failure rate
//    A deployment "fails" if its state is failure/error OR if it was followed
//    by a rollback. The second is the one most people forget to count, and it
//    is exactly the worst case: the deployment "worked" but broke something.
// ---------------------------------------------------------------------------
const runs = await api(
  `/repos/${REPO}/actions/workflows/rollback.yml/runs?per_page=100&created=%3E${SINCE.toISOString().slice(0, 10)}`,
).catch(() => ({ workflow_runs: [] }));
const rollbacks = (runs.workflow_runs ?? []).filter((r) => r.conclusion === 'success');

const failed = deployments.filter((d) => !d.success).length;
const cfr = ((failed + rollbacks.length) / deployments.length) * 100;

// ---------------------------------------------------------------------------
// 5. Metric 4: time to restore
//    From the first failed deployment to the next successful one.
// ---------------------------------------------------------------------------
const restores = [];
for (let i = 0; i < deployments.length; i++) {
  if (deployments[i].success) continue;
  const next = deployments.slice(i + 1).find((d) => d.success);
  if (next) restores.push((next.finishedAt - deployments[i].finishedAt) / 60_000);
}
for (const r of rollbacks) {
  const minutes = (new Date(r.updated_at) - new Date(r.created_at)) / 60_000;
  if (minutes > 0 && minutes < 24 * 60) restores.push(minutes);
}
const medianRestore = median(restores);

// ---------------------------------------------------------------------------
// 6. Classification (State of DevOps report thresholds)
// ---------------------------------------------------------------------------
const frequencyLevel = perWeek >= 7 ? 'Elite' : perWeek >= 1 ? 'High' : perWeek >= 0.25 ? 'Medium' : 'Low';
const leadLevel = medianLead < 24 ? 'Elite' : medianLead < 168 ? 'High' : medianLead < 720 ? 'Medium' : 'Low';
const cfrLevel = cfr <= 5 ? 'Elite' : cfr <= 10 ? 'High' : cfr <= 15 ? 'Medium' : 'Low';
const restoreLevel = medianRestore < 60 ? 'Elite' : medianRestore < 1440 ? 'High' : 'Medium';

const fmtHours = (h) => (h < 1 ? `${(h * 60).toFixed(0)} min` : h < 48 ? `${h.toFixed(1)} h` : `${(h / 24).toFixed(1)} d`);

const report = `## 📊 DORA metrics · last ${DAYS} days

| Metric | Value | Level |
|---|---|---|
| **Deployment frequency** | ${perWeek.toFixed(1)} / week | ${frequencyLevel} |
| **Lead time (median)** | ${fmtHours(medianLead)} | ${leadLevel} |
| **Change failure rate** | ${cfr.toFixed(1)} % | ${cfrLevel} |
| **Time to restore (median)** | ${medianRestore.toFixed(0)} min | ${restoreLevel} |

<details><summary>Calculation detail</summary>

- Deployments to \`${ENVIRONMENT}\` analysed: **${deployments.length}** (${successful.length} successful, ${failed} failed)
- Successful rollback runs: **${rollbacks.length}**
- Lead time samples: ${leadTimes.length}
- Restore samples: ${restores.length}
- Window: since ${SINCE.toISOString().slice(0, 10)}

</details>

> All four are computed from the GitHub API. Frequency and lead time measure
> **speed**; CFR and time to restore measure **stability**. Improving the first
> pair while worsening the second is not an improvement: all four are read together.
`;

console.log(report);
if (process.env.GITHUB_STEP_SUMMARY) {
  const { appendFile } = await import('node:fs/promises');
  await appendFile(process.env.GITHUB_STEP_SUMMARY, report);
}

The scheduled workflow, .github/workflows/dora.yml:

name: DORA Metrics

on:
  schedule:
    # Mondays at 08:00 UTC. Careful: `schedule` ALWAYS uses UTC and GitHub
    # may delay it by several minutes if there is a queue. Do not use it for
    # anything that depends on punctuality.
    - cron: '0 8 * * 1'
  workflow_dispatch:
    inputs:
      days:
        description: 'Analysis window in days'
        default: '30'
        type: string

permissions:
  contents: read
  deployments: read
  actions: read

jobs:
  calculate:
    name: Calculate DORA
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }

      - name: Compute the four metrics
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          REPO: ${{ github.repository }}
          DAYS: ${{ inputs.days || '30' }}
        run: node scripts/dora.js | tee dora-report.md

      - name: Store the historical report
        uses: actions/upload-artifact@v4
        with:
          name: dora-${{ github.run_id }}
          path: dora-report.md
          retention-days: 90

      # Optional but strongly recommended: open/update a pinned issue
      # so the metrics can be seen without hunting for them.
      - name: Publish it in an issue
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          NUM=$(gh issue list --label dora --state open --limit 1 --json number --jq '.[0].number // empty')
          if [ -n "$NUM" ]; then
            gh issue comment "$NUM" --body-file dora-report.md
          else
            gh issue create --title "DORA metrics (weekly report)" \
              --label dora --body-file dora-report.md
          fi

Run it by hand:

gh workflow run dora.yml -f days=30
gh run watch

What you should see in the run summary:

## 📊 DORA metrics · last 30 days

| Metric | Value | Level |
|---|---|---|
| Deployment frequency | 3.5 / week | High |
| Lead time (median) | 42 min | Elite |
| Change failure rate | 12.5 % | Medium |
| Time to restore (median) | 2 min | Elite |

A high CFR in this lab is normal and expected: you have caused failures on purpose. And there is the lesson: an isolated number means nothing; what means something is the trend. Reservalia went from 1.5 deployments/week, 68 h of lead time, 6.5 % CFR and 68 minutes to restore, to 12/week, 3.5 h, 3.8 % and 9 minutes. Not because somebody decided to "improve the DORA metrics", but because each concrete pipeline improvement —cache, parallelisation, immutable artifact, rollback by digest, automatic gates— moved one of the four. The metrics are the thermometer, not the medicine.

Automatic rollback driven by metrics

The closing of the loop. The smoke test validates thirty seconds; now we are going to watch for ten minutes and revert on our own if anything degrades.

scripts/watch-deployment.sh:

#!/usr/bin/env bash
# Watches the service metrics AFTER deploying.
# Exits 0 if all is well; exits 1 if a rollback is needed.
#
# Usage:
#   BASE=http://localhost:3002 WINDOW_MIN=10 ./scripts/watch-deployment.sh

set -Eeuo pipefail

BASE="${BASE:?BASE is missing}"
WINDOW_MIN="${WINDOW_MIN:-10}"
ERROR_THRESHOLD_PCT="${ERROR_THRESHOLD_PCT:-2.0}"
P95_THRESHOLD_SEC="${P95_THRESHOLD_SEC:-0.5}"
INTERVAL_SEC="${INTERVAL_SEC:-30}"
# How many consecutive bad checks are needed before reverting.
# With just 1, a transient spike would cause an unnecessary rollback:
# a rollback is also a change, and unnecessary changes have a cost.
MAX_CONSECUTIVE_BAD="${MAX_CONSECUTIVE_BAD:-3}"

log() { printf '[watch %s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }

# Reads a metric from the /metrics endpoint by exact series name.
read_metric() {
  local pattern="$1"
  curl -fsS --max-time 5 "$BASE/metrics" 2>/dev/null \
    | grep -E "^${pattern}" | awk '{s+=$NF} END {print (NR?s:0)}'
}

log "Watching $BASE for $WINDOW_MIN min"
log "Thresholds: 5xx errors < ${ERROR_THRESHOLD_PCT}% · p95 < ${P95_THRESHOLD_SEC}s"

# Baseline: the counters are cumulative since startup, so we measure
# INCREMENTS relative to the start of the watch.
BASE_TOTAL=$(read_metric 'http_requests_total\{')
BASE_5XX=$(read_metric 'http_requests_total\{code="5')
BASE_SUM=$(read_metric 'http_duration_seconds_sum')
BASE_CNT=$(read_metric 'http_duration_seconds_count')
log "Baseline: total=$BASE_TOTAL 5xx=$BASE_5XX"

END=$(( $(date +%s) + WINDOW_MIN * 60 ))
BAD=0
CYCLE=0

while [ "$(date +%s)" -lt "$END" ]; do
  sleep "$INTERVAL_SEC"
  CYCLE=$((CYCLE + 1))

  if ! curl -fsS --max-time 5 "$BASE/health" >/dev/null 2>&1; then
    log "CRITICAL: /health is not responding. Revert immediately."
    exit 1
  fi

  TOTAL=$(read_metric 'http_requests_total\{')
  FIVEXX=$(read_metric 'http_requests_total\{code="5')
  SUM=$(read_metric 'http_duration_seconds_sum')
  CNT=$(read_metric 'http_duration_seconds_count')

  D_TOTAL=$(awk "BEGIN{print $TOTAL - $BASE_TOTAL}")
  D_5XX=$(awk "BEGIN{print $FIVEXX - $BASE_5XX}")
  D_SUM=$(awk "BEGIN{print $SUM - $BASE_SUM}")
  D_CNT=$(awk "BEGIN{print $CNT - $BASE_CNT}")

  # With no traffic nothing can be concluded. Do not revert for lack of data:
  # "I do not know" is not the same as "it is going badly".
  if awk "BEGIN{exit !($D_TOTAL < 5)}"; then
    log "Cycle $CYCLE: only $D_TOTAL requests; sample too small, skipped."
    continue
  fi

  ERROR_PCT=$(awk "BEGIN{printf \"%.2f\", ($D_5XX * 100) / $D_TOTAL}")
  MEAN_LATENCY=$(awk "BEGIN{printf \"%.3f\", ($D_CNT > 0) ? $D_SUM / $D_CNT : 0}")

  log "Cycle $CYCLE: requests=$D_TOTAL errors5xx=${ERROR_PCT}% mean_latency=${MEAN_LATENCY}s"

  IS_BAD=0
  awk "BEGIN{exit !($ERROR_PCT > $ERROR_THRESHOLD_PCT)}" && { log "  ⚠ errors above the threshold"; IS_BAD=1; }
  awk "BEGIN{exit !($MEAN_LATENCY > $P95_THRESHOLD_SEC)}"  && { log "  ⚠ latency above the threshold"; IS_BAD=1; }

  if [ "$IS_BAD" -eq 1 ]; then
    BAD=$((BAD + 1))
    log "  Consecutive bad checks: $BAD/$MAX_CONSECUTIVE_BAD"
    if [ "$BAD" -ge "$MAX_CONSECUTIVE_BAD" ]; then
      log "DECISION: revert. $BAD consecutive checks outside the thresholds."
      exit 1
    fi
  else
    [ "$BAD" -gt 0 ] && log "  Recovered; bad counter reset to zero."
    BAD=0
  fi
done

log "Watch completed with no incidents. Deployment stable."
exit 0

And the job in cd.yml, after the production deployment:

  watch:
    name: Post-deployment watch
    runs-on: ubuntu-latest
    needs: [prepare, production]
    timeout-minutes: 20
    permissions:
      contents: read
      actions: write        # required to launch the rollback workflow
    steps:
      - uses: actions/checkout@v4

      - name: Generate synthetic background traffic
        run: |
          # With no traffic there are no metrics. In a real system this is
          # unnecessary: the traffic is generated by users.
          (for i in $(seq 1 600); do
             curl -s "http://localhost:3002/api/slots?date=2026-03-02" > /dev/null 2>&1 || true
             sleep 1
           done) &
          echo "generator=$!" >> "$GITHUB_ENV"

      - name: Watch for 10 minutes
        id: watch
        continue-on-error: true    # we want to DECIDE based on the result, not abort
        env:
          BASE: http://localhost:3002
          WINDOW_MIN: '10'
          ERROR_THRESHOLD_PCT: '2.0'
          P95_THRESHOLD_SEC: '0.5'
        run: ./scripts/watch-deployment.sh

      - name: Automatic rollback if the watch failed
        if: steps.watch.outcome == 'failure'
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          set -Eeuo pipefail
          PREVIOUS="${{ needs.production.outputs.previous_digest }}"
          if [ -z "$PREVIOUS" ]; then
            echo "::error::The watch failed but there is no known previous digest. MANUAL INTERVENTION."
            exit 1
          fi

          echo "::warning::Metrics degraded after the deployment. Reverting to $PREVIOUS"
          gh workflow run rollback.yml \
            -f environment=production \
            -f digest="$PREVIOUS" \
            -f reason="AUTOMATIC rollback: metrics outside thresholds after deploying ${{ needs.prepare.outputs.digest }}"

          {
            echo "## 🔴 Automatic rollback triggered"
            echo ""
            echo "| Field | Value |"
            echo "|---|---|"
            echo "| Problem digest | \`${{ needs.prepare.outputs.digest }}\` |"
            echo "| Reverted to | \`$PREVIOUS\` |"
            echo "| Reason | Error or latency thresholds exceeded for 3 cycles |"
            echo ""
            echo "**Action required:** write the post-mortem before retrying."
          } >> "$GITHUB_STEP_SUMMARY"
          exit 1

      - name: Confirm the deployment is stable
        if: steps.watch.outcome == 'success'
        run: echo "### ✅ Deployment stable after 10 minutes of watching" >> "$GITHUB_STEP_SUMMARY"

For needs.production.outputs.previous_digest to exist, add the output to the production job:

    outputs:
      previous_digest: ${{ steps.deploy.outputs.previous_digest }}

(The deploy.sh script from 07-03 already writes it to $GITHUB_OUTPUT.)

Try it for real: deploy the version with ARTIFICIAL_DELAY_MS=800, which is above the 0.5 s threshold.

What you should see:

[watch 10:15:00] Watching http://localhost:3002 for 10 min
[watch 10:15:30] Cycle 1: requests=29 errors5xx=0.00% mean_latency=0.812s
[watch 10:15:30]   ⚠ latency above the threshold
[watch 10:15:30]   Consecutive bad checks: 1/3
[watch 10:16:00] Cycle 2: ... mean_latency=0.809s
[watch 10:16:00]   Consecutive bad checks: 2/3
[watch 10:16:30] Cycle 3: ... mean_latency=0.815s
[watch 10:16:30] DECISION: revert. 3 consecutive checks outside the thresholds.

And straight after that, rollback.yml starting on its own. Total time from the bad deployment to the service being restored: about 3 minutes, without a single person doing anything. That is what moves "time to restore" from 68 minutes to 9 on Reservalia's baseline.

The three design decisions that make this safe rather than a chaos generator:

Decision	Why
3 consecutive bad cycles, not one	A transient spike must not cause a rollback. A rollback is also a change
Minimum sample (5 requests)	With no traffic, "0 errors out of 0 requests" is not information. Do not revert for lack of data
It aborts if there is no previous digest	A rollback to nowhere is worse than the problem. Better to escalate to a person

Final verification

#	Check	How	Expected
1	`/metrics` responds in Prometheus format	`curl localhost:3002/metrics`	`# HELP`, `# TYPE` lines, series
2	The buckets are cumulative	`npm test`	The histogram test green
3	There is no cardinality explosion	`curl -s .../metrics \| grep -c '^http_requests_total{'`	Fewer than 20 series
4	Prometheus scrapes both environments	`localhost:9090/targets`	Two UP targets
5	The dashboard provisions itself	Grafana with nothing imported	The dashboard is there
6	The dashboard cannot be edited in the UI	Try to save	Blocked
7	The SLO is documented with its arithmetic	`observability/SLO.md`	The four elements + budget
8	The rules are valid	`promtool check rules`	`SUCCESS: 4 rules found`
9	The alert goes through PENDING and reaches FIRING	Artificial delay + 6 min	All three states observed
10	The alert recovers on its own	Remove the delay	Back to `Inactive`
11	Deployments are annotated	Dashboard after a deployment	Vertical line with metadata
12	The DORA metrics are computed	`gh workflow run dora.yml`	Table with all four
13	The automatic rollback fires	Deploy with an 800 ms delay	Rollback launched on its own in ~3 min
14	It does not revert for lack of data	Watch with no traffic	"sample too small, skipped"

Common Mistakes and Tips

Symptom: histogram_quantile returns NaN or draws nothing. Cause: you have lost the le label in the aggregation. sum(rate(...bucket[5m])) without by (le) destroys the histogram's information. Fix: always sum by (le, <the others>) (rate(..._bucket[5m])). It is the number one PromQL mistake.

Symptom: Prometheus shows the target as DOWN with connection refused. Cause: from inside the Prometheus container, localhost is Prometheus itself, not your machine. Fix: host.docker.internal with the extra_hosts: host-gateway already in the compose file. Verify: docker exec prometheus wget -qO- http://host.docker.internal:3002/metrics | head -3.

Symptom: Prometheus memory grows out of control and queries take seconds. Cause: cardinality explosion. Some label has unbounded values: a full URL, a user id, a timestamp. Fix: the normaliseRoute function from section 2. Rule: the number of distinct values of a label must be bounded and small. Diagnosis: topk(10, count by (__name__)({__name__=~".+"})).

Symptom: the alert fires and clears itself every few minutes ("flapping"). Cause: the for is too short for the metric's volatility, or the threshold sits right on the usual value. Fix: raise the for or move the threshold further away. Rule of thumb: the threshold should be at least 50 % above the 99th percentile of normal operation. An alert that fires daily stops being read within a week.

Symptom: the counters suddenly reset to zero and rate() produces a strange spike. Cause: the process restarted (a deployment). The metrics live in memory. Fix: none needed. Prometheus's rate() detects counter resets and compensates for them. That is why rate() over a counter is correct and subtracting values by hand is not.

Symptom: the DORA schedule job does not run on time, or stops running. Causes: (1) cron is UTC, always; (2) GitHub delays schedule runs when under load, by quite a few minutes; (3) GitHub disables scheduled workflows in repositories with no activity for 60 days, and warns you by email once. Fix: do not depend on punctuality; keep the workflow_dispatch so you can launch it by hand.

Symptom: the automatic rollback fires on a perfectly good deployment. Cause: the threshold was evaluated during startup, with cold caches and the first requests running slow. Fix: add a grace period before you start counting (an initial sleep 60), or discard the first cycle. It is the equivalent of Docker's HEALTHCHECK start-period.

Tip — instrument the business flow, not just HTTP. The golden signals tell you whether the system works. An appointments_created_total counter tells you whether the product works. A deployment can leave every HTTP response at 200 and appointments created at zero, and that is the outage that really costs money. It is exercise 2.

Tip — the metric almost nobody adds and that is always needed. mini_reservalia_info{version="..."} measures nothing, but it answers "which version is deployed?" instantly, from the very place where you see the problem. It costs three lines.

Exercises

Exercise 1: an availability SLO, on top of the latency one

Define a second SLO —99.9 % of requests to /api/slots with no 5xx error over 30 days—, calculate its error budget with explicit arithmetic, write the SLI query and add a burn rate alert with two windows (a fast one and a slow one) to avoid false positives.

Exercise 2: business metrics

Add metrics that measure the product rather than the infrastructure: appointments created, appointments rejected by validation, and the distribution of the requested duration. Add an alert for "we have gone 30 minutes without creating a single appointment during business hours".

Exercise 3: DORA history with a trend

Make the DORA report store its history and show the variation against the previous week, with trend arrows. An isolated number is useless; a trend is not.

Solutions

Solution 1.

## SLO 2: availability of /api/slots

| Field | Value |
|---|---|
| SLI | Proportion of requests to `/api/slots` with no 5xx code |
| Target | 99.9 % |
| Window | 30 days |

### Budget arithmetic

    Traffic:  20 req/min
    Window:   30 days

    Requests = 20 × 60 × 24 × 30 = 864,000

    Budget = (100 % − 99.9 %) = 0.1 %
           = 0.001 × 864,000 = 864 failed requests

    In time (total outage):
                864 ÷ 20 req/min = 43.2 minutes every 30 days
                ≈ 1.44 minutes per day

Note: the latency SLO (99.5 %) allows 1,008 slow requests every 7 days;
the availability one allows 864 failed ones every 30 days. They are
INDEPENDENT budgets and the most restrictive one wins: exhausting either
triggers the freeze.

# Availability SLI
1 - (
  sum(rate(http_requests_total{route="/api/slots", code=~"5.."}[30d]))
  /
  sum(rate(http_requests_total{route="/api/slots"}[30d]))
)

Burn rate alert with two windows:

      # The LONG window (1 h) detects the sustained problem.
      # The SHORT window (5 m) confirms it is STILL happening NOW.
      # Requiring both eliminates alerts for an already resolved incident,
      # which is the number one cause of distrust in alerting.
      - alert: AvailabilityBudgetBurningFast
        expr: |
          (
            sum by (environment) (rate(http_requests_total{route="/api/slots",code=~"5.."}[1h]))
            / sum by (environment) (rate(http_requests_total{route="/api/slots"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum by (environment) (rate(http_requests_total{route="/api/slots",code=~"5.."}[5m]))
            / sum by (environment) (rate(http_requests_total{route="/api/slots"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels: { severity: critical, type: burn-rate-fast }
        annotations:
          summary: 'Availability budget burning 14.4x faster than is sustainable'
          description: 'At this rate, the 30-day budget is exhausted in ~2 days.'

      - alert: AvailabilityBudgetBurningSlow
        expr: |
          (
            sum by (environment) (rate(http_requests_total{route="/api/slots",code=~"5.."}[6h]))
            / sum by (environment) (rate(http_requests_total{route="/api/slots"}[6h]))
          ) > (6 * 0.001)
          and
          (
            sum by (environment) (rate(http_requests_total{route="/api/slots",code=~"5.."}[30m]))
            / sum by (environment) (rate(http_requests_total{route="/api/slots"}[30m]))
          ) > (6 * 0.001)
        for: 15m
        labels: { severity: warning, type: burn-rate-slow }

The SRE Workbook multipliers: 14.4× consumes the whole budget in 2 days (a wake-you-up alert); 6× consumes it in 5 days (a raise-a-ticket alert). The two-window pattern is what makes the alert clear itself when the incident is resolved, instead of carrying on ringing because of contamination from the long window.

Solution 2.

// In src/server.js, inside the POST /api/appointments handler:
      if (req.method === 'POST' && url.pathname === '/api/appointments') {
        const body = await readBody(req);
        const requestedDuration = body.start && body.end
          ? toMinutes(body.end) - toMinutes(body.start)
          : 0;
        try {
          const appointment = repository.createAppointment(body);
          metrics.increment('appointments_created_total', { environment: ENVIRONMENT });
          // Buckets in MINUTES: 15, 30, 45, 60, 90, 120.
          metrics.observe('appointment_duration_minutes', {}, requestedDuration);
          return respondJson(res, 201, appointment);
        } catch (error) {
          metrics.increment('appointments_rejected_total', {
            // The label is the error TYPE, not the message: messages are
            // free text and would make cardinality explode.
            reason: error instanceof RangeError ? 'invalid_range' : 'invalid_format',
          });
          throw error;
        }
      }

      - alert: NoAppointmentsCreated
        # `hour()` returns the UTC hour. The `and` with the hour ranges and the
        # day of the week stops the alert going off on a Sunday night, when
        # zero appointments is perfectly normal.
        expr: |
          (
            sum(increase(appointments_created_total{environment="production"}[30m])) == 0
            or
            absent(appointments_created_total{environment="production"})
          )
          and on() (hour() >= 8 and hour() < 18)
          and on() (day_of_week() > 0 and day_of_week() < 6)
        for: 10m
        labels: { severity: critical, type: business }
        annotations:
          summary: 'No appointment created in 30 minutes, during business hours'
          description: >-
            HTTP can be all 200s and the product broken all the same.
            Check the full booking flow before the infrastructure.

That last one is, by some distance, the most valuable alert in the file. A regression that breaks the booking form leaves every endpoint returning 200 —nobody even gets as far as calling them— and not one golden signal so much as twitches. The only metric that notices is the business one.

Solution 3.

// At the end of scripts/dora.js
import { readFile, writeFile, mkdir } from 'node:fs/promises';

const HISTORY_FILE = 'observability/dora-history.json';

const current = {
  date: new Date().toISOString().slice(0, 10),
  windowDays: DAYS,
  weeklyFrequency: Number(perWeek.toFixed(2)),
  leadTimeHours: Number(medianLead.toFixed(2)),
  cfrPct: Number(cfr.toFixed(2)),
  restoreMin: Number(medianRestore.toFixed(1)),
};

let history = [];
try {
  history = JSON.parse(await readFile(HISTORY_FILE, 'utf8'));
} catch { /* first run */ }

const previous = history.at(-1);
history.push(current);
await mkdir('observability', { recursive: true });
await writeFile(HISTORY_FILE, `${JSON.stringify(history.slice(-52), null, 2)}\n`);

/**
 * Trend arrow. `higherIsBetter` distinguishes the speed metrics
 * (more is better) from the stability ones (less is better): without that
 * parameter, a rising CFR would come out with a green arrow.
 */
function trend(currentVal, previousVal, higherIsBetter) {
  if (previousVal === undefined) return '—';
  const delta = currentVal - previousVal;
  if (Math.abs(delta) < 0.001) return '→ unchanged';
  const better = higherIsBetter ? delta > 0 : delta < 0;
  const sign = delta > 0 ? '+' : '';
  return `${better ? '🟢 ▲' : '🔴 ▼'} ${sign}${delta.toFixed(1)}`;
}

const table = `
### Trend against the previous measurement${previous ? ` (${previous.date})` : ''}

| Metric | Previous | Current | Trend |
|---|---|---|---|
| Frequency (/week) | ${previous?.weeklyFrequency ?? '—'} | ${current.weeklyFrequency} | ${trend(current.weeklyFrequency, previous?.weeklyFrequency, true)} |
| Lead time (h) | ${previous?.leadTimeHours ?? '—'} | ${current.leadTimeHours} | ${trend(current.leadTimeHours, previous?.leadTimeHours, false)} |
| CFR (%) | ${previous?.cfrPct ?? '—'} | ${current.cfrPct} | ${trend(current.cfrPct, previous?.cfrPct, false)} |
| Restore (min) | ${previous?.restoreMin ?? '—'} | ${current.restoreMin} | ${trend(current.restoreMin, previous?.restoreMin, false)} |

<details><summary>Historical series (${history.length} measurements)</summary>

\`\`\`
${history.slice(-12).map((h) =>
  `${h.date}  freq=${String(h.weeklyFrequency).padStart(5)}/wk  lead=${String(h.leadTimeHours).padStart(6)}h  cfr=${String(h.cfrPct).padStart(5)}%  mttr=${String(h.restoreMin).padStart(5)}min`,
).join('\n')}
\`\`\`

</details>
`;

console.log(table);
if (process.env.GITHUB_STEP_SUMMARY) {
  await appendFile(process.env.GITHUB_STEP_SUMMARY, table);
}

And in the workflow, so the history persists:

      - name: Commit the history
        run: |
          git config user.name  'github-actions[bot]'
          git config user.email 'github-actions[bot]@users.noreply.github.com'
          git add observability/dora-history.json
          # `|| exit 0`: if there are no changes, `git commit` exits 1 and that is not a failure.
          git diff --staged --quiet || git commit -m "chore: DORA metrics $(date -u +%Y-%m-%d)"
          git push

It needs permissions: contents: write on the job and, if main is protected, a ruleset bypass for github-actions[bot] or a dedicated branch.

Storing the history in the repository itself has a virtue that makes up for its crudeness: the data travels with the code, it is reviewed in PRs and it does not depend on any external service somebody has to pay for. For a small team it is more than enough.

Optional challenge

Replace the /metrics-based watching with a query against Prometheus (/api/v1/query with PromQL), which is what a real system would do: it lets you use genuine percentiles instead of the mean, compare against the same time last week, and correlate several metrics in a single expression. The query would be something like histogram_quantile(0.95, sum by (le) (rate(http_duration_seconds_bucket{environment="production"}[5m]))) and the rollback criterion would be comparing that value against the one from before the deployment instead of against an absolute threshold. It is the difference between "it is slow" and "it is slower than before your change", which is the right question.

What you have built

Your own metrics registry with counters, gauges and cumulative histograms, in Prometheus format, with no dependencies, protected against cardinality explosion.
An instrumentation middleware that measures latency and classifies client errors against server errors.
An observability stack with docker compose, provisioned as code: Prometheus with two targets labelled by environment and Grafana with a versioned datasource and dashboard.
A dashboard as code with the three percentiles, the SLO threshold drawn in and an environment variable.
A documented SLO with all four elements, its error budget worked out with explicit arithmetic and a banded decision policy.
Four symptom-based alert rules with for, a runbook and burn rate, validated with promtool, and the experience of watching one go from PENDING to FIRING.
Deployment annotations on the dashboard, which make the correlation between change and degradation visible.
A scheduled job that computes the four DORA metrics of your own repository and publishes them.
An automatic metrics-driven rollback with a minimum sample, consecutive cycles and escalation to a person when it cannot decide.

Conclusion

The loop is closed in both directions. Inwards: the system tells you how it is doing, with a numeric target, a budget that gets consumed and alerts that ring for the things users notice. Outwards: the pipeline measures itself, and you can now answer "is this working?" with numbers. And the two directions meet in the automatic rollback, where a degraded metric pulls a pipeline lever with nobody watching.

This is also the point at which it is worth stopping and noticing what you have built, because it has a problem. Your pipeline can now deploy code to production and revert it, by itself, with no human involvement. It has registry credentials. It has a token capable of launching workflows. It runs third-party actions pointing at moving tags their authors can move. It builds images running processes with dependencies nobody has audited. And that GRAFANA_TOKEN you set up a while ago is a long-lived credential stored in a variable several jobs can read.

Put another way: you have built a very capable and very permissive system, and you have not yet looked at it through an attacker's eyes.

In 07-05 that is exactly where we start: an audit of the pipeline you built yourself, with a list of everything that is wrong, and its step-by-step correction. You will apply least-privilege permissions and see how the error reads when one is missing; pin actions by SHA and automate their updating; add Gitleaks and commit a test secret on purpose so you can run the full response procedure —rotate first, clean the history afterwards, and understand why in that order—; implement a severity policy with jq over npm audit --json; enable CodeQL and introduce an obvious injection to watch it be detected; scan the image with Trivy with exceptions that expire; generate an SBOM, sign the image with Cosign and verify the signature before deploying; check secret masking and why it is not a guarantee; and see the concrete attack that makes pull_request_target the most dangerous trap in GitHub Actions.

Exercise 4: Monitoring and Feedback

Contents

Objective, prerequisites and starting point

Instrumenting the server: counters and histograms

The `/metrics` endpoint in Prometheus format

The four golden signals on top of these metrics

Prometheus and Grafana with `docker compose`

The dashboard as code

The SLO and its error budget, with the arithmetic

Symptom-based alert rules

Firing the alert on purpose

Marking deployments on the dashboard

Closing the loop: the four DORA metrics of your repository

Automatic rollback driven by metrics

Final verification

Common Mistakes and Tips

Exercises

Exercise 1: an availability SLO, on top of the latency one

Exercise 2: business metrics

Exercise 3: DORA history with a trend

Solutions

Optional challenge

What you have built

Conclusion

CI/CD Course: Continuous Integration and Deployment

Module 1: Introduction to CI/CD

Module 2: Continuous Integration (CI)

Module 3: Continuous Deployment (CD)

Module 4: Advanced CI/CD Practices

Module 5: Implementing CI/CD in Real Projects

Module 6: Tools and Technologies

Module 7: Practical Exercises

Module 8: Additional Resources

Exercise 4: Monitoring and Feedback

Contents

Objective, prerequisites and starting point

Instrumenting the server: counters and histograms

The /metrics endpoint in Prometheus format

The four golden signals on top of these metrics

Prometheus and Grafana with docker compose

The dashboard as code

The SLO and its error budget, with the arithmetic

Symptom-based alert rules

Firing the alert on purpose

Marking deployments on the dashboard

Closing the loop: the four DORA metrics of your repository

Automatic rollback driven by metrics

Final verification

Common Mistakes and Tips

Exercises

Exercise 1: an availability SLO, on top of the latency one

Exercise 2: business metrics

Exercise 3: DORA history with a trend

Solutions

Optional challenge

What you have built

Conclusion

CI/CD Course: Continuous Integration and Deployment

Module 1: Introduction to CI/CD

Module 2: Continuous Integration (CI)

Module 3: Continuous Deployment (CD)

Module 4: Advanced CI/CD Practices

Module 5: Implementing CI/CD in Real Projects

Module 6: Tools and Technologies

Module 7: Practical Exercises

Module 8: Additional Resources

The `/metrics` endpoint in Prometheus format

Prometheus and Grafana with `docker compose`