Across seven modules the course has been leaving doors ajar. In 03-04 you configured a canary by shifting ALB weights by hand and it was said that tools exist which do that on their own by analysing metrics. In 04-03 you stored secrets in the repository's vault and it was said that there comes a point where that stops being enough. In 02-04 you measured coverage and it was admitted that coverage lies. In 07-05 you signed images with Cosign and a framework called SLSA was mentioned without being developed.

This lesson opens those doors. But with a strict discipline, the same one 08-02 taught you to apply to other people's advice: every tool is presented by the problem it solves, not by what it is. For each one you will find the moment in the course where the problem appeared, how it fits the pipeline you already have, a short configuration example and — the part that almost never gets written — when you do NOT need it.

It is not a tutorial on anything. It is a recognition catalogue: so that when a problem appears, you know something exists that solves it and what it is going to cost you.

Contents

  1. Secret management beyond CI
  2. Automated progressive delivery
  3. GitOps
  4. Policy as code
  5. Quality and analysis
  6. Supply chain
  7. Testing
  8. Local development and reproducibility
  9. Internal platform
  10. Observability
  11. Adoption table by team size
  12. The criteria for letting a tool in
  13. Common Mistakes and Tips
  14. Exercises
  15. Conclusion

  1. Secret management beyond CI

The problem, and where it appeared. In 04-03 and 07-05 you stored secrets in the repository's or the organisation's secret store. It works well and it is the right thing to start with. It stops being enough when any of these four conditions appears:

  1. The running application needs the secret too, not just the pipeline. The CI store injects variables during the workflow; that is no use at all to a container that has been running in ECS for three weeks.
  2. You have to rotate and you cannot. Rotating the database password means changing it in the CI store, in the task definition and in the database itself, in a coordinated way and by hand.
  3. You need auditing. "Who read this secret and when?" The CI store tells you who edited it, not who used it.
  4. Secrets multiply across environments and teams and no longer fit in a flat list.

1.1. AWS Secrets Manager (and Parameter Store)

It is the lowest-friction option if you are already on AWS, as Reservalia is. The secret is stored in the service, the ECS task receives it by reference and it never passes through the pipeline: the workflow no longer sees the database password at any point, which cuts the exposure surface at a stroke.

{
  "containerDefinitions": [{
    "name": "api",
    "image": "<account>.dkr.ecr.eu-west-1.amazonaws.com/reservalia-api@sha256:abc123...",
    "secrets": [
      {
        "name": "DATABASE_URL",
        "valueFrom": "arn:aws:secretsmanager:eu-west-1:<account>:secret:reservalia/prod/db-Ab3xY9"
      }
    ]
  }]
}

The ECS agent resolves the ARN when starting the task using the execution role. Automatic rotation is configured in the service itself with a Lambda function.

Trade-offs. It costs money per secret and per API call — a little, but Diego will ask; Parameter Store with SecureString parameters is the cheaper and less capable alternative. And it ties you to the provider: ARNs do not migrate.

When you do NOT need it. If all your secrets are pipeline secrets (registry tokens, deployment credentials) and your application consumes none at runtime, the CI store plus OIDC is sufficient and simpler.

1.2. HashiCorp Vault

The general-purpose, provider-agnostic secret manager. It brings two things a basic cloud service does not:

  • Dynamic secrets. Instead of storing a database credential, Vault generates it on demand with an expiry. The application asks for credentials, receives a username and password valid for one hour, and Vault revokes them by itself. A secret that expires in an hour is a secret that can hardly be leaked.
  • Fine-grained policies and auditing, with its own identity engine and a record of every read.
# In a workflow: authenticate to Vault with the GitHub Actions OIDC token
- uses: hashicorp/vault-action@<full-sha>
  with:
    url: https://vault.internal.example.com
    method: jwt
    role: github-actions-reservalia
    secrets: |
      secret/data/reservalia/prod token | DEPLOY_TOKEN ;

Just as with AWS in 03-02, there is no long-lived credential here: the runner presents its OIDC token and Vault decides whether that repository and that branch may read that path.

Trade-offs, and they are big. Self-hosted Vault is one more critical system to operate: high availability, sealing and unsealing, backups, upgrades. If Vault goes down, nobody deploys and possibly nothing starts. It is a serious piece of infrastructure, and its managed version costs money. Lesson 06-01 said of Jenkins that the real cost is not the licence but who maintains it; with Vault it is exactly the same.

When you do NOT need it. Almost always, with fewer than 20 people and a single cloud. The right answer for Reservalia is Secrets Manager, not Vault. Vault starts to pay off with several clouds, strict audit requirements or when dynamic secrets solve a specific compliance problem.

1.3. SOPS + age: secrets versioned in Git

A different and very underrated approach: encrypt the secrets and commit them. SOPS encrypts only the values of a YAML or JSON file, leaving the keys readable, so that git diff remains useful.

# config/prod.enc.yaml — committed with no problem
database:
    host: db.reservalia.internal
    password: ENC[AES256_GCM,data:pQx8fK2m...,tag:9dK...,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# Decrypt in the pipeline
sops --decrypt config/prod.enc.yaml > config/prod.yaml

age is the modern encryption tool that replaces GPG for this use: short keys, no keyring, none of GPG's historical complexity. In the pipeline, the decryption key is the only secret living in the CI store — you have reduced N secrets to one.

Real advantages: secrets are versioned, reviewed in PRs, follow the same promotion path as the code, and the history says who changed what and when. Trade-offs: rotating a recipient requires re-encrypting every file; and even encrypted, the material is in the repository forever, so a broken algorithm or a leaked key five years from now compromises the entire history.

When it is the right choice: small teams with many configuration files per environment, especially with Kubernetes, where it fits naturally.

1.4. External Secrets Operator

Kubernetes-specific. It synchronises secrets from an external manager (Secrets Manager, Vault, etc.) into native cluster Secret objects, so that applications consume them in the standard way without knowing where they come from.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: reservalia-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: reservalia-db-secret
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: reservalia/prod/db
        property: url

When you do NOT need it. If you do not use Kubernetes, it is irrelevant. Reservalia, on ECS, does not need it.

1.5. The decision criteria

graph TD
    A["Is the secret used only by the pipeline?"] -->|Yes| B["CI store + OIDC<br/>Sufficient"]
    A -->|"No: the app needs it<br/>at runtime"| C["A single cloud?"]
    C -->|Yes| D["The cloud manager<br/>Secrets Manager / Parameter Store"]
    C -->|"No, or fine-grained<br/>auditing needed"| E["Vault"]
    B --> F["Many config files<br/>per environment?"]
    F -->|Yes| G["SOPS + age<br/>as a complement"]

And one rule worth more than the diagram: the best secret management is not having the secret. OIDC removed AWS keys from the pipeline in 03-02; IAM roles remove credentials between cloud services. Every secret you manage to eliminate is one you do not have to manage, rotate or audit.

  1. Automated progressive delivery

The problem, and where it appeared. In 03-04 you set up a canary diverting 10% of traffic with ALB weights, and then somebody — you — looked at Grafana and decided whether to promote or roll back. That has three flaws: it depends on a human watching, the criterion is subjective, and in practice you end up promoting out of impatience.

Automated progressive delivery turns that judgement into a declarative specification: which metrics, which thresholds, how often, and what to do if they fail.

2.1. Argo Rollouts

It replaces the Kubernetes Deployment with a Rollout resource that knows how to do canary and blue-green with automatic analysis.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: reservalia-api
spec:
  replicas: 6
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate-5xx
      interval: 1m
      count: 5
      successCondition: result[0] < 0.01   # fewer than 1% of 5xx
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{job="reservalia-api",status=~"5.."}[2m]))
            /
            sum(rate(http_requests_total{job="reservalia-api"}[2m]))

If the query exceeds the threshold once, the rollout reverts by itself, with nobody watching anything. It is exactly the automatic metric-based rollback from 03-05, but declared in the resource instead of written in a workflow script.

2.2. Flagger

It solves the same thing with a different philosophy: instead of replacing the Deployment, it wraps it. You declare a Canary resource that references the existing Deployment, and Flagger orchestrates the progressive rollout by manipulating the service mesh or the ingress.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: reservalia-api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: reservalia-api
  analysis:
    interval: 1m
    threshold: 5          # 5 consecutive failures -> rollback
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange: { min: 99 }
        interval: 1m
      - name: request-duration
        thresholdRange: { max: 500 }   # p99 in ms
        interval: 1m
    webhooks:
      - name: smoke-test
        type: pre-rollout
        url: http://flagger-loadtester.test/
        metadata:
          cmd: "curl -sf http://reservalia-api-canary/health"

Note the pre-rollout webhook: it is the smoke test from 03-02, run against the canary version before sending it real traffic.

The practical difference between the two. Argo Rollouts gives explicit step-by-step control and fits well with Argo CD; Flagger is more automatic and less intrusive with existing manifests. Both require reliable metrics and capable traffic control (a service mesh, a compatible ingress or a supported traffic provider).

When you do NOT need them — and this matters. Both are Kubernetes-only. Reservalia is on ECS: they do not apply. And there is a precondition harder than the platform: you need quality metrics and enough traffic volume. A canary at 10% on a service with 5 requests a minute generates no statistical signal: 30 requests in five minutes cannot tell a 1% error rate from a 4% one. In that regime, automatic analysis is theatre and a rolling deployment with health checks plus a fast rollback is honestly better.

The entry condition: enough traffic for a 1% degradation to be detectable within the analysis window, metrics already instrumented and trusted (03-06 done properly), and deployments frequent enough that automating the judgement is worth the configuration effort.

  1. GitOps

The problem, and where it appeared. In 03-02 your cd.yml does a push: the pipeline holds credentials over production and performs the deployment. That means CI is a piece with very high privileges, and that the real state of the environment can drift from the code without anybody noticing. Somebody changes something by hand in the console, and the repository stops describing reality. Lesson 03-03 called it drift.

GitOps reverses the direction: an agent inside the cluster watches a repository and applies whatever is there. The pipeline no longer deploys: it only writes to the deployment repository.

graph LR
    A[ci.yml<br/>builds and publishes<br/>image by digest] --> B[Deployment repository<br/>manifests + digest]
    B --> C{GitOps agent<br/>Argo CD / Flux}
    C -->|"reconciles every 3 min"| D[Cluster]
    D -.->|"detects drift"| C
    C -.->|"reverts manual<br/>changes"| D

The deployment repository pattern. The application code is separated from the deployment manifests. CI, on publishing the image, commits the new digest to the deployment repository; the agent detects it and reconciles. The git history is the deployment history, and reverting is a git revert.

# Argo CD: an Application watching the deployment repository
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: reservalia-prod
spec:
  project: default
  source:
    repoURL: https://github.com/reservalia/deployments.git
    targetRevision: main
    path: environments/production
  destination:
    server: https://kubernetes.default.svc
    namespace: reservalia
  syncPolicy:
    automated:
      prune: true         # deletes what is no longer in git
      selfHeal: true      # reverts manual changes in the cluster

selfHeal: true is drift detection in its most forceful form: if somebody edits something by hand, it gets undone on the next reconciliation.

Argo CD versus Flux. Argo CD has a web interface and is more accessible for teams starting out; Flux is lighter, more CLI-oriented and more composable with other pieces. Both are CNCF graduated projects — with everything that means according to 08-02 — and the choice between them is rarely the decisive factor.

Trade-offs. Another piece to operate; two repositories and therefore two histories you have to know how to correlate when something fails; and less direct debugging ("why has it not synced?" is a new question). Besides, not everything fits: the database migrations from 04-06 are not declarative and still need their own mechanism.

When you do NOT need it. Without Kubernetes, it does not apply in its canonical form. And with a single environment and a small team, the cd.yml from 03-02 with OIDC does the same job with far fewer pieces. GitOps pays off with several clusters, several environments, several teams and when drift is a real, measured problem rather than a hypothetical one.

  1. Policy as code

The problem, and where it appeared. In 04-03 and 03-03 you established rules: containers do not run as root, buckets are not public, actions are pinned by SHA. Those rules live today in code review, that is, in the attention of a tired human on a Friday afternoon. Policy as code turns them into an automated check that fails the PR.

4.1. OPA / Rego and Conftest

OPA is a generic policy engine; Rego is its language; Conftest is the tool that applies it to configuration files in the pipeline.

# policies/terraform.rego
package main

deny[msg] {
    resource := input.resource_changes[_]
    resource.type == "aws_s3_bucket_public_access_block"
    resource.change.after.block_public_acls == false
    msg := sprintf("Bucket '%s' allows public ACLs", [resource.address])
}

deny[msg] {
    resource := input.resource_changes[_]
    resource.type == "aws_db_instance"
    not resource.change.after.storage_encrypted
    msg := sprintf("Database '%s' does not have storage encryption", [resource.address])
}
- name: Validate infrastructure policies
  run: |
    terraform plan -out=plan.tfplan
    terraform show -json plan.tfplan > plan.json
    conftest test plan.json --policy policies/

What makes it powerful is that it evaluates the plan, not the applied state: the policy fails before anything is touched, in the PR, with a message that says which resource and why. It is the quality gate from 02-05 applied to infrastructure.

And it works over any data structure, including your own workflows:

package main

deny[msg] {
    job := input.jobs[name]
    step := job.steps[_]
    contains(step.uses, "@v")     # uses a tag instead of a SHA
    not startswith(step.uses, "actions/")
    msg := sprintf("Job '%s' uses a third-party action not pinned by SHA: %s", [name, step.uses])
}

This automates the rule from 07-05 that until now depended on somebody remembering it during review.

4.2. Kyverno

A Kubernetes-specific alternative written in YAML rather than Rego, which lowers the barrier to entry considerably. It acts as an admission controller: it rejects resources that violate the policy at the moment they are applied to the cluster.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-images-by-digest
spec:
  validationFailureAction: Enforce
  rules:
    - name: digest-only
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "Images must be referenced by digest (sha256:), not by tag."
        pattern:
          spec:
            containers:
              - image: "*@sha256:*"

It is the immutable artifact rule from 02-06, enforced by the cluster rather than by the goodwill of whoever writes the manifest.

Trade-offs. Rego has a real and barely transferable learning curve. And there is a bigger risk: a badly written policy blocks everybody. They are always deployed first in warning mode (Audit), you observe what they break over a few weeks, and only then do you switch them to blocking.

When you do NOT need it. With three people reviewing each other's PRs, writing Rego for rules that apply twice a month is overhead. It pays off when the rule is important, repeats often, and there are more people than fit in a conversation — from three or four teams upwards. Exception: if you have a single critical rule that must never fail, a grep in a workflow step enforces it just as well and in two minutes.

  1. Quality and analysis

The problem, and where it appeared. Lesson 02-05 set up ESLint, Prettier, tsc and a quality gate. Two layers are missing: deeper semantic analysis, and linters for the pipeline's own tooling, which is the gap almost everybody has left open.

5.1. Pipeline linters (start here)

This is the best cost-to-benefit section in the whole lesson. Five tools, all free, all one-liners:

Tool Checks Which real error it catches
actionlint GitHub Actions workflows Invalid expressions, needs pointing at a non-existent job, badly written shell inside run, contexts not available in that event
yamllint YAML in general Indentation, duplicate keys, the classic on: interpreted as a boolean
hadolint Dockerfile apt-get install without --no-install-recommends, badly ordered layers, a forgotten USER root, use of latest
tflint Terraform Non-existent instance types, deprecated attributes, conventions not followed
shellcheck Shell scripts Unquoted variables, badly written comparisons, errors that only show up with a filename containing spaces
- name: Lint the pipeline
  run: |
    actionlint
    hadolint Dockerfile
    shellcheck scripts/*.sh
    tflint --recursive

Why it matters more than it looks. Lesson 04-05 insisted that the pipeline is code and must be treated as such. These linters are the pipeline's unit tests: they catch in seconds errors that, without them, are discovered after a full cycle of commit, wait and fail. actionlint in particular saves most of the trial-and-error iterations with Actions YAML.

When you do NOT need them: never. They are the exception in this lesson: free, instantaneous and with no trade-off. If you take away a single tool, make it this one line.

5.2. SonarQube / SonarCloud

It already appeared in 02-05. It contributes semantic analysis, technical debt tracking over time and — most usefully — the concept of new code: the quality gate applies only to what the PR changes, not to the 200,000 inherited errors. It is what allows you to apply it to a legacy codebase like the one in 05-04 without blocking anybody.

Trade-offs. Self-hosted SonarQube is one more service to maintain, with its database; SonarCloud is free only for public projects. And it generates a lot of initial noise that has to be calibrated or the team learns to ignore it.

5.3. Semgrep

Syntax-aware pattern search. Its value is that you can write your own rules in minutes, with a syntax that resembles the code you are looking for:

rules:
  - id: no-console-log-in-production
    pattern: console.log(...)
    paths:
      include: ["apps/api/src/**"]
    message: "Use the structured logger, not console.log (see PIPELINE.md)"
    severity: WARNING
    languages: [typescript]

  - id: sql-by-concatenation
    pattern: |
      $DB.query("..." + $VAR)
    message: "Possible SQL injection: use parameterised queries"
    severity: ERROR
    languages: [typescript]

Compared with CodeQL (04-03), Semgrep is faster and far easier to extend; CodeQL goes deeper on data flow analysis. They do not compete as much as it seems: Semgrep for your rules, CodeQL for generic vulnerabilities.

When you do NOT need it. If you do not yet have rules of your own you want to enforce, ESLint with its security plugins covers a good part of it in a TypeScript project. Semgrep comes in when you find yourself saying "we have already commented on this three times in reviews".

  1. Supply chain

Lessons 04-03 and 07-05 covered the essentials. Here is what was missing and the framework that organises it.

6.1. SLSA: the framework that names what you already did

SLSA (Supply-chain Levels for Software Artifacts, slsa.dev) defines progressive levels of build chain integrity. The core idea: being able to demonstrate that the artifact you deploy comes from the code you think it does, built by the process you think it was.

Level What it requires, in essence What you did in the course
Level 1 The build process is documented and generates provenance Lesson 07-05 generates a provenance attestation
Level 2 The build happens in a hosted service, with signed, verifiable provenance GitHub runners + signing with Cosign: you are here
Level 3 In addition, the process is isolated and the provenance is unforgeable even against a compromised builder It requires guarantees from the builder; it cannot be achieved by configuration alone

Its practical usefulness is not certification: it is having the vocabulary to say where you are and what you are missing. In an interview or in front of a client asking about the supply chain, "we meet SLSA level 2 and this is the attestation" is worth far more than listing tools.

6.2. Sigstore and Cosign

Already used in 07-05. What is worth understanding is keyless signing. Instead of managing a private key — the problem that ruins most signing attempts — Cosign obtains an ephemeral certificate tied to the workflow's OIDC identity and records the signature in a public transparency log (Rekor).

# Sign (in the workflow, with no key to manage)
cosign sign --yes "$IMAGE@$DIGEST"

# Verify before deploying: not only that it is signed,
# but WHO signed it and from where
cosign verify \
  --certificate-identity-regexp "https://github.com/reservalia/.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "$IMAGE@$DIGEST"

The frequent mistake is verifying only that a signature exists. Without --certificate-identity-regexp, anyone can sign that image and the verification will pass. The signature answers "who?", not "is it good?".

6.3. Syft, Grype, Trivy and Dependency-Track

  • Syft generates SBOMs (CycloneDX or SPDX) from an image or a directory.
  • Grype scans that SBOM for vulnerabilities.
  • Trivy does both and additionally scans IaC, secrets and Kubernetes configuration. It is what you used in 07-05 and it remains the sensible choice through consolidation.

What is new is Dependency-Track: a service where you store the SBOMs of all your deployed artifacts and which alerts you when a new vulnerability appears affecting something already in production.

- name: Publish SBOM to Dependency-Track
  run: |
    curl -X POST "https://dtrack.internal.example.com/api/v1/bom" \
      -H "X-Api-Key: ${{ secrets.DTRACK_API_KEY }}" \
      -F "project=${{ vars.DTRACK_PROJECT_ID }}" \
      -F "bom=@sbom.cdx.json"

Why this matters and why it is the real gap the course leaves. Your scan in 07-05 answers "does this image have known vulnerabilities today?". The question that breaks companies is the reverse: "a critical vulnerability has just been published in a library, which of my 40 production services is it in?". Without an SBOM inventory, that answer costs days of manual work. With one, it is a query.

When you do NOT need it. With two services, the inventory fits in your head. From ten or fifteen deployed artifacts onwards, it stops fitting.

6.4. Renovate

A considerably more powerful alternative to Dependabot (04-02):

{
  "extends": ["config:recommended"],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch"],
      "matchCurrentVersion": "!/^0/",
      "groupName": "non-major dependencies",
      "automerge": true,
      "schedule": ["before 6am on monday"]
    },
    {
      "matchManagers": ["github-actions"],
      "pinDigests": true,
      "groupName": "GitHub actions"
    }
  ],
  "vulnerabilityAlerts": { "labels": ["security"], "automerge": false }
}

Three things Dependabot does not do as well: grouping updates into a single PR (10 weekly PRs become 1), conditional automerge when CI passes, and pinDigests, which automatically pins actions by SHA and keeps them up to date — resolving the tension in 08-02 between pinning and falling behind. It also covers more package managers and more files (Dockerfile, Terraform, workflows).

Trade-offs. More configuration and a steeper curve. And automerge requires genuinely trusting your test suite: if your CI has a "false green" (04-04), you have just automated the introduction of failures.

When you do NOT need it. Dependabot is built in and is enough until the PR noise becomes unbearable. The signal to migrate: when people start closing dependency PRs without looking at them.

  1. Testing

7.1. Testcontainers

The problem. In 02-04 the integration tests used a database as a workflow service, or doubles. The first means duplicated configuration between local and CI; the second lies about PostgreSQL's real behaviour.

Testcontainers spins up real dependencies in containers from the test code, with a managed life cycle:

import { PostgreSqlContainer } from "@testcontainers/postgresql";

let container: StartedPostgreSqlContainer;

beforeAll(async () => {
  container = await new PostgreSqlContainer("postgres:16-alpine").start();
  process.env.DATABASE_URL = container.getConnectionUri();
  await runMigrations();
}, 60_000);

afterAll(async () => { await container.stop(); });

test("overlapping booking is rejected by the exclusion constraint", async () => {
  await createBooking({ start: "10:00", end: "11:00" });
  await expect(createBooking({ start: "10:30", end: "11:30" }))
    .rejects.toThrow(/exclusion/);
});

That test verifies a PostgreSQL constraint that no double can simulate. And it works the same on your laptop as on the runner, with no duplicated configuration: it is the reproducibility of 02-03 applied to testing.

Trade-offs. Slower (starting containers costs seconds) and it needs Docker available on the runner. It is mitigated by reusing containers across suites.

7.2. Pact and contract testing

The problem, from 05-03. With several services, full E2E tests are slow and brittle, but without them nobody knows whether a change in the API breaks a consumer.

Pact inverts the approach: the consumer declares what it expects, that contract is published, and the provider verifies in its own pipeline that it meets it. Without standing both services up together.

// In the consumer's pipeline (apps/web)
await provider.addInteraction({
  state: "a booking with id 42 exists",
  uponReceiving: "request for booking 42",
  withRequest: { method: "GET", path: "/bookings/42" },
  willRespondWith: {
    status: 200,
    body: { id: 42, status: like("confirmed"), start: iso8601DateTime() }
  }
});

The real value is not technical but organisational: the provider discovers it is about to break somebody before deploying, in its own CI and without coordinating with anyone.

When you do NOT need it. With a single backend and a single frontend in the same repository and deployed together — Reservalia — the contract is guaranteed by the shared types in packages/shared and a handful of integration tests. Pact comes in when services are deployed separately and the teams are different. Before that it is ceremony.

7.3. k6

Load testing as code, runnable in the pipeline:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 50 },
    { duration: '1m',  target: 50 },
    { duration: '20s', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<400'],   // p95 below 400 ms
    http_req_failed:   ['rate<0.01'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/availability?business=42`);
  check(res, { 'status 200': (r) => r.status === 200 });
}

The important part is the thresholds: if they are not met, k6 exits with a non-zero code and the job fails. That turns performance into one more quality gate, rather than a report nobody reads.

Where to put it. Not on every PR: on the main branch after deploying to staging, or in a nightly run. It is the only realistic way to catch a performance regression before a customer catches it — the problem that appeared in 07-04 with the slow endpoint.

When you do NOT need it. If your traffic is nowhere near any limit and there are no latency complaints, it is premature optimisation converted into CI time.

7.4. Mutation testing (Stryker)

The problem, and it is the honest answer to 02-04. Coverage measures which lines are executed during the tests, not which behaviour is verified. A test with not a single assertion gives 100% coverage. It is the easiest metric in the whole pipeline to game, and everybody knows it but few do anything about it.

Mutation testing does what no other technique does: it introduces deliberate faults into your code (changes a > into a >=, inverts a condition, deletes a call) and checks whether any test fails. If nobody complains, that test was verifying nothing.

{
  "testRunner": "vitest",
  "mutate": ["apps/api/src/domain/**/*.ts"],
  "thresholds": { "high": 80, "low": 60, "break": 50 },
  "incremental": true
}

It is, literally, verifying from the negative side — the habit 07-06 pointed to as the most valuable in the course — applied automatically to the whole suite. The first run over a project with 85% coverage and 45% of mutants killed is a formative experience no argument can replace.

Trade-offs, and they are serious. It is extremely slow: every mutant runs the suite. That is why mutate points only at the domain, incremental is enabled, and it runs weekly or on demand, never on every PR.

When you do NOT need it. If your suite is small or your coverage is low, fix that first. Mutation testing is a refinement tool, not a starting one. And in plumbing code (controllers, mappings) it produces irrelevant surviving mutants that only generate noise.

  1. Local development and reproducibility

The problem. "It works on my machine", which 02-03 attacked from the build side, is still alive on the development environment side. And its cousin: the commit-wait-fail cycle for debugging a workflow.

8.1. act

It runs GitHub Actions workflows locally in Docker.

act pull_request -W .github/workflows/ci.yml -j test
act -W .github/workflows/cd.yml --secret-file .secrets.local

It turns a 6-minute cycle into a 40-second one. Important limitations: it does not faithfully emulate environments with reviewers, workflow_run, the Actions cache or token permissions. It works for step logic, not for platform behaviour. With that understood, it saves a lot of time.

8.2. Dev Containers

It defines the development environment as code, in the repository:

{
  "name": "Reservalia",
  "image": "mcr.microsoft.com/devcontainers/typescript-node:20",
  "features": {
    "ghcr.io/devcontainers/features/docker-in-docker:2": {},
    "ghcr.io/devcontainers/features/terraform:1": { "version": "1.7.5" }
  },
  "postCreateCommand": "npm ci",
  "customizations": {
    "vscode": { "extensions": ["dbaeumer.vscode-eslint", "esbenp.prettier-vscode"] }
  }
}

The measurable value: the onboarding time for somebody new drops from a day to half an hour. And it eliminates the whole class of "I've got Node 18" bugs.

8.3. asdf / mise and Nix

asdf and mise pin tool versions per project with a declarative file:

# .tool-versions
nodejs 20.11.1
terraform 1.7.5

Cheap, useful and adopted immediately. Nix offers far stronger reproducibility — down to system dependencies — in exchange for a notorious learning curve. Nix is one of the few tools in this lesson about which you can honestly say: extraordinary if the whole team commits to it, counterproductive if only one person understands it.

8.4. pre-commit

It runs checks before the commit, with a multi-language framework that goes beyond Husky (02-05):

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.2
    hooks: [{ id: gitleaks }]
  - repo: https://github.com/rhysd/actionlint
    rev: v1.6.27
    hooks: [{ id: actionlint }]

A warning 02-05 already gave: local hooks are a convenience, never a guarantee. They are bypassed with --no-verify. Everything that matters must also be checked in CI. Gitleaks in pre-commit is fine because it saves you the embarrassment; Gitleaks in CI is what protects you.

  1. Internal platform

Backstage (created at Spotify, today in the CNCF) is a developer portal: a catalogue of services with owners, templates for creating new services with a pipeline already in place, technical documentation alongside the code and a single interface over scattered tools.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: reservalia-api
  annotations:
    github.com/project-slug: reservalia/monorepo
    grafana/dashboard-selector: "tags @> 'reservalia-api'"
spec:
  type: service
  lifecycle: production
  owner: bookings-team
  system: bookings

The idea that does matter: golden paths. A recommended, documented and automated route for doing the usual thing. "Create a new service" goes from two days of copying and pasting to ten minutes with the pipeline, observability and alerts already configured. And — this is what makes it strategic — whoever steps off the path may do so, but takes on the maintenance.

The warning, and it is the most important in the lesson. Backstage is a software development project in its own right: it has to be deployed, maintained, extended with plugins and upgraded. An internal platform with no users is a vanity project, and there are a great many: beautiful portals nobody opens because the team had ten services and already knew where everything was.

When you do NOT need it. Almost always. The legitimate signal: several teams, dozens of services, and people losing real, measurable time finding who owns what or setting up new services. If you can name all your services and their owners from memory, you do not need it.

And the right order: first the golden path, then the portal. A repository template with the pipeline already set up — what 04-05 called reusable workflows — gives 80% of the benefit at 5% of the cost.

  1. Observability

OpenTelemetry is the cross-cutting standard: a set of APIs, SDKs and a collector that instrument traces, metrics and logs independently of the backend. Its value is not technical but strategic: you instrument once and change provider without touching the code.

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

new NodeSDK({
  serviceName: 'reservalia-api',
  instrumentations: [getNodeAutoInstrumentations()],
}).start();

With the auto-instrumentations you get traces from HTTP, PostgreSQL and the common libraries without writing manual instrumentation. It is the natural extension of 03-06, where traces were left as the least developed pillar.

The rest of the ecosystem completes the pillars: Prometheus (metrics, already used in 07-04), Grafana (visualisation), Loki (logs with the same label model as Prometheus, which lets you jump from a spike to its logs), Tempo (traces). Sentry covers something different and complementary: application errors with a stack trace, intelligent grouping and the context of the deployed version — it answers "which exception and on which line", which neither metrics nor logs answer well.

Automated DORA metrics. Specific tools exist to calculate them, but the honest conclusion — the same one as 07-04 — is that the four metrics are calculated with queries against your forge's API and your incident system's, and that doing it yourself forces you to define precisely what you count as a deployment and as a failure. That definition is half the value, and a tool that makes it for you steals it from you.

When you do NOT need it. Full OpenTelemetry with your own collector is overhead for a small service where the 07-04 metrics and some structured logs are enough. Start with the four golden signals and add traces when you have a distributed latency problem you cannot explain.

  1. Adoption table by team size

Tool 3 people 15 people 50+ people
Pipeline linters (actionlint, hadolint, shellcheck) Essential Essential Essential
Trivy / image scanning Essential Essential Essential
Dependabot Essential Migrate to Renovate Renovate
Cloud secret manager Advisable Essential Essential
Testcontainers Advisable Advisable Advisable
Cosign + SBOM Advisable Essential Essential
Dev Containers / mise Advisable Advisable Essential
act Useful Useful Useful
Semgrep with your own rules Premature Advisable Essential
SonarQube Premature Advisable Advisable
k6 in the pipeline Premature Advisable Essential
Dependency-Track Premature Advisable Essential
Mutation testing Premature Useful, weekly Useful, weekly
Pact Dead weight Advisable if services are separate Essential
GitOps (Argo CD / Flux) Dead weight Advisable with Kubernetes Essential with Kubernetes
Progressive delivery (Flagger / Rollouts) Dead weight Depends on traffic Advisable
Policy as code (OPA / Kyverno) Dead weight Useful for 2-3 critical rules Essential
Self-hosted Vault Dead weight Premature unless required Advisable
Backstage Dead weight Premature Advisable if there is real pain
Full OpenTelemetry Premature Advisable Essential
Nix Dead weight unless fully committed Depends Depends

How to read "dead weight". It does not mean a bad tool: it means that at that size the cost of operating it exceeds the benefit. Vault in a team of three is not a small Vault; it is a whole Vault maintained by people who also write the product. Complexity does not scale downwards.

  1. The criteria for letting a tool in

Three questions. If any of them has no answer, the tool does not come in.

1. What measured problem does it solve?

Measured is the key word. Not "it improves security" but "last month we lost 6 hours rotating a leaked secret, and this would have prevented it". If you cannot cite an incident, a period of lost time or a metric that is getting worse, you do not have a problem: you have curiosity. Curiosity is legitimate — explore it in a personal project, not in the production pipeline.

2. Who maintains it?

A double question. Outside: who develops the project? A company, a foundation, one person? When was the last commit? A CNCF graduated project (08-02) is a very different bet from a personal repository with twelve stars. Inside: who on your team maintains it when it fails on a Friday? If the answer is "whoever brought it in", you have a bus factor of one on a piece of the road to production.

3. What happens if we remove it?

If the answer is "nothing serious", it should not have come in. If it is "we cannot deploy", you need an exit plan before adopting it: how you get out, what it costs, what data has to be migrated. Tools are adopted in an afternoon and abandoned over years.

And a fourth, optional but very revealing: could we get 80% of the benefit with 20 lines of script? Surprisingly often, yes. A grep in a workflow step replaces an OPA policy for a single rule. A repository template replaces Backstage at the start. The answer is not always "use the script", but the question forces you to name what the tool contributes beyond the obvious.

Common Mistakes and Tips

  • Adopting by fashion rather than by problem. The central mistake of the module. It is recognised by one signal: you cannot explain in one sentence what will stop hurting.
  • Adopting several at once. When something breaks you do not know which one did it. One tool at a time, with at least two weeks of real use before the next.
  • Confusing "a big company uses it" with "it is useful to me". They have a dedicated platform team. It is the 08-02 warning sign applied to tools.
  • Putting slow tools on the critical path. Mutation testing or k6 on every PR turn a 4-minute CI into a 40-minute one, and the team starts skipping it. Slow things go outside the PR: nightly, weekly or post-merge (04-04).
  • Deploying policies in blocking mode from day one. Warning mode first, always. A badly written policy blocks everybody and burns the whole idea for months.
  • Verifying signatures without verifying identity. cosign verify without --certificate-identity-regexp gives a false sense of security. Anyone can sign.
  • Automerge without trusting the suite. Renovate with automerge over a CI with a false green automates the introduction of failures.
  • Building the platform before the path. First the template and the reusable workflow; the portal afterwards, and only if it hurts.
  • Tip: always start with the pipeline linters. Five minutes, zero cost, immediate benefit. No other tool in this lesson has that ratio.
  • Tip: document every adoption in PIPELINE.md in the 07-06 format: which problem, what was ruled out, which condition would force a review. Your future self two years from now needs to know why that is there.
  • Tip: write down the rejections too. "We evaluated Vault in March and ruled it out because X" saves somebody reopening the discussion every six months.

Exercises

Exercise 1 — Three teams, three decisions

For each situation, decide which tool (or none) from this lesson you would adopt, justify it with the three questions from section 12, and say what you rule out and why:

a) A team of 4, a Node.js monolith on ECS, one environment. They have suffered two incidents in three months from badly rotated database credentials: somebody changes the password in RDS and forgets to update it somewhere. Tooling budget: low but not zero.

b) A team of 18, 12 microservices on Kubernetes, deploying about 30 times a week. When a critical vulnerability is published in a popular library, it takes them one to two days to know which services are affected.

c) A team of 6, a web application with 400 internal users. The tech lead proposes setting up Backstage "so that everything is organised and because it is what everyone is doing".

Exercise 2 — Reservalia's adoption plan

Reservalia is today where module 4 left it: 12 deployments/week, 3.5 h lead time, 3.8% CFR, 9 min restore. A team of three. ECS Fargate, GitHub Actions, no Kubernetes. Diego is still watching every euro.

Choose three tools from this lesson to adopt over the next six months, in order, and for each one: which DORA metric (or which other concrete pain) it aims to move, how you would justify it to Diego, and what signal would indicate that the adoption was a mistake.

Exercise 3 — The reasoned rejection

Choose one tool from this lesson that appeals to you and that you have decided not to adopt in your context. Write the corresponding PIPELINE.md entry, in the 07-06 format: which problem it would solve, why it is rejected today, which cheaper alternative is used instead, and the concrete, observable condition that would force the decision to be reopened.

Solutions

Solution 1

a) A team of 4, incidents from credential rotation.

Adopt: AWS Secrets Manager, with the application reading the secret by reference from the ECS task definition.

  • What measured problem? Two incidents in three months with an identical cause. The root cause is not carelessness: it is that the same value lives in three places and consistency depends on somebody remembering all three. It is a design problem, and removing the duplication removes it.
  • Who maintains it? AWS. A managed service, with no operational burden on the team. Automatic rotation configurable with Lambda if it becomes necessary later.
  • What happens if we remove it? You go back to the previous state with one change to the task definition. A cheap exit with no data migration.

Ruled out: Vault. It would solve the same thing and more, but it requires operating a critical system with four people who are already at 100%. It is exactly the "dead weight" case from the table in section 11.

Ruled out: SOPS. It would version the secrets, but it does not solve the stated problem — the application would still read the value from the deployed configuration, and rotation would still be a coordinated change.

b) 18 people, 12 microservices, one or two days to know whether they are affected.

Adopt: Dependency-Track, fed with the SBOMs the pipeline should already be generating with Syft or Trivy.

  • What measured problem? 24 to 48 hours of manual work for every critical vulnerability published. It is expensive time from expensive people, it is recurrent and — most seriously — it is time during which you are exposed without knowing it.
  • Who maintains it? An OWASP project, with an established community. Inside: an owner has to be assigned; with 18 people that is viable.
  • What happens if we remove it? You go back to per-artifact scanning and manual response. The SBOMs keep being generated, so the exit loses nothing.

The key to the reasoning: their current scanning answers "is this image safe today?" and their problem is the reverse, "where is this library right now?". No amount of pipeline scanning answers that; you need an inventory.

A reasonable complement: Renovate with grouping, because with 12 services Dependabot's noise is high and most of those vulnerabilities are closed by updating.

c) 6 people, 400 internal users, a Backstage proposal.

None. Reject the proposal, and do it with the three questions rather than with an opinion:

  • What measured problem? None stated. "So that everything is organised" is not a problem: it is an aesthetic aspiration. The question that settles the conversation: how many hours have we lost over the last three months looking for who owns a service, or setting up a new one? With six people, the answer will be close to zero, because the catalogue fits in a conversation.
  • "Because it is what everyone is doing" is literally the 08-02 warning sign and the central antipattern of this lesson.
  • What happens if we remove it? Nothing. That the answer is "nothing" before you have even adopted it is the proof that it should not come in.

A constructive counter-proposal, because rejecting without an alternative usually fails: if what genuinely bothers people is inconsistency between services, 80% of the benefit is in a repository template with the pipeline already set up and a well-maintained CODEOWNERS (02-07, 04-05). Cost: an afternoon. And if in a year there are 30 services and people getting lost, Backstage comes in with a measured problem behind it.

Solution 2

There are several defensible combinations. This one prioritises zero cost, low risk and verifiable benefit, which is what Reservalia's context demands.

First: the pipeline linters (actionlint, hadolint, shellcheck, tflint). Month 1.

  • What it moves: lead time. Every trial-and-error iteration with Actions YAML costs a full CI cycle. actionlint catches them in seconds, locally and in the PR.
  • To Diego: zero cost in euros, about 15 seconds of CI, and an afternoon of configuration. The argument that convinces him is about runner minutes: fewer failed runs from syntax errors is literally less spend.
  • Failure signal: if it generates warnings the team systematically silences, the configuration is badly calibrated, not the tool. You adjust rules, you do not remove it.

Second: Testcontainers for the integration tests. Months 2-3.

  • What it moves: change failure rate (3.8%). The failures that reach production in a system with good unit coverage usually come from the database boundary: constraints, transactions, PostgreSQL behaviour no double reproduces. And along the way it reduces time to restore, because these are failures caught earlier.
  • To Diego: free, and the argument is one of avoided cost. A production incident at Reservalia costs more in the three people's time than the entire month's CI. It can be quantified by looking at the last three incidents and asking how many would have been caught by a test against a real PostgreSQL.
  • Failure signal: if CI gets more than two minutes longer and no caught failure appears in three months, it is surplus. You measure before and after.

Third: k6 with thresholds, after deploying to staging, not on every PR. Months 4-6.

  • What it moves: change failure rate and time to restore, in the failure mode Reservalia handles worst: gradual degradation. An endpoint that goes from 200 ms to 900 ms breaks no test and fires no alert until the SLO starts being consumed. It is exactly what appeared in 07-04.
  • To Diego: free, run once per deployment to staging, about 2 minutes outside the PR's critical path. And the business argument is strong: the critical booking window is where the load is highest and where an outage costs customers — constraint 3 of the 07-06 brief.
  • Failure signal: if the thresholds fail constantly because of noise from the staging environment and people start retrying the job without looking, the problem is the environment or the thresholds. A threshold that gets ignored is worse than not having it, because it teaches the team to ignore gates.

Explicitly ruled out and why: Vault (dead weight for three people), GitOps and progressive delivery (they require Kubernetes; Reservalia is on ECS), Backstage (no measured problem), policy as code (with three people the rules fit in the review; if one is critical, a grep in the workflow is enough), mutation testing (interesting, but the priority is for the tests to catch real failures before refining the ones already there).

Solution 3

An example, on automated progressive delivery:

## Decision 14 — Automated progressive delivery (Flagger / Argo Rollouts)

**Date:** 2026-08-02
**Status:** Rejected for now

**Problem it would solve.** Today the production deployment is rolling with
health checks, and the "this is going badly, roll back" decision is made by a
person watching Grafana for the following ten minutes. That depends on
somebody watching, the criterion is subjective and in practice we promote out
of impatience. Flagger or Argo Rollouts would turn that judgement into
declared thresholds with automatic metric analysis.

**Why it is rejected today.** Two reasons, and the second is decisive:
1. Both tools are Kubernetes-only. Reservalia runs on ECS Fargate and
   adopting them would mean migrating the entire platform — a
   disproportionate cost for the problem they solve.
2. Even if we were on Kubernetes, **we do not have the traffic volume for
   the analysis to be statistically significant**. At our current peak, a
   canary at 10% for 5 minutes receives a few dozen requests: it cannot
   tell a 1% error rate from a 4% one. The automatic analysis would give a
   false sense of rigour.

**Alternative in use.** Rolling with health checks + rollback by digest in
~4 minutes (decision 6) + symptom-based alerting on the SLO (decision 11).
The measured time to restore is 9 minutes, within target.

**Review condition.** This decision is reopened if both hold:
- (a) sustained traffic exceeds 500 requests/minute in the usual deployment
  window, so that a canary at 10% generates signal within 5 min; and
- (b) we migrate to Kubernetes for some other independent reason, or we adopt
  an equivalent tool compatible with ECS.

While only (a) holds, the correct action is to lengthen the post-deployment
observation window and automate metric-based rollback inside `cd.yml`
itself, not to change platform.

What makes this entry valid is not the rejection but the fact that the review condition is observable and not a matter of opinion: "500 requests per minute" can be checked in Grafana; "when we have more traffic" can never be checked and guarantees the discussion reopens every six months with no new data.

Conclusion

This whole module has been organised around one idea worth stating clearly: a tool is an answer, and it only makes sense if the question exists. All the tools in this lesson are good; none of them is good for everyone.

The essentials:

  • Secrets: the best secret is the one that does not exist (OIDC, roles). When the application needs secrets at runtime, the cloud manager is the sensible answer; Vault comes in with several clouds or audit requirements; SOPS+age is an underrated option for versioning encrypted configuration.
  • Progressive delivery and GitOps solve real problems from 03-04 and 03-02, but they require Kubernetes and, in the case of the automatic canary, enough traffic for the metrics to mean anything.
  • Policy as code turns review rules into automated checks: valuable from several teams onwards, overhead before that, and always deployed in warning mode first.
  • Pipeline linters are the only recommendation with no trade-off in the whole lesson. Five minutes, zero euros, immediate benefit.
  • Supply chain: SLSA gives you the vocabulary to say where you are; verifying a signature with identity is what makes signing worthwhile; and the SBOM inventory answers the question that breaks companies, which is not "is this image safe?" but "where is this library right now?".
  • Testing: Testcontainers brings tests closer to reality, k6 turns performance into a gate, and mutation testing is the honest answer to the lie of coverage — it is verifying from the negative side, automated.
  • Internal platform: first the golden path, then the portal. A platform with no users is a vanity project.
  • And the criteria, which are worth more than the catalogue: what measured problem does it solve? Who maintains it inside and outside? What happens if we remove it? Plus the fourth, uncomfortable and revealing: would twenty lines of script do?

The adoption table by size says something worth repeating: complexity does not scale downwards. Vault in a team of three is not a small Vault. Backstage with six services is not a lightweight Backstage. Copying the architecture of a five-hundred-person company without having its problems leaves you with its costs and without its benefits.

You now have the foundation (08-01), the context (08-02) and the catalogue (08-03). What is missing is the question none of the three answers: where are you heading? What you can do now and what you cannot, in which direction to go deeper, which certifications are worth something and which are not, how to demonstrate what you know without a piece of paper, and how to keep up to date without burning out.

That is the last lesson of the course: 08-04, Learning Path, Certifications and Next Steps.

CI/CD Course: Continuous Integration and Deployment

Module 1: Introduction to CI/CD

Module 2: Continuous Integration (CI)

Module 3: Continuous Deployment (CD)

Module 4: Advanced CI/CD Practices

Module 5: Implementing CI/CD in Real Projects

Module 6: Tools and Technologies

Module 7: Practical Exercises

Module 8: Additional Resources

© Copyright 2026. All rights reserved