The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Exercise 3: Deploying to a Production Environment

Next ►
►

So far the pipeline produces a dist/ directory that is kept for seven days and goes nowhere. In this lesson you close the loop: you will package Mini-Reservalia into an immutable container image, publish it to a real registry, deploy it to two environments with a human approval gate between them, automatically check that the deployment works, promote to production exactly the same artifact you validated in staging —without rebuilding it— and write the emergency button that takes you back to the previous version in under two minutes. And, as this module demands, you will break something on purpose: you will deploy a version that fails its own health check so you can watch the gate close and the rollback work.

All of this without AWS and without a credit card. The registry is ghcr.io, included free with your GitHub account; the frontend goes to GitHub Pages; and the "production host" is a Docker container running on the runner itself or on your machine. Wherever the real Reservalia would use ECS, ECR and OIDC against AWS, you will find a note with the exact equivalent. The concepts —immutable artifact, digest, idempotency, promotion, smoke test, rollback— are identical; only where the container lands changes.

Contents

  1. Objective, prerequisites and starting point
  2. The immutable artifact: a multi-stage Dockerfile
  3. Building and publishing to ghcr.io from the pipeline
  4. Why the digest and not the tag
  5. Separating CI from CD with workflow_run
  6. Environments: automatic staging and production with a reviewer
  7. The idempotent deployment script
  8. The three ways to run it: with a host, without a host, and on the runner
  9. The smoke test with retries
  10. The complete cd.yml with promotion by digest
  11. The frontend on GitHub Pages
  12. The rollback.yml and the stopwatch
  13. Causing a bad deployment
  14. Final verification
  15. Common Mistakes and Tips
  16. Exercises
  17. Conclusion

  1. Objective, prerequisites and starting point

Objective. By the end, a merge to main will build an image, publish it to ghcr.io, deploy it automatically to staging, verify it with a smoke test, wait for your approval and promote the same digest to production; and you will have a rollback.yml that reverts in under two minutes, timed.

Prerequisites.

  • Lessons 07-01 and 07-02 completed.
  • Docker installed on your machine (docker --version, 24 or above) and docker compose version. Now it really is mandatory.
  • The repository on GitHub with main protected and the CI OK check.

Starting point. The repository as it stood after 07-02: code with persistence, three layers of tests, coverage with a threshold, matrix and sharding.

git checkout main && git pull
git checkout -b deployment

  1. The immutable artifact: a multi-stage Dockerfile

The rule from 02-06: build once, deploy many. An artifact is built exactly once and that same artifact —byte for byte— travels through every environment. If each environment rebuilds, each environment runs something different and validating staging tells you nothing about production.

Dockerfile:

# syntax=docker/dockerfile:1.7

# ---------- Stage 1: production dependencies ----------
# Isolated so the layer stays cached for as long as the lockfile does not change.
FROM node:20-bookworm-slim AS deps
WORKDIR /app
# build-essential and python3 are needed ONLY if some native module
# (better-sqlite3) cannot find a prebuilt binary. They stay in this
# stage and never reach the final image.
RUN apt-get update && apt-get install -y --no-install-recommends \
      python3 make g++ \
    && rm -rf /var/lib/apt/lists/*
COPY package.json package-lock.json ./
# --omit=dev: no eslint and no test tooling in production.
RUN npm ci --omit=dev

# ---------- Stage 2: build ----------
FROM node:20-bookworm-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
ARG COMMIT=unknown
ENV GITHUB_SHA=${COMMIT}
RUN npm run build

# ---------- Stage 3: final image ----------
FROM node:20-bookworm-slim AS runtime
WORKDIR /app

# Unprivileged user. The official Node image already ships the `node`
# user (uid 1000); there is no need to create it.
ENV NODE_ENV=production \
    PORT=3000 \
    DATABASE_URL=sqlite:/data/mini.db

# Data directory with permissions for the non-root user.
RUN mkdir -p /data && chown -R node:node /data

COPY --from=deps  --chown=node:node /app/node_modules ./node_modules
COPY --from=build --chown=node:node /app/dist ./dist
COPY --chown=node:node package.json ./

USER node
EXPOSE 3000
VOLUME ["/data"]

# OCI metadata: who built this, from which commit and when.
# It is read with `docker inspect` and is the minimum traceability of an artifact.
ARG COMMIT=unknown
ARG BUILT_AT=unknown
LABEL org.opencontainers.image.source="https://github.com/OWNER/mini-reservalia" \
      org.opencontainers.image.revision="${COMMIT}" \
      org.opencontainers.image.created="${BUILT_AT}" \
      org.opencontainers.image.title="mini-reservalia" \
      org.opencontainers.image.description="Appointment slot calculation"

# HEALTHCHECK: Docker probes the container and marks its state.
# `docker inspect --format '{{.State.Health.Status}}'` returns
# starting -> healthy | unhealthy. The deployment script will use it.
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
  CMD node -e "fetch('http://127.0.0.1:'+(process.env.PORT||3000)+'/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

ENV APP_VERSION=${COMMIT}
CMD ["node", "dist/src/server.js"]

.dockerignore —every bit as important as the Dockerfile, because it determines what gets sent to the daemon and what invalidates the cache—:

node_modules
dist
coverage
reports
.git
.github
.gitignore
*.md
*.log
.env
Dockerfile
docker-compose*.yml
test

Excluding test/ and .git is not just about speed: it is about surface area. A production image carrying the Git history and the tests inside it is an image that leaks information and takes up three times the space.

Build it and try it locally:

docker build \
  --build-arg COMMIT="$(git rev-parse HEAD)" \
  --build-arg BUILT_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -t mini-reservalia:local .

docker run --rm -d --name mini-local -p 3000:3000 mini-reservalia:local
sleep 3
curl -s localhost:3000/health
# {"status":"ok","version":"8f3c1e2...","uptimeSec":3}

# The HEALTHCHECK state, which is what the deployment will look at:
docker inspect --format '{{.State.Health.Status}}' mini-local
# healthy    (it can take up to 15 s to move on from "starting")

# Basic hardening checks:
docker exec mini-local whoami       # node   (NOT root)
docker image inspect mini-reservalia:local --format '{{.Size}}' | numfmt --to=iec
# ~230M

docker rm -f mini-local

What you should see: status: ok, Health.Status: healthy and whoami: node. If whoami returns root, the USER node line is missing or sits before a COPY that cancels it out.

  1. Building and publishing to ghcr.io from the pipeline

ghcr.io is GitHub's container registry. For a public repository it is free and unlimited, and best of all: you do not need any new credentials. The GITHUB_TOKEN the runner already has will do, as long as you grant it the packages: write permission.

Add a publish job to ci.yml that depends on ci-ok:

  publish:
    name: Publish image
    runs-on: ubuntu-latest
    needs: [ci-ok]
    # Only publish from main: a PR must not leave images in the registry.
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    permissions:
      contents: read
      packages: write      # required to write to ghcr.io
    outputs:
      digest: ${{ steps.build.outputs.digest }}
      image: ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to ghcr.io
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}   # ephemeral token for the run

      - name: Metadata and tags
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha,format=long,prefix=sha-
            type=raw,value=main,enable={{is_default_branch}}

      - name: Build and push
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          build-args: |
            COMMIT=${{ github.sha }}
            BUILT_AT=${{ github.event.repository.updated_at }}
          # Layer cache on the Actions backend itself: it cuts the time of
          # subsequent builds dramatically (04-04 and 06-05).
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: true    # provenance attestation (SLSA)

      - name: Publish the digest in the summary
        run: |
          {
            echo "## Image published"
            echo ""
            echo "| Field | Value |"
            echo "|---|---|"
            echo "| Repository | \`ghcr.io/${{ github.repository }}\` |"
            echo "| Digest | \`${{ steps.build.outputs.digest }}\` |"
            echo "| Commit | \`${{ github.sha }}\` |"
            echo ""
            echo "Immutable reference to deploy:"
            echo '```'
            echo "ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}"
            echo '```'
          } >> "$GITHUB_STEP_SUMMARY"

Merge the PR and watch. What you should see:

  1. In the run summary, the table with the digest: sha256:3f9a....
  2. On your profile page or the repository page, the Packages section with mini-reservalia.
  3. The image is private by default even if the repository is public. Go to Package settings → Change visibility → Public if you want to be able to pull it without authenticating (handy for the lab). Alternatively, in Package settings → Manage Actions access, add the repository with the Write role.

Pull it from your machine to check it really exists:

echo "$GH_TOKEN" | docker login ghcr.io -u YOUR_USERNAME --password-stdin
docker pull ghcr.io/YOUR_USERNAME/mini-reservalia@sha256:3f9a...
docker run --rm -p 3000:3000 ghcr.io/YOUR_USERNAME/mini-reservalia@sha256:3f9a...

Real equivalent in Reservalia. The registry is ECR and authentication does not use a password but OIDC: the workflow asks AWS for an ephemeral token with aws-actions/configure-aws-credentials@v4 and role-to-assume: arn:aws:iam::...:role/reservalia-ci, with no key stored in GitHub at all (03-02 and 04-03). Here ghcr.io with GITHUB_TOKEN follows exactly the same principle —an ephemeral, minimally scoped credential that expires when the run ends—, which is why the exercise loses nothing pedagogically. Lesson 07-05 will come back to this.

  1. Why the digest and not the tag

This section is short and it is the most important one in the lesson.

A tag is a mutable pointer. ghcr.io/you/mini-reservalia:main points at one image today and a different one tomorrow. A digest (sha256:...) is the hash of the manifest's content: a reference by digest always resolves to the same bytes, forever.

Tag Digest
:main, :latest, :v1.2 Mutable pointer —
@sha256:3f9a... — Immutable content
Can it change under your feet? Yes No
Is it any good for promotion? No Yes
Is it any good for rolling back? Only if nobody moved it Always
Is it any good for talking to humans? Yes Not really

The scenario that ruins somebody's day every month: you deploy :main to staging, test it, approve it, and by the time the production job runs docker pull, another merge has already moved :main. Production is running code nobody validated. This is not a theoretical failure; it is the reason this pipeline passes the digest from one job to the next and the reason the production job checks that the digest it is about to deploy is the very same one that was validated.

See it for yourself:

docker buildx imagetools inspect ghcr.io/YOUR_USERNAME/mini-reservalia:main --format '{{.Manifest.Digest}}'
# sha256:3f9a...   <- right now
# do another merge to main and repeat: the digest has changed, the tag has not

The rule: tags are for people, digests are for machines. Publish both; always deploy by digest.

  1. Separating CI from CD with workflow_run

CI and CD are two cycles with different speeds and different permissions. CI runs on every PR, needs no deployment credentials and must be fast. CD runs only after a merge to main, needs elevated permissions and may spend minutes waiting for an approval. Putting them in the same workflow forces you to give every PR the deployment permissions: exactly the opposite of least privilege.

flowchart LR
    P["push to main"] --> CI["ci.yml<br/>quality, test, coverage,<br/>build, publish image"]
    CI -->|workflow_run: completed + success| CD["cd.yml"]
    CD --> S["deploy staging<br/>automatic"]
    S --> SM1["smoke test"]
    SM1 --> G{"Environment<br/>production<br/>reviewer required"}
    G -->|approved| PR2["deploy production<br/>SAME digest"]
    PR2 --> SM2["smoke test"]
    SM2 -->|fails| RB["rollback.yml"]

The trigger:

on:
  workflow_run:
    workflows: ['CI']        # the `name:` of the other workflow, not its file
    types: [completed]
    branches: [main]
  workflow_dispatch:          # so a deployment can be relaunched by hand
    inputs:
      digest:
        description: 'Digest to deploy (sha256:...). Empty = latest on main'
        required: false

And the indispensable guard on the first job:

    # `completed` includes failure and cancelled. Without this condition
    # you would be deploying the result of a red CI run.
    if: >-
      github.event_name == 'workflow_dispatch' ||
      github.event.workflow_run.conclusion == 'success'

Two quirks of workflow_run that confuse everybody the first time round:

  1. The workflow must exist on main for it to fire. While cd.yml lives only on your branch it will never run, no matter how green CI is. You have to merge it first and test it afterwards. It is counter-intuitive and it costs half an afternoon to anyone who does not know.
  2. The context is the triggering workflow's, not the commit's. github.sha inside a workflow_run is the SHA of the default branch at the moment of the trigger. If you need the exact commit that was built, read it from github.event.workflow_run.head_sha.

  1. Environments: automatic staging and production with a reviewer

GitHub Environments are the materialisation of the gate between Delivery and Deployment from 03-01: the artifact is ready, but somebody decides when it goes in.

Create them in Settings → Environments:

staging

  • No protection rules.
  • Environment URL: the address of your staging (or http://localhost:3001).
  • Environment variable (Variables tab): APP_PORT = 3001.

production

  • Required reviewers: add yourself. This is the whole point of the exercise.
  • Wait timer: 0 (or 1 minute if you want to see the countdown).
  • Deployment branches: Selected branches → main. Stops production being deployed from any old branch.
  • Variable: APP_PORT = 3002.

With gh:

gh api --method PUT "repos/{owner}/{repo}/environments/staging"

gh api --method PUT "repos/{owner}/{repo}/environments/production" \
  -F "wait_timer=0" \
  -F "reviewers[][type]=User" \
  -F "reviewers[][id]=$(gh api user --jq .id)" \
  -F "deployment_branch_policy[protected_branches]=true" \
  -F "deployment_branch_policy[custom_branch_policies]=false"

A job is bound to an environment with two lines:

    environment:
      name: production
      url: ${{ steps.deploy.outputs.url }}

What GitHub does with that: when the run reaches that job, it stops, marks the run as Waiting, sends a notification to the reviewers and waits. Not a single step of the job runs —not even the checkout— until somebody approves. The environment's secrets and variables only materialise after the approval, which means an unapproved job cannot touch the production credentials, neither by accident nor on purpose. That is the security value, on top of the process value.

  1. The idempotent deployment script

Here we apply the rule from 06-07: logic in scripts, thin YAML. The script must be runnable from your laptop, from the runner or from a server over SSH, unchanged. If it only works inside GitHub Actions, you cannot debug it and you cannot use it in an emergency.

scripts/deploy.sh:

#!/usr/bin/env bash
# Deploys Mini-Reservalia as a Docker container.
#
# IDEMPOTENT: running it N times with the same image leaves the system in the
# same state as running it once. That property is what makes it safe to retry
# a failed deployment (03-02).
#
# Usage:
#   ENVIRONMENT=staging PORT=3001 IMAGE=ghcr.io/x/y@sha256:... ./scripts/deploy.sh
#
# Variables:
#   IMAGE        (required) FULL reference by digest
#   ENVIRONMENT  (default: staging) suffix for the container name
#   PORT         (default: 3001) host port
#   KEEP         (default: 3) how many old images to keep

set -Eeuo pipefail   # -E: traps are inherited; -e: abort on failure;
                     # -u: an undefined variable is an error; -o pipefail: the whole pipe fails

IMAGE="${IMAGE:?IMAGE is missing (reference by digest)}"
ENVIRONMENT="${ENVIRONMENT:-staging}"
PORT="${PORT:-3001}"
KEEP="${KEEP:-3}"

CONTAINER="mini-reservalia-${ENVIRONMENT}"
VOLUME="mini-reservalia-data-${ENVIRONMENT}"

log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }

# --- 0. Early validations -----------------------------------------------------
if [[ "$IMAGE" != *"@sha256:"* ]]; then
  echo "ERROR: IMAGE must be a reference by DIGEST (@sha256:...), not by tag." >&2
  echo "       Received: $IMAGE" >&2
  echo "       A tag is mutable: it does not guarantee you deploy what you validated." >&2
  exit 2
fi
command -v docker >/dev/null || { echo "ERROR: docker is not installed" >&2; exit 3; }

log "Environment: $ENVIRONMENT"
log "Container:   $CONTAINER"
log "Port:        $PORT"
log "Image:       $IMAGE"

# --- 1. Pull the image BEFORE touching anything -------------------------------
# If the pull fails, the current service stays alive. Never stop what works
# before you have what is going to replace it.
log "Pulling image..."
docker pull --quiet "$IMAGE"

LOCAL_DIGEST="$(docker image inspect "$IMAGE" --format '{{index .RepoDigests 0}}')"
log "Digest verified: $LOCAL_DIGEST"

# --- 2. Idempotency: if THIS image is already running, do nothing -------------
if docker ps --filter "name=^${CONTAINER}$" --format '{{.Names}}' | grep -q .; then
  CURRENT="$(docker inspect "$CONTAINER" --format '{{.Image}}')"
  INCOMING="$(docker image inspect "$IMAGE" --format '{{.Id}}')"
  if [[ "$CURRENT" == "$INCOMING" ]]; then
    log "The container already runs this image. Nothing to do (idempotency)."
    docker ps --filter "name=^${CONTAINER}$" --format 'table {{.Names}}\t{{.Status}}'
    exit 0
  fi
  log "A different version is running; it will be replaced."
fi

# --- 3. Save the previous version for the rollback ----------------------------
PREVIOUS=""
if docker inspect "$CONTAINER" >/dev/null 2>&1; then
  PREVIOUS="$(docker inspect "$CONTAINER" --format '{{index .Config.Labels "mini.digest"}}' 2>/dev/null || true)"
fi
if [[ -n "$PREVIOUS" ]]; then
  log "Previous version (for rollback): $PREVIOUS"
  echo "$PREVIOUS" > "/tmp/mini-reservalia-${ENVIRONMENT}.previous"
  if [[ -n "${GITHUB_OUTPUT:-}" ]]; then
    echo "previous_digest=$PREVIOUS" >> "$GITHUB_OUTPUT"
  fi
fi

# --- 4. Data volume (idempotent by definition) --------------------------------
docker volume create "$VOLUME" >/dev/null

# --- 5. Stop and remove the previous one --------------------------------------
# `|| true` because on the first deployment it does not exist, and that is not an error.
log "Stopping the previous version (if any)..."
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true

# --- 6. Start the new version -------------------------------------------------
log "Starting the new version..."
docker run -d \
  --name "$CONTAINER" \
  --restart unless-stopped \
  -p "${PORT}:3000" \
  -v "${VOLUME}:/data" \
  -e "APP_VERSION=${APP_VERSION:-$IMAGE}" \
  -e "ENVIRONMENT=${ENVIRONMENT}" \
  --label "mini.digest=${IMAGE}" \
  --label "mini.environment=${ENVIRONMENT}" \
  --label "mini.deployed=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --health-cmd "node -e \"fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))\"" \
  --health-interval 5s --health-retries 6 --health-start-period 3s \
  "$IMAGE" >/dev/null

# --- 7. Wait for the HEALTHCHECK to turn healthy ------------------------------
log "Waiting for the container to become healthy..."
for i in $(seq 1 24); do
  STATE="$(docker inspect "$CONTAINER" --format '{{.State.Health.Status}}' 2>/dev/null || echo 'no-data')"
  case "$STATE" in
    healthy)   log "Healthy after ${i} probes."; break ;;
    unhealthy) log "ERROR: the container is unhealthy."; docker logs --tail 50 "$CONTAINER"; exit 4 ;;
    *)         sleep 5 ;;
  esac
  if [[ "$i" -eq 24 ]]; then
    log "ERROR: it did not become healthy within 120 s (state: $STATE)."
    docker logs --tail 50 "$CONTAINER"
    exit 5
  fi
done

# --- 8. Cleanup: keep the last N images ---------------------------------------
# Without this, the host disk fills up in a few weeks. It is the number one
# cause of "the deployment failed and we do not know why" on small hosts.
log "Cleaning up old images (keeping $KEEP)..."
docker image prune -f --filter "until=168h" >/dev/null 2>&1 || true

log "Deployment complete: $CONTAINER on port $PORT"
docker ps --filter "name=^${CONTAINER}$" --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
chmod +x scripts/deploy.sh

The four properties that make this script usable in production:

Property How it is achieved What happens without it
Idempotent Checks whether that image is already running and exits (step 2) Retrying a deployment restarts the service for no reason
Fails early pull before stopping anything (step 1) You stop what works and then discover the new image does not exist
Verifiable Waits for healthy with a time limit (step 7) The script finishes "fine" with a container that does not even start
Traceable Saves the previous digest in a label (steps 3 and 6) You have no idea what to go back to in a rollback

That last point deserves emphasis: the mini.digest label turns the container itself into the record of what is deployed. docker inspect mini-reservalia-production --format '{{index .Config.Labels "mini.digest"}}' answers the most urgent question of any incident: what on earth are we running?

  1. The three ways to run it: with a host, without a host, and on the runner

The same script, three destinations. Pick the one you can manage.

Option A (the main one, free): the simulated target inside the runner

The deployment job runs the script against the runner's own Docker. The "server" lives for the three minutes of the job and is destroyed afterwards. It is artificial —there is no persistence between deployments— but it exercises 100 % of the path: pull by digest, startup, health check, smoke test, rollback. It is this lab's default option.

      - name: Deploy
        run: ./scripts/deploy.sh
        env:
          IMAGE: ${{ needs.prepare.outputs.image }}
          ENVIRONMENT: staging
          PORT: '3001'

Option B: your machine, with a self-hosted runner

If you want real persistence between deployments and to see the container alive on your laptop, register a self-hosted runner:

mkdir ~/runner && cd ~/runner
# Copy the exact commands from Settings > Actions > Runners > New self-hosted runner
./config.sh --url https://github.com/YOUR_USERNAME/mini-reservalia --token XXXX --labels home
./run.sh

And swap runs-on: ubuntu-latest for runs-on: [self-hosted, home]. Now http://localhost:3001 and http://localhost:3002 are two real environments of yours, they survive deployments and you can see them with docker ps.

Security warning, not optional. Do not put a self-hosted runner on a public repository: anyone opening a PR could run arbitrary code on your machine. Lesson 06-06 explained it and 07-05 will come back to it. If you go for option B, make the repository private or use a disposable container as the runner.

Option C: a real host over SSH

If you have an accessible machine (an old VM, a Raspberry Pi, a one-euro VPS), this is the closest thing to production:

      - name: Deploy over SSH
        env:
          IMAGE: ${{ needs.prepare.outputs.image }}
        run: |
          install -m 600 /dev/null key
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > key
          # StrictHostKeyChecking=accept-new: trust the first time and then
          # detect host changes. NEVER use `no`: it turns off protection
          # against impersonation for good.
          scp -i key -o StrictHostKeyChecking=accept-new \
            scripts/deploy.sh "${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}:/tmp/"
          ssh -i key -o StrictHostKeyChecking=accept-new \
            "${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}" \
            "IMAGE='$IMAGE' ENVIRONMENT=production PORT=3002 bash /tmp/deploy.sh"
          shred -u key

With a docker-compose.yml on the host so you also get the reverse proxy:

# docker-compose.yml - the simulated "production host" on your machine
services:
  api-staging:
    image: ${STAGING_IMAGE:-ghcr.io/OWNER/mini-reservalia:main}
    container_name: mini-reservalia-staging
    restart: unless-stopped
    ports: ['3001:3000']
    environment:
      ENVIRONMENT: staging
      DATABASE_URL: sqlite:/data/mini.db
    volumes: ['data-staging:/data']
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
      interval: 10s
      timeout: 3s
      retries: 3

  api-production:
    image: ${PRODUCTION_IMAGE:-ghcr.io/OWNER/mini-reservalia:main}
    container_name: mini-reservalia-production
    restart: unless-stopped
    ports: ['3002:3000']
    environment:
      ENVIRONMENT: production
      DATABASE_URL: sqlite:/data/mini.db
    volumes: ['data-production:/data']
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
      interval: 10s
      timeout: 3s
      retries: 3

volumes:
  data-staging:
  data-production:

Real equivalent in Reservalia. The deployment is aws ecs update-service with a new task definition pointing at the digest, and ECS performs a rolling update honouring the health check of the ALB target group (03-02 and 03-04). Reservalia's deploy.sh script does the same as yours —check idempotency, apply, wait for stabilisation, report— with aws ecs wait services-stable instead of the docker inspect loop. The shape of the script is identical; only the primitive changes.

  1. The smoke test with retries

A deployment that finishes is not a deployment that works. The smoke test is the difference.

scripts/smoke.sh:

#!/usr/bin/env bash
# Post-deployment smoke test.
# It does not test functionality exhaustively (that is what the suite is for):
# it checks that the DEPLOYED system responds and serves its main purpose.
#
# Usage: BASE=http://localhost:3001 ./scripts/smoke.sh [expected_digest]

set -Eeuo pipefail

BASE="${BASE:?BASE is missing (e.g. http://localhost:3001)}"
EXPECTED="${1:-}"
ATTEMPTS="${ATTEMPTS:-20}"
WAIT="${WAIT:-3}"

log() { printf '[smoke] %s\n' "$*"; }
fail() { echo "[smoke] FAILED: $*" >&2; exit 1; }

# --- 1. Stabilisation wait ----------------------------------------------------
# The service may take a while to accept connections. Retrying is NOT papering
# over a problem: it is acknowledging that startup is not instantaneous. What
# WOULD paper over the problem is retrying WITHOUT a limit or ignoring the final result.
log "Waiting for $BASE to respond (max $((ATTEMPTS * WAIT))s)..."
for i in $(seq 1 "$ATTEMPTS"); do
  if curl -fsS --max-time 5 "$BASE/health" >/dev/null 2>&1; then
    log "It responded after roughly $((i * WAIT))s."
    break
  fi
  [[ "$i" -eq "$ATTEMPTS" ]] && fail "it did not respond to /health within $((ATTEMPTS * WAIT))s"
  sleep "$WAIT"
done

# --- 2. /health returns status ok ---------------------------------------------
HEALTH="$(curl -fsS --max-time 5 "$BASE/health")"
log "/health -> $HEALTH"
echo "$HEALTH" | grep -q '"status":"ok"' || fail "/health does not return status ok"

# --- 3. The deployed version is the expected one ------------------------------
# This check is the one that catches the quietest failure of them all:
# the deployment "worked" but traffic is still going to the old version.
if [[ -n "$EXPECTED" ]]; then
  VERSION="$(echo "$HEALTH" | sed -n 's/.*"version":"\([^"]*\)".*/\1/p')"
  if [[ "$VERSION" != *"$EXPECTED"* ]]; then
    fail "deployed version '$VERSION' != expected '$EXPECTED'"
  fi
  log "Version verified: $VERSION"
fi

# --- 4. The main functionality responds ---------------------------------------
RESPONSE="$(curl -fsS --max-time 10 "$BASE/api/slots?date=2026-03-02&duration=60")"
echo "$RESPONSE" | grep -q '"slots"' || fail "/api/slots does not return slots: $RESPONSE"
TOTAL="$(echo "$RESPONSE" | sed -n 's/.*"total":\([0-9]*\).*/\1/p')"
[[ "${TOTAL:-0}" -gt 0 ]] || fail "/api/slots returns 0 slots on a day that should have some"
log "/api/slots -> $TOTAL slots"

# --- 5. Errors are still errors -----------------------------------------------
# An API that answers 200 to everything also "passes" a naive smoke test.
CODE="$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$BASE/api/slots")"
[[ "$CODE" == "400" ]] || fail "a request with no date should give 400, it gave $CODE"
log "Error validation OK (400 with no date)"

# --- 6. Reasonable latency ----------------------------------------------------
MS="$(curl -s -o /dev/null -w '%{time_total}' --max-time 10 "$BASE/api/slots?date=2026-03-02" \
      | awk '{printf "%d", $1 * 1000}')"
log "/api/slots latency: ${MS} ms"
[[ "$MS" -lt 2000 ]] || fail "latency of ${MS} ms, above the 2000 ms threshold"

log "ALL CHECKS OK"
chmod +x scripts/smoke.sh

Step 5 is what separates a useful smoke test from a decorative one: it checks that something which should fail does fail. A misconfigured server returning 200 with an error page for any route would sail through steps 1 to 4 without breaking a sweat.

  1. The complete cd.yml with promotion by digest

# .github/workflows/cd.yml
name: CD

on:
  workflow_run:
    workflows: ['CI']
    types: [completed]
    branches: [main]
  workflow_dispatch:
    inputs:
      digest:
        description: 'Digest to deploy (sha256:...). Empty = latest published on main'
        required: false
        type: string

# NEVER cancel a deployment halfway through: it can leave the system inconsistent.
# Deployments queue, they do not cancel. A key difference from the CI `concurrency`.
concurrency:
  group: cd-mini-reservalia
  cancel-in-progress: false

permissions:
  contents: read

env:
  REGISTRY: ghcr.io
  IMAGE_BASE: ghcr.io/${{ github.repository }}

jobs:
  # ---------------------------------------------------------------
  # 1. Resolve WHAT is going to be deployed. Once, for everyone.
  # ---------------------------------------------------------------
  prepare:
    name: Resolve artifact
    runs-on: ubuntu-latest
    if: >-
      github.event_name == 'workflow_dispatch' ||
      github.event.workflow_run.conclusion == 'success'
    permissions:
      contents: read
      packages: read
    outputs:
      digest: ${{ steps.resolve.outputs.digest }}
      image: ${{ steps.resolve.outputs.image }}
      commit: ${{ steps.resolve.outputs.commit }}
    steps:
      - name: Log in to the registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Resolve the digest to deploy
        id: resolve
        run: |
          set -Eeuo pipefail
          INPUT="${{ inputs.digest }}"
          if [[ -n "$INPUT" ]]; then
            DIGEST="$INPUT"
            echo "Digest given by hand: $DIGEST"
          else
            # We resolve the moving :main tag to its digest ONCE only.
            # From here on, the whole pipeline uses the fixed digest.
            DIGEST=$(docker buildx imagetools inspect "${{ env.IMAGE_BASE }}:main" \
                       --format '{{.Manifest.Digest}}')
            echo "Digest resolved from :main -> $DIGEST"
          fi

          IMAGE="${{ env.IMAGE_BASE }}@${DIGEST}"
          COMMIT=$(docker buildx imagetools inspect "$IMAGE" --format \
                    '{{json .Image}}' | grep -o '"org.opencontainers.image.revision":"[^"]*"' \
                    | cut -d'"' -f4 || echo "${{ github.sha }}")

          {
            echo "digest=$DIGEST"
            echo "image=$IMAGE"
            echo "commit=$COMMIT"
          } >> "$GITHUB_OUTPUT"

          {
            echo "## Artifact to deploy"
            echo ""
            echo "| Field | Value |"
            echo "|---|---|"
            echo "| Image | \`$IMAGE\` |"
            echo "| Commit | \`$COMMIT\` |"
            echo "| Source | ${{ github.event_name }} |"
          } >> "$GITHUB_STEP_SUMMARY"

  # ---------------------------------------------------------------
  # 2. Staging: automatic, no approval.
  # ---------------------------------------------------------------
  staging:
    name: Deploy to staging
    runs-on: ubuntu-latest
    needs: [prepare]
    timeout-minutes: 15
    environment:
      name: staging
      url: http://localhost:3001
    permissions:
      contents: read
      packages: read
    outputs:
      validated_digest: ${{ steps.mark.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Deploy
        id: deploy
        run: ./scripts/deploy.sh
        env:
          IMAGE: ${{ needs.prepare.outputs.image }}
          ENVIRONMENT: staging
          PORT: ${{ vars.APP_PORT || '3001' }}
          APP_VERSION: ${{ needs.prepare.outputs.commit }}

      - name: Smoke test
        run: ./scripts/smoke.sh "${{ needs.prepare.outputs.commit }}"
        env:
          BASE: http://localhost:${{ vars.APP_PORT || '3001' }}

      - name: Mark the digest as validated
        id: mark
        run: |
          echo "digest=${{ needs.prepare.outputs.digest }}" >> "$GITHUB_OUTPUT"
          echo "### ✅ Staging validated: \`${{ needs.prepare.outputs.digest }}\`" >> "$GITHUB_STEP_SUMMARY"

      - name: Diagnostics if something fails
        if: failure()
        run: |
          echo "::group::Containers"
          docker ps -a
          echo "::endgroup::"
          echo "::group::Logs"
          docker logs --tail 100 mini-reservalia-staging || true
          echo "::endgroup::"

  # ---------------------------------------------------------------
  # 3. Production: SAME digest, with human approval.
  # ---------------------------------------------------------------
  production:
    name: Deploy to production
    runs-on: ubuntu-latest
    needs: [prepare, staging]
    timeout-minutes: 30
    environment:
      name: production          # <- this is where it stops, waiting for approval
      url: http://localhost:3002
    permissions:
      contents: read
      packages: read
    steps:
      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # THE KEY CHECK OF THE PROMOTION.
      # Production deploys EXACTLY what staging validated. If for any
      # reason the digest does not match, we abort: better not to deploy
      # than to deploy something other than what was validated.
      - name: Verify it is the SAME artifact validated in staging
        run: |
          set -Eeuo pipefail
          VALIDATED="${{ needs.staging.outputs.validated_digest }}"
          TO_DEPLOY="${{ needs.prepare.outputs.digest }}"
          echo "Validated in staging: $VALIDATED"
          echo "To deploy in prod:    $TO_DEPLOY"
          if [[ "$VALIDATED" != "$TO_DEPLOY" ]]; then
            echo "::error::The production digest does NOT match the one validated in staging."
            exit 1
          fi
          echo "✅ Same artifact. Nothing is rebuilt." >> "$GITHUB_STEP_SUMMARY"

      - name: Deploy
        id: deploy
        run: ./scripts/deploy.sh
        env:
          IMAGE: ${{ needs.prepare.outputs.image }}
          ENVIRONMENT: production
          PORT: ${{ vars.APP_PORT || '3002' }}
          APP_VERSION: ${{ needs.prepare.outputs.commit }}

      - name: Production smoke test
        run: ./scripts/smoke.sh "${{ needs.prepare.outputs.commit }}"
        env:
          BASE: http://localhost:${{ vars.APP_PORT || '3002' }}

      - name: Record the deployment
        run: |
          {
            echo "## 🚀 Deployed to production"
            echo ""
            echo "| Field | Value |"
            echo "|---|---|"
            echo "| Digest | \`${{ needs.prepare.outputs.digest }}\` |"
            echo "| Commit | \`${{ needs.prepare.outputs.commit }}\` |"
            echo "| Approved by | @${{ github.actor }} |"
            echo "| Time (UTC) | $(date -u +%Y-%m-%dT%H:%M:%SZ) |"
            echo ""
            echo "**Previous digest (for rollback):** \`${{ steps.deploy.outputs.previous_digest || 'none' }}\`"
          } >> "$GITHUB_STEP_SUMMARY"

      - name: Diagnostics if something fails
        if: failure()
        run: |
          docker ps -a
          docker logs --tail 100 mini-reservalia-production || true
          echo "::error::Production deployment failed. Launch the Rollback workflow with the previous digest."

Open the PR, merge it and watch the whole sequence:

  1. ci.yml runs and publishes the image.
  2. A few seconds later cd.yml starts on its own (notice the run says "triggered by CI").
  3. Resolve artifact prints the digest.
  4. Deploy to staging runs and the smoke test passes.
  5. The run stops. The Deploy to production job appears with a yellow notice: "Deployment protection rules — Review required" and a Review deployments button.
  6. Press the button, tick production, write a comment and press Approve and deploy.
  7. The job starts, verifies the digest, deploys and publishes the summary.

That moment when the pipeline waits for you is the gate from 03-01 made real. It is worth looking at it for a second before you approve.

  1. The frontend on GitHub Pages

Mini-Reservalia also has a web page. Create web/index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Mini-Reservalia</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 40rem; margin: 2rem auto; padding: 0 1rem; }
    .slot { display: inline-block; padding: .4rem .8rem; margin: .2rem; border: 1px solid #888; border-radius: .4rem; }
    #meta { color: #666; font-size: .85rem; margin-top: 2rem; }
  </style>
</head>
<body>
  <h1>Mini-Reservalia</h1>
  <label>Date <input type="date" id="date" value="2026-03-02"></label>
  <label>Duration <input type="number" id="duration" value="60" min="15" step="15"></label>
  <button id="search">Find slots</button>
  <div id="result"></div>
  <div id="meta"></div>
  <script src="app.js"></script>
</body>
</html>

web/app.js — with the configuration read at runtime, not baked into the build (05-01):

// web/app.js
// The configuration is read from config.json AT RUNTIME.
// That way the SAME static artifact serves staging and production:
// the only thing that changes is the config.json the deployment writes.
// If the API URL were baked into the build, every environment would need
// its own build and "build once, deploy many" falls apart.
let CONFIG = { apiBase: 'http://localhost:3001', environment: 'unknown', version: 'dev' };

async function loadConfig() {
  try {
    CONFIG = { ...CONFIG, ...(await (await fetch('config.json', { cache: 'no-store' })).json()) };
  } catch {
    console.warn('config.json not available; falling back to the default configuration');
  }
  document.getElementById('meta').textContent =
    `Environment: ${CONFIG.environment} · Version: ${CONFIG.version} · API: ${CONFIG.apiBase}`;
}

async function search() {
  const date = document.getElementById('date').value;
  const duration = document.getElementById('duration').value;
  const output = document.getElementById('result');
  output.textContent = 'Searching...';
  try {
    const response = await fetch(`${CONFIG.apiBase}/api/slots?date=${date}&duration=${duration}`);
    const data = await response.json();
    if (!response.ok) throw new Error(data.error ?? 'unknown error');
    output.innerHTML = data.total === 0
      ? '<p>No slots left that day.</p>'
      : `<p>${data.total} slots:</p>` +
        data.slots.map((s) => `<span class="slot">${s.start}–${s.end}</span>`).join('');
  } catch (error) {
    output.textContent = `Error: ${error.message}`;
  }
}

document.getElementById('search').addEventListener('click', search);
loadConfig();

The Pages deployment job, in cd.yml:

  web:
    name: Deploy web (Pages)
    runs-on: ubuntu-latest
    needs: [prepare, staging]
    permissions:
      contents: read
      pages: write            # publish to Pages
      id-token: write         # OIDC towards the Pages service
    environment:
      name: github-pages
      url: ${{ steps.publish.outputs.page_url }}
    steps:
      - uses: actions/checkout@v4

      - name: Generate the environment's config.json
        run: |
          cat > web/config.json <<JSON
          {
            "apiBase": "${{ vars.API_BASE || 'http://localhost:3002' }}",
            "environment": "production",
            "version": "${{ needs.prepare.outputs.commit }}",
            "digest": "${{ needs.prepare.outputs.digest }}"
          }
          JSON
          cat web/config.json

      - uses: actions/configure-pages@v5
      - uses: actions/upload-pages-artifact@v3
        with:
          path: web/
      - id: publish
        uses: actions/deploy-pages@v4

Enable Pages in Settings → Pages → Source: GitHub Actions. What you should see: after the deployment, https://YOUR_USERNAME.github.io/mini-reservalia/ with the form and, at the bottom, the line Environment: production · Version: 8f3c1e2 · API: ....

One honest detail about the lab: the public web page will not be able to call your API on localhost (and if it did, the browser would block the request because of CORS). That is fine: what this job illustrates is the atomic deployment of a static artifact with runtime configuration, which is the transferable part. In Reservalia this very pattern uploads Vite's dist/ to S3 and issues a CloudFront invalidation, with the same config.json generated at deployment time (05-01).

  1. The rollback.yml and the stopwatch

Going back must not require any thinking. A rollback that demands you remember commands is a rollback that will not be used at three in the morning.

# .github/workflows/rollback.yml
name: Rollback

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to revert'
        required: true
        default: 'production'
        type: choice
        options: [staging, production]
      digest:
        description: 'Digest to go back to (sha256:...)'
        required: true
        type: string
      reason:
        description: 'Reason (recorded in the summary)'
        required: true
        type: string

concurrency:
  group: cd-mini-reservalia    # SAME group as cd.yml: a rollback and a
  cancel-in-progress: false    # deployment must never overlap

permissions:
  contents: read

jobs:
  rollback:
    name: Revert ${{ inputs.environment }}
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
      packages: read
    # Careful: for a rollback to be FAST, the rollback environment must NOT
    # have a required reviewer, or you will be waiting for an approval again
    # in the middle of an incident. We use a separate environment, with no gate.
    environment:
      name: ${{ inputs.environment }}-rollback
    steps:
      - name: Stopwatch - start
        id: start
        run: echo "t=$(date +%s)" >> "$GITHUB_OUTPUT"

      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Validate the digest received
        run: |
          set -Eeuo pipefail
          DIGEST="${{ inputs.digest }}"
          [[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] || {
            echo "::error::'$DIGEST' is not a valid digest (sha256: + 64 hex)"; exit 2; }
          # Check it exists BEFORE touching production.
          docker buildx imagetools inspect "ghcr.io/${{ github.repository }}@${DIGEST}" >/dev/null
          echo "Digest verified and available in the registry."

      - name: Run the rollback
        run: ./scripts/deploy.sh
        env:
          IMAGE: ghcr.io/${{ github.repository }}@${{ inputs.digest }}
          ENVIRONMENT: ${{ inputs.environment }}
          PORT: ${{ inputs.environment == 'production' && '3002' || '3001' }}

      - name: Verify with the smoke test
        run: ./scripts/smoke.sh
        env:
          BASE: http://localhost:${{ inputs.environment == 'production' && '3002' || '3001' }}

      - name: Stopwatch - stop and record
        if: always()
        run: |
          ELAPSED=$(( $(date +%s) - ${{ steps.start.outputs.t }} ))
          {
            echo "## ⏪ Rollback of ${{ inputs.environment }}"
            echo ""
            echo "| Field | Value |"
            echo "|---|---|"
            echo "| Restored digest | \`${{ inputs.digest }}\` |"
            echo "| Reason | ${{ inputs.reason }} |"
            echo "| Run by | @${{ github.actor }} |"
            echo "| **Duration** | **${ELAPSED} s** |"
            echo "| Result | ${{ job.status }} |"
            echo ""
            echo "> Time to restore service (DORA metric #4)."
          } >> "$GITHUB_STEP_SUMMARY"
          echo "Rollback completed in ${ELAPSED} s"

Run it:

# Find the previous digest (the second most recent published)
gh api "/user/packages/container/mini-reservalia/versions" \
  --jq '.[1] | "\(.name)  \(.created_at)"'

gh workflow run rollback.yml \
  -f environment=production \
  -f digest=sha256:PREVIOUS... \
  -f reason="Timed test of the rollback procedure"

gh run watch

What you should see: the summary with the duration. In this lab the rollback takes 60-110 seconds, most of it in the docker pull. Reservalia takes 4 minutes because ECS does a rolling update with connection draining. Both are well below the 10-minute target, and both are measured, which is what matters: a rollback procedure nobody has ever timed is an assumption.

  1. Causing a bad deployment

Now the fun part: checking that the gate closes.

git checkout -b break-health

Edit src/server.js and sabotage /health:

       if (req.method === 'GET' && url.pathname === '/health') {
+        // DELIBERATE FAILURE: simulates a critical dependency being down
+        if (process.env.ENVIRONMENT) {
+          return respondJson(res, 503, { status: 'degraded', error: 'database unavailable' });
+        }
         return respondJson(res, 200, {

The process.env.ENVIRONMENT condition means the local tests and CI stay green (they do not set ENVIRONMENT) while the deployed container fails. It is a fairly faithful simulation of the most dangerous category of bug: the one that only shows up with a real environment's configuration.

npm test        # green: the tests do NOT catch this
git commit -am "fix: extra check in /health"
git push -u origin break-health
gh pr create --fill && gh pr merge --squash --delete-branch --auto

What you should see, in order:

  1. CI green. All 38 tests pass. The image is published. That is the point: CI does not catch it.
  2. cd.yml starts. Resolve artifact OK.
  3. Deploy to staging fails. And it fails in two places, which is interesting:
    • First, inside deploy.sh, at step 7: the container's HEALTHCHECK never turns healthy. 
      [10:42:31] Waiting for the container to become healthy...
[10:44:31] ERROR: it did not become healthy within 120 s (state: unhealthy)
    • If the health check were more lenient, smoke.sh would catch it: 
      [smoke] FAILED: /health does not return status ok
  4. The diagnostics step dumps the container logs, thanks to if: failure().
  5. Deploy to production does not run. It does not even ask for approval: its needs: [staging] was not satisfied. Production never sees this code.

That is exactly the promise of well-built continuous deployment: a bug CI cannot catch is stopped in the first real environment, automatically, without anyone having to look at anything.

Now pretend it had reached production. Force the bad version into production and run the rollback with a stopwatch:

# 1. Manually deploy the bad version to production
gh workflow run cd.yml -f digest=sha256:BAD...
# approve it on the web when it asks; the production smoke test will fail

# 2. Rollback, with the stopwatch running
date +%s
gh workflow run rollback.yml \
  -f environment=production \
  -f digest=sha256:GOOD... \
  -f reason="Incident: /health returns 503 after deploying sha256:BAD"
gh run watch
date +%s

Write down all three numbers: detection time (how long the smoke test took to fail), decision time (how long it took you to find the good digest) and execution time (what the rollback summary says). In a real incident the second is usually the largest of the three, and it is the one you cut down by recording the previous digest in every deployment's summary, which is exactly what our cd.yml does.

Undo the sabotage:

git checkout main && git pull
git checkout -b fix-health
git revert --no-edit <sha-of-the-bad-commit>
git push -u origin fix-health
gh pr create --fill && gh pr merge --squash --delete-branch --auto

  1. Final verification

# Check How Expected
1 The image does not run as root docker run --rm IMAGE whoami node
2 The HEALTHCHECK works docker inspect --format '{{.State.Health.Status}}' healthy
3 The image is on ghcr.io Packages tab mini-reservalia with versions
4 The deployment is idempotent Run deploy.sh twice The 2nd says "Nothing to do"
5 The script rejects a tag IMAGE=ghcr.io/x/y:main ./scripts/deploy.sh Exit 2, message about the digest
6 CD fires on its own after CI Merge to main CD run "triggered by CI"
7 Production waits for approval Look at the run Review required + button
8 The same digest is promoted Log of the verification step Both digests identical
9 Nothing is rebuilt for production Job log There is no docker build at all
10 The smoke test catches the failure Sabotage /health Staging red, production never runs
11 The rollback is timed Run summary Duration in seconds, < 10 min
12 The previous digest is recorded Summary of every deployment A "Previous digest" line

Common Mistakes and Tips

Symptom: denied: installation not allowed to Create organization package when pushing to ghcr.io. Cause: the job is missing permissions: packages: write, or the repository restricts the GITHUB_TOKEN default permission to read-only. Fix: add the permissions block to the job (having it at workflow level is not enough if the job overrides it) and check Settings → Actions → General → Workflow permissions.

Symptom: Error response from daemon: unauthorized when running docker pull on your own image from elsewhere. Cause: the package is private by default, even if the repository is public. Fix: Packages → mini-reservalia → Package settings → Change visibility → Public, or Manage Actions access to grant the repository access.

Symptom: cd.yml never fires, even though CI finishes green. Possible causes, in order of frequency: (1) the file is not on main yet —workflow_run only fires with the default branch's version—; (2) workflows: ['CI'] does not match the other workflow's name: (it is case-sensitive); (3) CI finished with conclusion: failure and the if guard correctly blocked it. Fix: merge first, check the exact name: and look at the conclusion in the API: gh run list --workflow=ci.yml --json conclusion,name.

Symptom: deploy.sh fails with unbound variable on the GITHUB_OUTPUT line. Cause: set -u with an undefined variable when running the script outside Actions. Fix: it is already handled with ${GITHUB_OUTPUT:-}. If you write your own scripts, default-value expansion is mandatory for anything coming from the environment.

Symptom: the container starts and dies immediately, with no useful logs. Usual cause: volume permissions. The node user (uid 1000) cannot write to /data if the volume was created owned by root. Fix: the Dockerfile's RUN mkdir -p /data && chown -R node:node /data sorts it out for new volumes. If the volume already existed with different permissions: docker volume rm mini-reservalia-data-staging and deploy again. Diagnosis: docker logs mini-reservalia-staging (which is why the if: failure() step dumps them).

Symptom: the smoke test passes but it is testing the old version. Cause: the new container did not start and the reverse proxy is still routing to the old one, or the port is answering something else. Fix: this is precisely why smoke.sh compares the version returned by /health with the expected one. Without that check, a deployment that did nothing passes the smoke test perfectly. It is the quietest failure of them all.

Symptom: the host disk fills up after a few weeks. Cause: every deployment leaves behind an unused ~230 MB image. Fix: the docker image prune in step 8. On a real host, add a weekly docker system prune -af --filter "until=720h" task and a disk space alert. Lesson 07-04 will set that alert up.

Tip — never latest. docker pull image:latest is the most efficient way of not knowing what you are running. Our script rejects it explicitly. Having your tooling prevent the mistake is far better than documenting that it must not be made.

Tip — the CD concurrency is different from the CI one. In CI, cancel-in-progress: true (nobody wants the result of a superseded commit). In CD, false (cancelling a deployment halfway leaves the system in a state nobody designed). And rollback.yml shares a group with cd.yml so they cannot tread on each other.

Exercises

Exercise 1: rollback without hunting for the digest by hand

In an incident, finding the previous digest is the slow step. Make rollback.yml accept an empty digest input and, in that case, automatically resolve the version immediately preceding the deployed one.

Exercise 2: percentage-based canary deployment

Implement a canary deployment: bring the new version up alongside the old one and send only 10 % of traffic to the new one for 2 minutes; if the extended smoke test passes, promote to 100 %; if not, withdraw the canary. Use an Nginx container as the balancer.

Exercise 3: block deployments outside working hours

Add a rule that prevents deploying to production on Friday afternoons and at weekends, with an explicit escape hatch for emergencies that gets recorded.

Solutions

Solution 1.

      - name: Resolve the previous digest if none was given
        id: resolve
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          set -Eeuo pipefail
          DIGEST="${{ inputs.digest }}"

          if [[ -z "$DIGEST" ]]; then
            echo "No digest given: looking for the version before the deployed one."

            # 1. What is running RIGHT NOW (the label deploy.sh set)
            CURRENT=$(docker inspect "mini-reservalia-${{ inputs.environment }}" \
                       --format '{{index .Config.Labels "mini.digest"}}' 2>/dev/null \
                     | sed 's/.*@//' || echo "")
            echo "Currently deployed: ${CURRENT:-(unknown)}"

            # 2. The package versions, most recent first
            mapfile -t VERSIONS < <(
              gh api "/user/packages/container/mini-reservalia/versions" \
                --jq '.[] | select(.metadata.container.tags | length > 0 or true) | .name' \
              | head -20
            )

            # 3. The first one that is NOT the current one
            for v in "${VERSIONS[@]}"; do
              if [[ "$v" != "$CURRENT" ]]; then DIGEST="$v"; break; fi
            done

            [[ -n "$DIGEST" ]] || { echo "::error::No previous version found"; exit 1; }
            echo "Previous version resolved: $DIGEST"
          fi

          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

And use ${{ steps.resolve.outputs.digest }} in the following steps. Change the input to required: false as well.

A more robust version does not query the registry but a deployment history that cd.yml itself maintains: a deployments.jsonl file on a state branch, or GitHub's Deployments API (gh api "repos/{owner}/{repo}/deployments?environment=production"), which already records every deployment with its ref. The difference matters: the registry tells you which images exist; the history tells you what was deployed and in what order, which is the real question.

Solution 2.

nginx-canary.conf:

upstream mini_reservalia {
    # The weight splits the requests: 9 out of every 10 to the stable one.
    server host.docker.internal:3002 weight=9;   # stable
    server host.docker.internal:3003 weight=1;   # canary (10 %)
}

server {
    listen 8080;
    location / {
        proxy_pass http://mini_reservalia;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        # If the canary fails, nginx takes it out of the pool and retries on the
        # stable one: the user never sees the error. A poor man's circuit breaker.
        proxy_next_upstream error timeout http_502 http_503;
    }
}
  canary:
    name: Canary deployment
    runs-on: ubuntu-latest
    needs: [prepare, staging]
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} }

      - name: 1. Bring the canary up alongside the stable one
        run: ./scripts/deploy.sh
        env:
          IMAGE: ${{ needs.prepare.outputs.image }}
          ENVIRONMENT: canary
          PORT: '3003'

      - name: 2. Balancer at 10 %
        run: |
          docker rm -f balancer 2>/dev/null || true
          docker run -d --name balancer -p 8080:8080 \
            --add-host host.docker.internal:host-gateway \
            -v "$PWD/nginx-canary.conf:/etc/nginx/conf.d/default.conf:ro" \
            nginx:alpine
          sleep 3

      - name: 3. Observe for 2 minutes and measure the error rate
        id: observe
        run: |
          set -Eeuo pipefail
          END=$(( $(date +%s) + 120 ))
          TOTAL=0; ERRORS=0
          while [ "$(date +%s)" -lt "$END" ]; do
            CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
                      "http://localhost:8080/api/slots?date=2026-03-02" || echo 000)
            TOTAL=$((TOTAL+1))
            [[ "$CODE" == "200" ]] || ERRORS=$((ERRORS+1))
            sleep 1
          done
          RATE=$(awk "BEGIN {printf \"%.2f\", $ERRORS*100/$TOTAL}")
          echo "Requests: $TOTAL · Errors: $ERRORS · Rate: ${RATE}%"
          echo "rate=$RATE" >> "$GITHUB_OUTPUT"
          echo "### Canary: ${RATE}% errors over $TOTAL requests" >> "$GITHUB_STEP_SUMMARY"
          # 1 % threshold: above that, no promotion.
          awk "BEGIN {exit !($RATE > 1.0)}" && { echo "::error::Error rate above 1 %"; exit 1; }

      - name: 4a. Promote to 100 %
        if: success()
        run: |
          ./scripts/deploy.sh
          docker rm -f mini-reservalia-canary balancer || true
        env:
          IMAGE: ${{ needs.prepare.outputs.image }}
          ENVIRONMENT: production
          PORT: '3002'

      - name: 4b. Withdraw the canary
        if: failure()
        run: |
          echo "::warning::Canary withdrawn. Production stays on the previous version."
          docker logs --tail 100 mini-reservalia-canary || true
          docker rm -f mini-reservalia-canary balancer || true

What this exercise teaches, and what 03-04 explained in theory: a canary is not "deploying slowly", it is "deploying and measuring". Without step 3 —a metric, a threshold and an automatic decision— what you have is a slow deployment, not a canary. And notice that a failure in step 3 leaves production untouched: 90 % of the traffic never saw the new version.

Solution 3.

      - name: Deployment window
        if: inputs.emergency != true
        run: |
          set -Eeuo pipefail
          # The runner runs in UTC; we convert to the team's local time.
          export TZ='Europe/Madrid'
          DAY=$(date +%u)      # 1=Monday ... 7=Sunday
          HOUR=$(date +%H)
          NOW=$(date '+%A %H:%M %Z')

          block() {
            {
              echo "## ⛔ Deployment blocked by the window"
              echo ""
              echo "**Moment:** $NOW"
              echo "**Reason:** $1"
              echo ""
              echo "The permitted window is **Monday to Thursday 09:00 to 17:00** and"
              echo "**Friday 09:00 to 13:00**."
              echo ""
              echo "For a genuine emergency, relaunch the workflow with"
              echo "\`emergency: true\` and a reason. It will be recorded."
            } >> "$GITHUB_STEP_SUMMARY"
            echo "::error::Outside the deployment window: $1"
            exit 1
          }

          [ "$DAY" -ge 6 ] && block "weekend"
          [ "$DAY" -eq 5 ] && [ "$HOUR" -ge 13 ] && block "Friday afternoon"
          { [ "$HOUR" -lt 9 ] || [ "$HOUR" -ge 17 ]; } && block "outside working hours"

          echo "✅ Inside the deployment window ($NOW)." >> "$GITHUB_STEP_SUMMARY"

      - name: Record the use of the escape hatch
        if: inputs.emergency == true
        run: |
          {
            echo "## 🚨 EMERGENCY DEPLOYMENT"
            echo ""
            echo "| Field | Value |"
            echo "|---|---|"
            echo "| Authorised by | @${{ github.actor }} |"
            echo "| Reason | ${{ inputs.emergency_reason }} |"
            echo "| Moment (UTC) | $(date -u) |"
            echo "| Digest | \`${{ needs.prepare.outputs.digest }}\` |"
            echo ""
            echo "> This deployment skipped the window. It must be reviewed in the retrospective."
          } >> "$GITHUB_STEP_SUMMARY"
          echo "::warning::Emergency out-of-window deployment by @${{ github.actor }}"

With the corresponding inputs:

  workflow_dispatch:
    inputs:
      emergency:
        description: 'Skip the deployment window (it will be recorded)'
        type: boolean
        default: false
      emergency_reason:
        description: 'Required if emergency = true'
        type: string

The discussion this exercise opens up is more interesting than the code, and it is worth being clear about. Deployment windows are an anti-pattern in a team with good CD: if deploying on a Friday is frightening, the problem is not the Friday, it is the deployment. With a 90-second rollback and automated smoke tests, a Friday is just another day, and the State of DevOps report data is consistent on this: elite teams deploy whenever they need to.

That said, a window is a reasonable transitional measure while the team builds confidence, and it is also a legitimate business requirement in some contexts (a booking platform may well not want anything touched on a Saturday morning). If you put one in place, give it a review date and a recorded escape hatch like the one above: a restriction with no escape ends up being circumvented in worse ways (deploying by hand over SSH), and a restriction with no review date becomes permanent through sheer inertia.

Optional challenge

Replace the manual production approval with a data-driven automatic gate: let production deploy on its own if the staging smoke test passes and the staging error rate over the last 10 minutes is below 0.5 % and the change does not touch files marked as sensitive (migrations, security configuration); and let it ask for human approval only otherwise. It is the step from "continuous deployment with a gate" to "continuous deployment with a conditional gate", and it is what teams that deploy 50 times a day without anyone approving anything actually do. You will need the metrics from 07-04, so save it for later.

What you have built

  • A multi-stage Dockerfile with a non-root user, a HEALTHCHECK, OCI metadata and a .dockerignore.
  • Publishing to ghcr.io from the pipeline with an ephemeral credential, layer cache and provenance attestation.
  • CI/CD separation with workflow_run and the guard that stops a red CI from being deployed.
  • Two Environments —automatic staging and production with a required reviewer— and the experience of watching the pipeline wait for you.
  • An idempotent deployment script, runnable against three different targets unchanged, which rejects tags, saves the previous digest and waits for healthy.
  • A smoke test with retries that verifies the deployed version and checks that errors are still errors.
  • Promotion by digest with an explicit verification that production deploys exactly what was validated, without rebuilding.
  • A timed rollback.yml that reverts in under two minutes.
  • Proof that a bug CI cannot catch is stopped in staging and never reaches production.

Conclusion

The loop is closed: a commit can reach production without anyone running a command by hand, and it can be rolled back just as fast. You have seen the piece that makes all of this safe, and it fits in one sentence: the same artifact, identified by its content, travels through every environment, and every step checks that it really is the same one. Without that chain, staging validates nothing and the rollback is a lottery.

But there is a question your pipeline still cannot answer, and it is the one that matters most. The smoke test tells you the service responded correctly for the thirty seconds after the deployment. What about twenty minutes later? What about when the real traffic arrives? What if the new version works but is three times slower? And how many times have you deployed this week, how long did each change take from commit to production, and how many of those deployments ended in a rollback? Right now the only honest answer is "I do not know", and a pipeline that does not know whether it improved things is a pipeline you cannot defend to anybody.

In 07-04 you instrument the system. You will add latency measurement and per-status-code counting to src/server.js, expose /metrics in Prometheus format with no dependencies, bring up Prometheus and Grafana with docker compose with a dashboard versioned in the repository, define an SLO with its error budget worked out in explicit arithmetic, write a symptom-based alert and fire it on purpose by slipping a sleep into an endpoint, mark deployments on the dashboard to see the correlation between "we deployed" and "it got worse", calculate the four DORA metrics for your own repository with a scheduled job, and —closing the loop this lesson leaves open— make cd.yml poll the metrics after deploying and trigger the rollback on its own if errors go above the threshold.

◄ Previous
◄
Next ►
►

CI/CD Course: Continuous Integration and Deployment

Module 1: Introduction to CI/CD

Module 2: Continuous Integration (CI)

Module 3: Continuous Deployment (CD)

Module 4: Advanced CI/CD Practices

Module 5: Implementing CI/CD in Real Projects

Module 6: Tools and Technologies

Module 7: Practical Exercises

Module 8: Additional Resources

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved