MercadoFresco's code now lives in a repository with rules, short branches and pull requests that somebody reviews. But reviewing a PR is a person reading a diff, and there are questions nobody can answer by reading: do the 214 tests pass? does the project start on a clean machine, or only on Luis's laptop where there is a psycopg2 installed by hand in 2024? does any of the 87 dependencies have a vulnerability published this week? has a key slipped into a configuration file?

AWS CodeBuild answers all of that automatically. It is a managed, serverless build service: it starts a clean container, clones your code, runs whatever you tell it in a buildspec.yml, saves the results and shuts down, charging per minute of execution and not for existing. This is where "it works on my laptop" stops being an argument, because from now on the only thing that counts is whether it works in an empty container nobody has touched.

Cost warning. CodeBuild charges per minute according to the compute size: in eu-west-1, a BUILD_GENERAL1_SMALL around 0.005 USD/min and a MEDIUM around 0.010, with 100 free minutes a month of the small type. With 20 daily builds of 4 minutes on MEDIUM, MercadoFresco will pay around 24 USD a month. That is not the expensive part: it is a 22-minute build nobody optimises, or building inside the VPC without being aware of the NAT cost, which we will see at the end. Fictitious data.

Contents

  1. What continuous integration is and what problem it solves
  2. Anatomy of a CodeBuild project
  3. The service role and its permissions
  4. MercadoFresco's buildspec.yml, commented
  5. The phases, one by one
  6. Environment variables and secrets without leaks in the logs
  7. Unit tests and test and coverage reports
  8. What to do when a test fails
  9. Integration tests: mocking or the Aurora clone
  10. Static analysis, dependencies and leaked secrets
  11. Quality as a gate, not as a report
  12. Artefacts, cache and parallel builds
  13. CodeBuild inside the VPC, with its cost warning
  14. Debugging a build that fails
  15. Metrics, alarms and real cost
  16. Common mistakes and tips
  17. Exercises
  18. Conclusion

What continuous integration is and what problem it solves

Continuous integration means that every change is integrated into the shared branch and verified automatically, several times a day. The important part is the second half: it is verified automatically. Without verification, "continuous integration" is just pushing code often.

Symptom Real cause What eliminates it
"It works on my laptop" The laptop has accumulated state A clean container on every build
"I forgot to run the tests" Running them is voluntary Automatic execution on every push
"That test has been failing for months" Nobody looks at the result A red build blocks the PR
"It worked before your change" Nobody knows when it broke Build history per commit
"I do not know what version is deployed" A folder is deployed, not an artefact Versioned, immutable artefact

Notice the pattern: in every case the fix is taking away from people the responsibility of remembering. A process that depends on individual discipline fails on the day there is a rush, which is exactly the day it is needed most. The flow we are going to build:

flowchart LR
    A[Push or PR] --> B[Clean container]
    B --> C[install] --> D[pre_build<br/>secrets, lint, scans]
    D --> E[build<br/>tests + packaging] --> F[post_build]
    F --> G{All green?}
    G -->|Yes| H[Artefact in<br/>mercadofresco-artefactos] --> J[Deployment: 08-03]
    G -->|No| I[Red: alertas-mercadofresco] --> K[The PR is not merged]

Anatomy of a CodeBuild project

A project is the definition of how to build something. Its seven pieces:

Piece What it defines MercadoFresco's decision
Source Where the code comes from GitHub via conn-mercadofresco-github
Environment Image, compute size, privileges amazonlinux2-x86_64-standard:5.0, MEDIUM
Service role What the build can do in AWS rol-codebuild-mercadofresco-tienda
Buildspec The commands to run buildspec.yml at the root of the repository
Artefacts What is saved and where ZIP in mercadofresco-artefactos
Cache What is reused between builds Local (custom + docker layer)
Logs Where the output goes /aws/codebuild/build-mercadofresco-tienda

About the environment image. The AWS managed ones come with Python, Node, Java, Go, the CLI and Docker preinstalled, and they update themselves. Your own image in ECR makes sense when installing system dependencies takes more than two or three minutes on every build: you move that time into an occasional image build. MercadoFresco starts with the managed one.

About the compute size. It is the decision with most impact on cost and duration, and intuition lies:

Size vCPU / RAM USD/min Duration at MercadoFresco Cost per build
SMALL 2 / 3 GB 0.005 9 min 40 s 0.048 USD
MEDIUM 4 / 7 GB 0.010 4 min 10 s 0.042 USD
LARGE 8 / 15 GB 0.020 3 min 30 s 0.070 USD

MEDIUM is cheaper than SMALL here, because double the price per minute is offset by more than double the speed: the tests run with pytest -n auto and take advantage of the 4 vCPUs. From MEDIUM to LARGE, by contrast, the time drops little and the cost rises, because the work is no longer parallelisable. You have to measure, not assume: the best size depends on whether your build can use the cores you give it.

About privilegedMode. It is only needed for building Docker images, because the daemon needs privileges. Enable it only when you need it: a privileged container running code from a repository has a larger attack surface.

aws codebuild create-project \
  --name build-mercadofresco-tienda \
  --source '{"type": "GITHUB",
    "location": "https://github.com/mercadofresco/mercadofresco-tienda.git",
    "buildspec": "buildspec.yml", "gitCloneDepth": 1, "reportBuildStatus": true}' \
  --environment '{"type": "LINUX_CONTAINER",
    "image": "aws/codebuild/amazonlinux2-x86_64-standard:5.0",
    "computeType": "BUILD_GENERAL1_MEDIUM", "privilegedMode": false,
    "environmentVariables": [
      {"name": "BUCKET_ARTEFACTOS", "value": "mercadofresco-artefactos", "type": "PLAINTEXT"}]}' \
  --artifacts '{"type": "S3", "location": "mercadofresco-artefactos",
                "packaging": "ZIP", "namespaceType": "BUILD_ID"}' \
  --cache '{"type": "LOCAL", "modes": ["LOCAL_CUSTOM_CACHE", "LOCAL_DOCKER_LAYER_CACHE"]}' \
  --service-role arn:aws:iam::111122223333:role/rol-codebuild-mercadofresco-tienda \
  --timeout-in-minutes 20 --queued-timeout-in-minutes 30 \
  --tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
         Key=Componente,Value=cicd Key=Propietario,Value=luis Key=CentroCoste,Value=plataforma \
  --profile mercadofresco-dev --region eu-west-1

Two parameters deserve attention. gitCloneDepth: 1 clones only the last commit: Luis's 1,847 commits add 40 seconds to every build without contributing anything — unless you need the history for git describe, in which case set 0. And timeout-in-minutes 20 is an emergency brake: a hung build would consume the default 60-minute timeout, and those are billed minutes.

The service role and its permissions

The build acts in AWS with the service role, and here the least privilege of 04-01 matters a lot: this role is used by code that changes on every commit. With PowerUserAccess, anybody who can merge a PR could do anything in the account.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "Registros", "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:eu-west-1:111122223333:log-group:/aws/codebuild/build-mercadofresco-*:*" },
    { "Sid": "ArtefactosSoloEnSuPrefijo", "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::mercadofresco-artefactos/tienda/*" },
    { "Sid": "ConfiguracionNoSensible", "Effect": "Allow",
      "Action": ["ssm:GetParameters", "ssm:GetParameter"],
      "Resource": "arn:aws:ssm:eu-west-1:111122223333:parameter/mercadofresco/construccion/*" },
    { "Sid": "SoloElSecretoDePruebas", "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:mercadofresco/pruebas/*" },
    { "Sid": "InformesDePruebas", "Effect": "Allow",
      "Action": ["codebuild:CreateReportGroup", "codebuild:CreateReport",
                 "codebuild:UpdateReport", "codebuild:BatchPutTestCases",
                 "codebuild:BatchPutCodeCoverages"],
      "Resource": "arn:aws:codebuild:eu-west-1:111122223333:report-group/build-mercadofresco-*" }
  ]
}

What matters is in the Resources, not in the actions. The role cannot read mercadofresco/produccion/rds/mfadmin: only secrets under mercadofresco/pruebas/. That is the difference between a compromised build that spoils a test environment and one that holds the production password. A build project never needs production credentials, and if you think it does, it means you are deploying from the build instead of from CodeDeploy (08-03).

MercadoFresco's buildspec.yml, commented

This file lives at the root of mercadofresco-tienda, versioned alongside the code, and it is the heart of the lesson:

version: 0.2

# Variables. The ones under 'variables' are public and appear in the logs.
# Those under 'parameter-store' and 'secrets-manager' are resolved at run time
# and CodeBuild masks them in the output.
env:
  variables:
    BUCKET_ARTEFACTOS: "mercadofresco-artefactos"
    COBERTURA_MINIMA: "75"
  parameter-store:
    URL_API_PRUEBAS: "/mercadofresco/construccion/url_api_pruebas"
    HOST_AURORA_PRUEBAS: "/mercadofresco/construccion/host_aurora_pruebas"
  secrets-manager:
    CLAVE_PASARELA_PRUEBAS: "mercadofresco/pruebas/pasarela:clave_api"
  exported-variables:
    - VERSION_APP
    - COMMIT_CORTO

phases:

  install:
    runtime-versions:
      python: 3.12
    commands:
      - pip install --upgrade pip
      - pip install -r requirements.txt -r requirements-dev.txt
      - pip install --quiet bandit pip-audit ruff

  pre_build:
    commands:
      - export COMMIT_CORTO=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c1-7)
      - export VERSION_APP=$(cat VERSION)-${COMMIT_CORTO}
      - echo "Building version ${VERSION_APP}"
      # Gates ordered from fastest to slowest
      - ruff check app/ tests/ && ruff format --check app/ tests/
      - git secrets --scan || (echo "SECRET DETECTED IN THE CODE"; exit 1)
      - pip-audit --strict --desc
      - bandit -r app/ -ll -f json -o informe-bandit.json

  build:
    commands:
      # Unit tests in parallel, with a JUnit report and coverage.
      - |
        pytest tests/unitarias -n auto \
          --junitxml=informes/junit-unitarias.xml \
          --cov=app --cov-report=xml:informes/cobertura.xml \
          --cov-fail-under=${COBERTURA_MINIMA}
      # Integration tests against mocked services (moto).
      - pytest tests/integracion --junitxml=informes/junit-integracion.xml
      # Packaging: only what is needed in production.
      - mkdir -p paquete
      - pip install -r requirements.txt --target paquete/ --quiet
      - cp -r app/ scripts/ appspec.yml paquete/
      - echo "${VERSION_APP}" > paquete/VERSION
      - cd paquete && zip -rq ../mercadofresco-tienda-${VERSION_APP}.zip . && cd ..

  post_build:
    # finally runs EVEN IF the previous commands fail.
    commands:
      - |
        if [ "${CODEBUILD_BUILD_SUCCEEDING}" = "1" ]; then
          aws s3 cp mercadofresco-tienda-${VERSION_APP}.zip \
            s3://${BUCKET_ARTEFACTOS}/tienda/${VERSION_APP}/ \
            --metadata commit=${CODEBUILD_RESOLVED_SOURCE_VERSION},version=${VERSION_APP}
          echo "Artefact published: ${VERSION_APP}"
        else
          echo "Build failed: no artefact is published"
        fi
    finally:
      - echo "Total duration: $((SECONDS / 60))m $((SECONDS % 60))s"

# Reports are registered in CodeBuild and seen in the console with their history.
reports:
  pruebas-unitarias:
    files: ["informes/junit-unitarias.xml"]
    file-format: JUNITXML
  cobertura:
    files: ["informes/cobertura.xml"]
    file-format: COBERTURAXML

artifacts:
  files: [mercadofresco-tienda-*.zip]
  discard-paths: yes

cache:
  paths: ["/root/.cache/pip/**/*", ".ruff_cache/**/*"]

The phases, one by one

Phase What it is for If it fails Duration
install Runtimes and tools Aborts; only finally runs 20-90 s
pre_build Credentials, variables, quick checks Aborts before spending on tests 30-60 s
build Compile, test, package Aborts, but post_build does run 2-4 min
post_build Publish, notify, clean up Marks the build as failed 10-30 s

Two behaviours confuse everybody the first time. post_build runs even if build fails, and it is deliberate: you want to publish the test reports precisely when they have failed. That is why the post_build above checks CODEBUILD_BUILD_SUCCEEDING; without that check you would publish to S3 the artefact of a build whose tests failed, the most dangerous mistake in this lesson. And the order inside pre_build is not accidental: ruff takes 2 seconds, the secret scan 5, bandit 15 and pip-audit 20, so a formatting error fails in 2 seconds instead of 40. Always order your gates by increasing cost.

Environment variables and secrets without leaks in the logs

The logs go to CloudWatch Logs and are read by anybody with permission on that log group. Any value you print there is visible, and a set -x or a debugging echo can print a secret without anybody realising it.

Type Where it is stored Masked? What for
PLAINTEXT Project definition No Public configuration
PARAMETER_STORE Parameter Store Yes (if SecureString) Per-environment configuration
SECRETS_MANAGER Secrets Manager Yes Passwords, API keys

The syntax of the secrets-manager block — VARIABLE_NAME: secret-name:json-key:label — is the one people get wrong most often. The colons separate the secret's name from the specific key inside the JSON: without :clave_api, the variable would receive the whole JSON — {"clave_api": "...", "url": "..."} — and your code would fail with a confusing error. And if the secret has the rotation from 04-03, AWSCURRENT guarantees that you are using the current version.

How CodeBuild masks. It replaces with *** the values resolved from Parameter Store and Secrets Manager. But it is string matching, not magic, and it escapes in three cases: if you transform the value (echo $CLAVE | base64), what is printed no longer matches and comes out in the clear; if the secret ends up in a file that you then print with cat; and if a tool includes it in an error message, such as a full connection string in a driver's exception.

Three practical rules: never set -x in a buildspec handling secrets; redirect to /dev/null whatever might carry credentials when you do not need the output; and never put secrets in PLAINTEXT variables, not even "just to test", because they stay in the project definition and in CloudTrail.

Unit tests and test and coverage reports

MercadoFresco's tests live in tests/ and test what matters. A real example — the idempotency of the consumer we built in 07-05, exactly the kind of property nobody verifies by hand:

# tests/unitarias/test_idempotencia.py
import pytest
from app.pedidos import process_order_message

def test_same_message_twice_charges_only_once(idempotency_table, fake_gateway):
    """The module 7 scenario: SQS delivers the same order twice."""
    message = {"id_pedido": "PED-2026-00841", "importe_eur": 48.20,
               "clave_idempotencia": "PED-2026-00841:cobro"}

    first = process_order_message(message)
    second = process_order_message(message)   # the duplicate

    assert first["estado"] == "cobrado"
    assert second["desde_cache"] is True
    assert fake_gateway.number_of_charges == 1           # THE assertion that matters
    assert fake_gateway.total_charged == pytest.approx(48.20)


def test_invalid_slot_rejected_without_touching_aurora(fake_aurora):
    with pytest.raises(ValueError, match="franja_entrega"):
        process_order_message({"id_pedido": "X", "franja_entrega": "noche"})
    assert fake_aurora.writes == 0    # rejected BEFORE writing

Notice what is being asserted. It does not check "that the function returns something": it checks that the gateway was called exactly once and that an invalid piece of data never got written to Aurora. A test that cannot fail when the behaviour breaks is not a test, it is decoration.

The reports block turns the JUnit XML into something CodeBuild understands: in the console you will see, per build, how many tests passed, which ones failed with their trace, and the historical trend. Without it you would have to hunt for the failure in 2,000 lines of log.

About --cov-fail-under=75. Coverage measures which lines the tests executed, not whether they did so well: you can have 95 % without a single useful assertion. Even so it is useful for spotting code that nobody runs, and a threshold that fails the build stops it dropping without anybody noticing. Marta's criterion is sensible: the threshold starts at the current value and can only go up, never lowered to "unblock" a PR. Lowering it once is lowering it forever.

What to do when a test fails

It is the shortest and most important section of the module, because everything else depends on it.

Situation Correct reaction Reaction that ruins the system
It fails because of a badly written test Fix the test in the same PR Mark it @pytest.mark.skip
It fails intermittently Investigate today: usually a real race Retry until it passes
It fails because an external service is down Mock it; a unit test does not go to the network Disable the test
It fails and there is a rush to deploy Revert the commit and deploy the previous one Skip the build

Intermittent tests are the biggest danger, more so than the ones that always fail. One that always fails gets fixed; one that fails one time in fifteen teaches the team to press "retry", and from then on a real failure gets retried too until it passes. They usually point at something true: dependence on execution order, a fixed wait instead of a condition, or a race condition that also exists in production. Treat it as a high-priority failure or delete it, but do not leave it flickering. And the rule that holds everything up: a red build blocks the merge; without it, the team learns within two weeks that red is optional. That gate is closed completely in 08-04.

Integration tests: mocking or the Aurora clone

Unit tests do not touch the network. Integration tests do, and there are two strategies with very different trade-offs:

Strategy Fidelity Speed Cost When
Mocked services (moto, LocalStack) Medium Seconds 0 Most cases
Real container (Postgres in Docker) High for SQL 30-60 s 0 Queries and schema
aurora-mf-pruebas-luis clone Total Minutes ENI + NAT Migrations and performance

With moto, calls to AWS are intercepted in memory:

# tests/integracion/test_flujo_pedido.py
import boto3, json
from moto import mock_aws
from app.pedidos import confirm_order

@mock_aws
def test_confirmed_order_enqueues_and_publishes_event():
    sqs = boto3.client("sqs", region_name="eu-west-1")
    sns = boto3.client("sns", region_name="eu-west-1")
    queue_url = sqs.create_queue(QueueName="cola-mercadofresco-pedidos")["QueueUrl"]
    topic = sns.create_topic(Name="mercadofresco-pedido-confirmado")["TopicArn"]

    result = confirm_order(
        {"id_cliente": "CLI-4471", "lineas": [{"sku": "FRUT-0012", "uds": 3}],
         "franja_entrega": "tarde"}, queue_url=queue_url, topic_arn=topic)

    messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
    body = json.loads(messages["Messages"][0]["Body"])
    assert body["id_pedido"] == result["id_pedido"]
    assert body["franja_entrega"] == "tarde"
    assert body["version_evento"] == "2"     # see 08-05

This verifies the complete flow — confirm, enqueue, publish — in less than a second and at no cost, and for 90 % of cases it is enough. The Aurora clone is reserved for what mocking cannot tell you: whether a migration takes 4 seconds or 40 minutes over 340 real GB, whether a new index is actually used, whether a query degrades at production volume. That requires a VPC, and the VPC has a cost.

Static analysis, dependencies and leaked secrets

Tool What it looks for Blocking Cost
ruff Style errors, unused imports, obvious bugs Yes 0
git-secrets Credential patterns in the code Yes, always 0
pip-audit Known CVEs in dependencies Yes, with nuances 0
bandit Insecure code patterns Report 0
Amazon Inspector Vulnerabilities in ECR images and Lambda Configurable Per resource

git-secrets is the net that was missing in 08-01. The .gitignore protects you from carelessness; this also protects you from git add -f:

- git secrets --register-aws                          # AWS patterns: AKIA, secret keys
- git secrets --add 'sk_live_[0-9a-zA-Z]{24}'         # payment gateway keys
- git secrets --add 'postgres://[^:]+:[^@]+@'         # connection strings with password
- git secrets --add --allowed 'AKIAIOSFODNN7EXAMPLE'  # the documentation one, allowed

Whether it should block is not up for discussion: a false positive costs adding an --allowed exception; a false negative costs rotating credentials in production.

pip-audit with nuances. It fails if any dependency has a known CVE, which is correct but has a failure mode you have to anticipate: a CVE is published in a popular library on a Tuesday afternoon and all your builds go red, including the one for an urgent fix that has nothing to do with it. Avoid the easy way out of --ignore-vuln, which turns into a list that only grows; the right thing is to fail and alert through alertas-mercadofresco with the build identifier, so that there is a procedure and not a permanent exception.

Amazon CodeGuru Reviewer deserves a mention: it analyses code with models trained on Amazon's own code and detects what a linter does not see — resource leaks, race conditions, incorrect use of the AWS APIs. It is paid for and its recommendations are suggestions, not gates: for three people it is a dispensable luxury; for fifty it may pay off.

Quality as a gate, not as a report

Here is the criterion that separates a useful pipeline from quality theatre. A check is either a gate — it fails the build and blocks the merge — or a report nobody is going to read. There is no middle ground: a report that blocks nothing is ignored by the third day.

Check Gate? Why
Unit tests Yes A red test is broken behaviour
Coverage < 75 % Yes Stops it dropping without anybody noticing
Secrets detected Yes The cost of a false negative is rotating in production
ruff Yes Fast and objective; arguing about style in PRs is wasted time
CVEs in dependencies Yes, with an alert With a procedure for Tuesday afternoon
bandit No, report Many false positives; reviewed weekly
Cyclomatic complexity No, report Arbitrary threshold; generates sterile arguments

The rule for deciding: if you are not willing to stop a deployment because of it, it is not a gate. And if it is not, be honest about the fact that almost nobody will read it: give it an explicit moment of review — "on Mondays we look at the bandit report" — or take it out of the buildspec.

Artefacts: the application package and the images

The result of the build is an immutable, versioned artefact, and this is the most important conceptual change compared with Luis's scp: you no longer deploy "whatever is in the folder", you deploy mercadofresco-tienda-1.5.0-a3f9c21.zip, a file that exists, has an identifier, can be downloaded six months later and is exactly what was tested.

The mercadofresco-artefactos bucket is created with three non-negotiable properties: versioning, which CodePipeline requires and which protects against accidental overwriting; encryption with alias/mercadofresco-datos and BucketKeyEnabled to reduce the KMS calls, which are billed separately; and a lifecycle rule that expires the artefacts.

aws s3api put-bucket-versioning --bucket mercadofresco-artefactos \
  --versioning-configuration Status=Enabled --profile mercadofresco-dev

aws s3api put-bucket-lifecycle-configuration --bucket mercadofresco-artefactos \
  --lifecycle-configuration '{"Rules":[{
    "ID":"caducar-artefactos-antiguos","Status":"Enabled",
    "Filter":{"Prefix":"tienda/"},
    "Expiration":{"Days":90},
    "NoncurrentVersionExpiration":{"NoncurrentDays":30}}]}' \
  --profile mercadofresco-dev

Without a lifecycle rule, twenty daily artefacts of 45 MB are 27 GB a year of ZIP that nobody will ever deploy; with 90 days you have plenty of room to go back and the cost stays in cents.

Container images. If instead of a ZIP you build an image, you need privilegedMode: true, a docker login against ECR in pre_build with aws ecr get-login-password, a docker build tagging with ${VERSION_APP} and a docker push in post_build. On this, only the essentials: ECR, layers, image scanning and container deployment are module 10. Here it is enough to know that CodeBuild builds images just as well as ZIPs and that LOCAL_DOCKER_LAYER_CACHE reuses the layers.

Dependency cache and its real effect

Installing 87 packages from PyPI takes 45-70 seconds on every build: over 20 daily builds, that is 20 minutes a day downloading the same thing.

Cache type Where it lives Measured saving Limitation
Local (LOCAL_CUSTOM_CACHE) The build host 40-55 s Only if it lands on the same host
S3 The bucket you specify 25-40 s You pay to upload and download
Docker layers The build host 1-3 min Only with privilegedMode

The real numbers: with no cache, 4 min 10 s; on a hit, 3 min 20 s; on a miss, 4 min 12 s. The saving is real but modest, and it is worth saying so because expectations are exaggerated: the local cache depends on the build landing on a host that has it, and with 20 a day the hit rate is around 60 %.

Two warnings. A poisoned cache is worse than no cache: if you store site-packages instead of the pip cache, you drag old versions along and debugging it takes hours — always cache ~/.cache/pip. And if you suspect the cache, invalidate it: aws codebuild invalidate-project-cache is the first command to try when a build fails inexplicably.

Batch builds

When the suite grows, the batch block of the buildspec defines a graph of builds: several — pruebas_unitarias, pruebas_integracion, escaneos — run at the same time, and an empaquetado stage with depend-on waits for the three; fast-fail: true cancels the rest as soon as one fails. A sequential build of 12 minutes can come down to 6, but the total cost in minutes does not drop — you pay the same, just in parallel —; what you buy is developer waiting time, usually worth far more than 0.04 USD. Start simple: do not parallelise a 4-minute build.

CodeBuild inside the VPC, with its cost warning

By default CodeBuild runs outside your VPC, with internet access and without access to your private subnets: it cannot talk to aurora-mf-pruebas-luis, which is in snet-mercadofresco-datos-a.

aws codebuild update-project --name build-mercadofresco-tienda \
  --vpc-config '{"vpcId": "vpc-mercadofresco",
    "subnets": ["snet-mercadofresco-app-a", "snet-mercadofresco-app-b"],
    "securityGroupIds": ["sg-mercadofresco-construccion"]}' \
  --profile mercadofresco-dev --region eu-west-1

Three things to get right. Private subnets, always: in a public one with no public IP the build will have no internet access and pip install will hang until the timeout, and it is the most common and most baffling failure. Egress through NAT to reach PyPI, or VPC endpoints for the AWS APIs. And a security group sg-mercadofresco-construccion with an inbound rule on sg-mercadofresco-basedatos allowing 5432 from it: never open the database by CIDR.

Important cost warning. Putting CodeBuild in the VPC has two costs that come as a surprise on the bill. The ENIs: every build creates and destroys a network interface, which adds 30-60 seconds of start-up to each execution — with 20 a day, 20 minutes a day billed for doing nothing. And the NAT gateway: 0.048 USD/hour plus 0.048 USD per GB; downloading 300 MB of dependencies 20 times a day is around 9 GB a month in transit alone. The practical recommendation is to have two projects: build-mercadofresco-tienda outside the VPC for 90 % of the builds with mocked services, and build-mercadofresco-integracion inside, only on PRs into main or nightly. And Gateway-type VPC endpoints for S3 and DynamoDB, which are free and keep that traffic off the NAT.

Debugging a build that fails

1. The CloudWatch logs, in /aws/codebuild/build-mercadofresco-tienda. To get straight to the point without reading 2,000 lines, aws codebuild batch-get-builds --ids <id> with --query 'builds[0].phases[?phaseStatus==FAILED].[phaseType,contexts[0].message]' gives you the phase that failed and its message.

2. Local reproduction with the official image. The fastest way to iterate, and it costs nothing:

./codebuild_build.sh -i public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:5.0 \
  -a /tmp/salida -s ~/proyectos/tienda -b buildspec.yml

The script is in the aws/aws-codebuild-docker-images repository and runs the same buildspec on the same image, on your machine: if it fails the same way, you have a debugging loop of seconds instead of minutes. Watch out: secrets are not resolved locally unless you pass credentials, so export test values.

3. The interactive debugging session. When the failure only happens in CodeBuild — and it does — put codebuild-breakpoint as the command just before the one that fails: the build stops there and waits. Launch it with --debug-session-enabled and connect with Session Manager from the console. You find yourself inside the container, at the exact point, with the code cloned and the variables resolved; codebuild-resume continues. It requires SSM permissions on the service role, and the build keeps billing while you wait: do not leave a session open over lunch.

4. The variables CodeBuild gives you, which usually settle the question before anything else: CODEBUILD_RESOLVED_SOURCE_VERSION (the exact SHA built), CODEBUILD_WEBHOOK_TRIGGER (what triggered it: branch/main, pr/42, tag/v1.5.0), CODEBUILD_BUILD_SUCCEEDING (essential in post_build), CODEBUILD_BUILD_ID and CODEBUILD_SRC_DIR.

Metrics, alarms and real cost

CodeBuild publishes metrics in AWS/CodeBuild. The three that matter are FailedBuilds (alarm: ≥ 3 in 1 hour, or a single one if it is main), Duration (p90 > 8 min) and SucceededBuilds for the hit rate.

aws cloudwatch put-metric-alarm --alarm-name mercadofresco-construcciones-fallidas \
  --namespace AWS/CodeBuild --metric-name FailedBuilds --statistic Sum \
  --dimensions Name=ProjectName,Value=build-mercadofresco-tienda \
  --period 3600 --evaluation-periods 1 --threshold 3 \
  --comparison-operator GreaterThanOrEqualToThreshold --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

--treat-missing-data notBreaching is important: with no builds there is no data, and you do not want an alarm going off every night. It is the same lesson as 05-01.

MercadoFresco's real monthly cost:

Item Calculation Cost
PR and branch builds 20/day × 4.2 min × 22 days × 0.010 18.48 USD
Nightly integration (in the VPC) + NAT transit 1/day × 9 min × 30 × 0.010 + 9 GB 3.13 USD
Artefacts in S3 with a 90-day expiry ~4 GB 0.09 USD
CloudWatch logs ~2 GB ingested 1.15 USD
Total ≈ 22.85 USD/month

Twenty-three dollars a month to remove "it works on my laptop" from the team's vocabulary. Compare it with a broken deployment on a Friday at 19:00 with 900 orders/hour at stake.

Cleanup: delete-project, delete-report-group --delete-reports and delete-log-group. Log groups are not deleted when you delete the project and keep billing for storage; give them retention with aws logs put-retention-policy --retention-in-days 30 from day one.

Common Mistakes and Tips

Publishing the artefact without checking CODEBUILD_BUILD_SUCCEEDING. post_build runs even if build fails, so you upload to S3 the package of a build whose tests failed. It is the most dangerous mistake in the lesson.

Putting secrets in PLAINTEXT variables, which stay in the project definition and in CloudTrail; or forgetting :key in the Secrets Manager reference, so the variable receives the whole JSON and the application fails with an error that does not point at the cause. And using set -x with secrets: masking is string matching and as soon as you transform the value it comes out in the clear.

CodeBuild in a public subnet. With no public IP there is no internet access and pip install hangs until the timeout. Private subnets with NAT, always. And do not put every build in the VPC "just in case": you pay 30-60 s of ENI on each one plus the NAT transit.

Lowering the coverage threshold to unblock a PR. It gets lowered once and never goes back up. And retrying an intermittent test until it passes teaches the team that red means nothing, so on the day it is real it will be retried too.

Caching site-packages instead of ~/.cache/pip. You drag old versions along and debugging it takes hours. And not setting retention on log groups: they bill forever, even after deleting the project.

Tip: order the gates from fastest to slowest. A formatting error should fail in 2 seconds, not after 4 minutes of tests.

Tip: make the buildspec runnable locally with codebuild_build.sh and the official image; if it only works inside CodeBuild, every debugging iteration costs minutes. Export the version with exported-variables so that CodePipeline (08-04) uses it in the following stages, and set a tight timeout-in-minutes: it is the brake that stops you paying 60 minutes for a hung build.

Exercises

Exercise 1: the artefact that should not exist

On Friday at 18:40, Luis sees in mercadofresco-artefactos the file tienda/1.5.2-c7a91f3/mercadofresco-tienda-1.5.2-c7a91f3.zip, published at 18:32. But in CodeBuild the c7a91f3 build shows as red: 3 tests in tests/unitarias/test_pagos.py failed.

Answer: (a) how it is possible for the artefact to exist if the build failed; (b) what allowed it and how it is fixed; (c) why it is dangerous beyond this case, bearing in mind what CodePipeline will do in 08-04; (d) what check you would put in place so that an artefact like that cannot be deployed; (e) what alarm would have warned Marta at 18:35.

Exercise 2: the 22-minute build

The build takes 22 minutes: install 4 min (140 packages and psycopg2 compiled from source), pre_build 3 min (pip-audit, bandit, ruff and git-secrets in series), build 14 min (620 unit tests in series, 40 integration ones against the Aurora clone and the packaging), post_build 1 min. The project is in the VPC, on SMALL, with no cache. Luis has started skipping the local build.

Propose a plan that brings the time below 6 minutes. For each measure state the estimated saving, the effect on cost and what is lost. Say explicitly which one you would apply first and why.

Exercise 3: the secret that appeared in the logs

Sara, reviewing the logs of a build, finds this line in /aws/codebuild/build-mercadofresco-tienda:

[pre_build] Connecting to postgresql://mfadmin:V3rd3_Manzana_2026@aurora-mf-pruebas-luis...

The secret is correctly declared in env.secrets-manager and CodeBuild masks it in direct output. In pre_build there are these two lines:

      - export CADENA_CONEXION="postgresql://mfadmin:${PASS_AURORA}@${HOST_AURORA_PRUEBAS}:5432/mf"
      - echo "Connecting to ${CADENA_CONEXION}"

Answer: (a) why the masking did not work; (b) what you do in the first 30 minutes, in order; (c) how you fix the buildspec; (d) the logs have had indefinite retention for 6 months and that line appears in 340 builds: how do you deal with it; (e) what automatic control you would add to detect it next time.

Solutions

Solution 1

(a) Because post_build runs even if build fails. It is deliberate — it lets you publish reports for failed tests — but it means the publishing commands you put there run anyway: the tests failed in build, the phase aborted, and post_build uploaded the ZIP quite happily.

(b) The CODEBUILD_BUILD_SUCCEEDING guard is missing: the guilty buildspec has an unconditional aws s3 cp in post_build. The fix is the one that does appear in the lesson's buildspec:

  post_build:
    commands:
      - |
        if [ "${CODEBUILD_BUILD_SUCCEEDING}" != "1" ]; then
          echo "Build failed: no artefact is published"; exit 1
        fi
      - aws s3 cp mercadofresco-tienda-*.zip s3://${BUCKET_ARTEFACTOS}/tienda/${VERSION_APP}/

(c) Because it breaks the guarantee everything else depends on: that a published artefact is an artefact that passed the tests. In 08-04 the pipeline will take the artefact from one stage and pass it to the next; and if on top of that somebody deploys by file name from S3, code with three broken payment tests would get deployed. And it would be silent: the ZIP exists, it looks fine and its name says nothing about where it came from. Trust in the system is lost entirely with a single case like this.

(d) Two cheap layers. Metadata on the object--metadata estado=exitosa,commit=... — and a check at deployment time that rejects anything without it. And, more robustly, separating responsibilities: have the build publish nothing and let the native mechanism (artifacts:) hand the package to CodePipeline, which only advances if the previous stage finished well. The general rule: if publishing is a side effect of a script, one day it will run when it should not; if it is a pipeline state transition, it cannot happen out of order.

(e) The mercadofresco-construcciones-fallidas alarm on FailedBuilds, with a low threshold — 1 in 5 minutes for main — and alertas-mercadofresco as the destination: the threshold of 3 in 1 hour is useful for spotting a trend, but on main a single failure deserves an immediate warning. In addition, an EventBridge rule on CodeBuild Build State Change with build-status: FAILED warns within seconds with a direct link to the logs.

Solution 2

# Measure Saving Cost What is lost
1 Take the project out of the VPC; a separate nightly build-mercadofresco-integracion ~7 min Down: less NAT and ENI Immediate feedback against real Aurora
2 pytest -n auto with MEDIUM ~7 min Price/min up, total down Nothing, if the tests are isolated
3 pip cache + psycopg2-binary instead of compiling ~3 min None The prebuilt wheel, acceptable
4 Scans in parallel ~2 min Same in total minutes Logs harder to read
5 gitCloneDepth: 1 ~30 s None git describe for the version
6 pytest-split across 3 batch builds ~2 min Same minutes, in parallel Configuration complexity

First, measure 1, and the justification is the key to the exercise: it is the one that saves the most time and the only one that also reduces the cost. The 40 tests against the Aurora clone add value, but not on every push: they add it before merging into main and in the nightly run. With 90 % of the builds outside the VPC, the 30-60 s of ENI, the NAT transit and much of the network slowness disappear. It is also the lowest-risk one: it does not change a single test.

Second, measure 2, counter-intuitive and very profitable: going from SMALL to MEDIUM doubles the price per minute but reduces the total, because 620 tests with -n auto on 4 vCPUs almost divide the suite's time by four. The requirement is that they be isolated: if they share a SQLite or a fixed temporary directory, parallelisation will make them fail intermittently — precisely the problem we were talking about — and then the fix is to isolate them, not to go back to running in series.

Result: install 4 → 1 min, pre_build 3 → 1, build 14 → 3, post_build 1. About 6 minutes, with almost the same cost per build and a smaller NAT bill. And the consequence that is not in the table and matters most: at 6 minutes, Luis stops skipping the build.

Solution 3

(a) Because the masking is a literal substitution of the secret's string in the output, and not a semantic analysis. The buildspec interpolates ${PASS_AURORA} inside a larger string; the result is a new value that CodeBuild knows nothing about, so when the echo runs it finds no match to mask and prints the whole thing in the clear. It is the first of the three cases: transforming the value breaks the masking.

(b) The first 30 minutes, in order. Rotate the test mfadmin password, because you always start by cutting short the validity of what leaked. Work out who could have read it, with CloudTrail looking for GetLogEvents and FilterLogEvents on that group over the last 6 months; the scope depends on how many people have read access to CloudWatch Logs, which is usually more than you think. Check whether it affects production — and here you see the value of the role's prefix separation: CodeBuild cannot read mercadofresco/produccion/rds/mfadmin, so the incident stays confined to testing. And fix the buildspec before the next build, or the log will keep being written.

(c) Never print the full string: if you need tracing, trace only the non-sensitive part (echo "Connecting to ${HOST_AURORA_PRUEBAS}:5432/mf as mfadmin"). The deeper improvement is a different one: if the application reads the secret from Secrets Manager with its own role, the value never passes through a shell environment variable and this whole class of leak disappears. The environment variable is convenient, but it is also the surface through which secrets escape.

(d) Treat the historical logs as compromised data: export the group to S3 if it has to be kept for auditing, delete the affected streams — there is no partial editing in CloudWatch Logs — and set 30-day retention. Since the password has already been rotated in (b), the residual exposure gives access to nothing: deleting them is hygiene and compliance, not containment. If the secret had been a production one, you would also have to weigh up the internal notification we saw in 08-01.

(e) A CloudWatch Logs metric filter over /aws/codebuild/* counting matches of patterns such as postgresql://*:*@, AKIA or sk_live_, with an alarm towards alertas-mercadofresco. It is better than a check inside the buildspec because it works even if nobody remembers to maintain it and it covers every project, present and future. It is the idea from 05-01: if a piece of data should never appear in a log, put a metric that counts it and alarm as soon as it goes above zero.

Conclusion

"It works on my laptop" no longer means anything at MercadoFresco. Every push starts a clean container that knows nothing about the psycopg2 Luis installed by hand in 2024: it installs the declared dependencies, runs 214 tests, checks that coverage does not drop below 75 %, scans the 87 dependencies for CVEs, looks for leaked credentials and produces a versioned, immutable artefact in mercadofresco-artefactos. What gets deployed will be that ZIP, with its commit and its metadata.

You know the anatomy of a project — source, environment, service role, buildspec, artefacts, cache and logs — and you know that the compute size has to be measured and not assumed, because MEDIUM can work out cheaper than SMALL when the build knows how to use the four cores. You know that the service role is used by code that changes on every commit, and that this is why its scope is limited to the mercadofresco/pruebas/ prefix: a build never needs production credentials. You have mastered the buildspec.yml, the order of the phases and the two traps that catch everybody: that post_build runs even if build fails — hence the CODEBUILD_BUILD_SUCCEEDING guard, without which you publish artefacts from broken builds — and that gates are ordered from fastest to slowest.

You know how to inject secrets with the secret:key:label syntax and — more importantly — why the masking breaks as soon as you transform the value or interpolate it into a larger string. You have tests that verify what really matters, such as the gateway being called exactly once in the face of a duplicated message, and the reports that turn a JUnit XML into a queryable history. And you have the discipline that holds all of this up: an intermittent test is investigated or deleted, never retried; the coverage threshold only ever goes up; and a check you are not willing to have stop a deployment is not a gate, it is decoration.

You know when moto is enough — 90 % of cases, in a second and at no cost — and when the aurora-mf-pruebas-luis clone is needed, with the warning that comes with it: CodeBuild in the VPC costs 30-60 seconds of ENI per build plus the NAT transit, and that is why MercadoFresco has two projects, a fast one outside and an integration one inside. And you know how to debug without guessing: the logs, local reproduction with the official image and codebuild-breakpoint to get inside the container at the exact point. All for about 23 dollars a month.

But there is something this lesson has not changed. The artefact is there, verified, waiting in a bucket. And to get it onto the instances of the ASG asg-mercadofresco-tienda, Luis still does exactly what he did on day one: SSH in, copy the ZIP, unzip it, restart the service and look at the site with his fingers crossed. For the forty seconds it takes, the instance serves errors; if there are two and he does them one after the other, half the traffic sees one version and the other half another; and if the ZIP turns out to be broken, going back means finding the previous one and repeating the ritual with the shop down. The artefact is good; the way of putting it into production is still a manual ritual with no net.

In 08-03, "AWS CodeDeploy", that gets solved. We will see the three targets — EC2, Lambda and ECS — and which strategies each one supports; the concepts of application, deployment group, revision and deployment configuration, with the agent installed on the ASG's instances; the appspec.yml and its lifecycle hooks one by one, with MercadoFresco's real scripts to stop the service, warm the cache and check /salud; the in-place strategies with AllAtOnce, HalfAtATime and OneAtATime, the blue/green one with the ALB's target groups, and the canary and linear ones for deploying the mercadofresco-cobrar-pago Lambda; the warning about schema migrations and why they must never go in the same step as the code; and above all the piece that closes the course's fourth problem: automatic rollback when mercadofresco-alb-latencia-alta fires or a hook fails, with nobody watching.

© Copyright 2026. All rights reserved