In mercadofresco-artefactos there is a verified ZIP: it passed 214 tests, carries its commit in the
metadata and has the version in its name. And to get it into production, Luis still does what he has always
done: ssh into the first instance of the ASG, scp the ZIP, unzip, systemctl restart, look at the site,
repeat on the second. Forty seconds per instance serving errors, two versions coexisting while it lasts, and
if something goes wrong, hunt for the previous ZIP and repeat the ritual with the shop down.
AWS CodeDeploy turns that ritual into a governed operation. It knows how to take an instance out of the target group before touching it, run your scripts in a defined order, check that the application responds before giving it traffic back, advance across the fleet a bit at a time and — what really matters — roll back on its own when a CloudWatch alarm fires during the deployment. It is the lesson that closes the fourth problem of the course: risky deployments.
Cost warning. CodeDeploy is free for EC2, Lambda and ECS; it only charges 0.02 USD per on-premises instance update. What costs money is everything around it: in blue/green you temporarily double the ASG instances — around 0.09 USD for a 30-minute deployment with two
t3.medium— and Lambda versions do not expire on their own. The cleanup section comes at the end. Fictitious data.
Contents
- What CodeDeploy solves and what it does not
- The three targets and the strategies each one supports
- Concepts: application, deployment group, revision and configuration
- The agent on the ASG instances
- The
appspec.ymlfor EC2 and the lifecycle hooks - MercadoFresco's real scripts
- In-place deployment:
AllAtOnce,HalfAtATimeandOneAtATime - Blue/green with the ASG and the target groups
- The Lambda and ECS
appspec - Canary and linear: deploying
mercadofresco-cobrar-pago - Automatic rollback: the piece that solves problem 4
- Comparison with manual rollback and with weighted Route 53
- Schema migrations: expand and contract
- Deployment observability and diagnosing a failure
- Cost and cleanup
- Common mistakes and tips
- Exercises
- Conclusion
What CodeDeploy solves and what it does not
CodeDeploy does not build, does not decide when to deploy and does not define infrastructure. It does one thing: it takes a revision — a ZIP in S3 or an image — and installs it on a set of targets following a plan, with hooks where you put your scripts and with stop conditions.
| Problem with manual deployment | What CodeDeploy brings |
|---|---|
| The instance serves errors while it is updated | It takes it out of the target group before touching it |
| Two versions coexisting with no control | It controls how many instances are on each version |
| "Did the service restart properly?" | ValidateService checks it and fails if not |
| Going back means repeating the ritual by hand | Automatic rollback in minutes |
| Nobody knows what was deployed or when | History with revision, author and result |
| The deployment is the same in dev as in prod | Deployment groups with different configuration |
And what it does not bring: it does not coordinate the order between different services — that is the job of the 08-04 pipeline — it does not migrate your database and it does not make up for an application that cannot coexist with itself in two versions. That last limitation is the subject of the schema migration section.
The three targets and the strategies each one supports
| Target | In place | Blue/green | Canary | Linear | Agent |
|---|---|---|---|---|---|
| EC2 / ASG | Yes | Yes | No | No | Yes |
| On-premises servers | Yes | No | No | No | Yes |
| AWS Lambda | No | Yes (by alias) | Yes | Yes | No |
| Amazon ECS | No | Yes | Yes | Yes | No |
Three observations that avoid frequent confusion. On EC2 there is no canary and no linear: percentage
traffic shifting belongs to Lambda and ECS, where traffic is split by alias or target group weight; on EC2
the closest thing is OneAtATime, which is instance granularity, not a percentage of requests. On Lambda
and ECS there is no in-place deployment: the new version is always created alongside the old one and the
traffic is shifted, which is safer by construction. And the agent is only needed on EC2 and on-premises
servers; on Lambda and ECS, CodeDeploy talks to the service API.
Concepts: application, deployment group, revision and configuration
Four objects, and confusing them causes half of the initial doubts:
| Object | What it is | In MercadoFresco |
|---|---|---|
| Application | Logical container: name and platform | app-mercadofresco-tienda, app-mercadofresco-cobrar-pago |
| Deployment group | Where and how: targets, strategy, alarms, rollback | dg-mercadofresco-tienda-desarrollo / -produccion |
| Revision | What is deployed: the ZIP with its appspec.yml |
s3://mercadofresco-artefactos/tienda/1.5.0-a3f9c21/ |
| Deployment configuration | The pace and how much must stay healthy | CodeDeployDefault.HalfAtATime, or your own |
The important one is the deployment group: the same application and the same revision behave very
differently depending on the group. In development, AllAtOnce with no alarms so that it is fast; in
production, blue/green with alarms and automatic rollback. That is the mechanism that separates environments.
aws deploy create-application --application-name app-mercadofresco-tienda \
--compute-platform Server --profile mercadofresco-dev --region eu-west-1
aws deploy create-deployment-group \
--application-name app-mercadofresco-tienda \
--deployment-group-name dg-mercadofresco-tienda-produccion \
--service-role-arn arn:aws:iam::111122223333:role/rol-codedeploy-mercadofresco \
--auto-scaling-groups asg-mercadofresco-tienda \
--deployment-config-name CodeDeployDefault.HalfAtATime \
--load-balancer-info '{"targetGroupInfoList":[{"name":"tg-mercadofresco-tienda"}]}' \
--alarm-configuration '{"enabled": true, "ignorePollAlarmFailure": false,
"alarms": [{"name":"mercadofresco-alb-latencia-alta"},
{"name":"mercadofresco-pedidos-fallidos"}]}' \
--auto-rollback-configuration '{"enabled": true,
"events": ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM"]}' \
--profile mercadofresco-dev --region eu-west-1Notice ignorePollAlarmFailure: false: if CodeDeploy cannot query an alarm, the deployment fails
instead of carrying on blind. It is the right behaviour — if the safety mechanism does not answer, do not
carry on — but it produces baffling failures when the role is missing cloudwatch:DescribeAlarms.
The agent on the ASG instances
The agent is a process that polls CodeDeploy, downloads the revision and runs the hooks. Without it, the
instance stays in Pending until the timeout and the deployment fails with an unhelpful message.
It can be installed baked into the AMI, from the user data of lt-mercadofresco-tienda, or — the
recommended option — as a Systems Manager association with automatic updates:
aws ssm create-association --name AWSCodeDeployAgentUpdate \
--targets Key=tag:Proyecto,Values=mercadofresco \
--schedule-expression "rate(14 days)" \
--profile mercadofresco-dev --region eu-west-1That last one is what MercadoFresco uses because it solves the real problem: an out-of-date agent stops working without warning, and updating it by hand on instances the ASG creates and destroys is impossible.
Two requirements that get forgotten. The instance role needs to be able to read the artefact:
{
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow",
"Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket"],
"Resource": ["arn:aws:s3:::mercadofresco-artefactos",
"arn:aws:s3:::mercadofresco-artefactos/tienda/*"] },
{ "Effect": "Allow", "Action": ["kms:Decrypt"],
"Resource": "arn:aws:kms:eu-west-1:111122223333:key/*",
"Condition": {"StringEquals": {"kms:ViaService": "s3.eu-west-1.amazonaws.com"}} }
]
}The second block is needed because the bucket is encrypted with alias/mercadofresco-datos: without KMS
permission, the agent downloads the object and fails to decrypt it, with an access denied error that looks
like an S3 problem. The second requirement is internet egress or VPC endpoints: the agent talks to the
CodeDeploy API and to S3, and a private subnet with no NAT and no endpoints leaves the deployment hanging. To
diagnose, systemctl status codedeploy-agent and the log at
/var/log/aws/codedeploy-agent/codedeploy-agent.log.
The appspec.yml for EC2
The appspec.yml goes in the root of the ZIP, not in a subdirectory, and it tells the agent what to copy
and what to run:
version: 0.0
os: linux
files:
- source: /app
destination: /opt/mercadofresco/app
- source: /scripts
destination: /opt/mercadofresco/scripts
- source: /VERSION
destination: /opt/mercadofresco
# Without this, a file that already exists at the destination fails the whole deployment
file_exists_behavior: OVERWRITE
permissions:
- object: /opt/mercadofresco/app
owner: mercadofresco
mode: 640
type: [file]
hooks:
ApplicationStop:
- { location: scripts/parar_servicio.sh, timeout: 60, runas: root }
BeforeInstall:
- { location: scripts/comprobar_espacio.sh, timeout: 30, runas: root }
AfterInstall:
- { location: scripts/instalar_dependencias.sh, timeout: 300, runas: mercadofresco }
- { location: scripts/cargar_configuracion.sh, timeout: 60, runas: mercadofresco }
ApplicationStart:
- { location: scripts/arrancar_servicio.sh, timeout: 120, runas: root }
ValidateService:
- { location: scripts/comprobar_salud.sh, timeout: 180, runas: mercadofresco }
- { location: scripts/calentar_cache.sh, timeout: 120, runas: mercadofresco }file_exists_behavior: OVERWRITE deserves a warning: without it, the default value is DISALLOW and the
deployment fails if a file already exists at the destination. Since the destination almost always has the
previous version, it is the first failure everybody runs into. RETAIN exists for files the application
generates and that must not be overwritten.
The lifecycle hooks, one by one
| Hook | When | Revision | What it is for |
|---|---|---|---|
ApplicationStop |
Before anything | The PREVIOUS one | Stopping the service cleanly |
BeforeInstall |
Before copying | The new one | Preliminary checks, backups |
AfterInstall |
After copying | The new one | Dependencies, permissions, configuration |
ApplicationStart |
After installing | The new one | Starting the service |
ValidateService |
Last | The new one | Checking that it really works |
BeforeAllowTraffic |
Blue/green only | The new one | Warming up before receiving traffic |
AfterAllowTraffic |
Blue/green only | The new one | Validating with real traffic |
Three things to understand properly. ApplicationStop runs from the PREVIOUS revision, and it is the
source of a classic problem: if the previous deployment left a broken parar_servicio.sh, the new
deployment fails on the very first hook even if the new code is perfect — and on an instance's first
deployment it does not run at all, because there is no previous revision. Write it bulletproof and make it
finish successfully even if it cannot find the service. ValidateService is the hook that justifies
everything else: without it, "deployed" means "the files are copied and systemctl did not complain",
which is not the same as "the shop works". And timeouts are per hook, with a maximum of one hour: an
AfterInstall with no cache can go over 300 seconds, and a ValidateService with 10 seconds always fails on
the first instance, which is the slowest to warm up.
MercadoFresco's real scripts
Stopping the service without failing if it is not there:
#!/bin/bash
# scripts/parar_servicio.sh -> ApplicationStop
# CAREFUL: it runs from the PREVIOUS revision. It has to be bulletproof.
set -u # We do NOT use 'set -e': we want to control the failures ourselves
systemctl list-unit-files | grep -q '^mercadofresco.service' || {
echo "The service does not exist yet (first deployment)."; exit 0; }
systemctl stop mercadofresco
for i in $(seq 1 30); do
systemctl is-active --quiet mercadofresco || {
echo "Stopped cleanly in ${i}s"; exit 0; }
sleep 1
done
echo "It did not stop in 30s. Forcing."
systemctl kill -s SIGKILL mercadofresco || true
exit 0 # NEVER fail here: it would block every future deploymentThe final exit 0 is deliberate: an ApplicationStop that fails leaves the instance in a state you can only
get out of by recreating the deployment group.
Checking health for real:
#!/bin/bash
# scripts/comprobar_salud.sh -> ValidateService
set -euo pipefail
for i in $(seq 1 30); do
RESPONSE=$(curl -sf -m 5 http://localhost:8080/salud 2>/dev/null || echo '{}')
if [ "$(echo "$RESPONSE" | jq -r '.estado // "ko"')" = "ok" ]; then
# A 200 is not enough: we check the critical dependencies one by one
for dep in aurora redis sqs; do
HEALTH=$(echo "$RESPONSE" | jq -r ".dependencias.${dep} // \"ko\"")
[ "$HEALTH" = "ok" ] || { echo "Dependency ${dep} in state ${HEALTH}"; exit 1; }
done
# And that the live version is the one we have just installed
LIVE_VERSION=$(echo "$RESPONSE" | jq -r '.version')
[ "$LIVE_VERSION" = "$(cat /opt/mercadofresco/VERSION)" ] || {
echo "Serving ${LIVE_VERSION}, not the expected one"; exit 1; }
echo "Health OK, version ${LIVE_VERSION}"; exit 0
fi
echo "Attempt ${i}/30"; sleep 5
done
echo "Did not pass the health check in 150s"
exit 1This script is the heart of deployment safety, and it makes three checks that a simple curl -f /salud does not: that the critical dependencies respond — it is no use for the process to be alive if it
cannot reach Aurora — that the version answering is the one you have just installed — which catches the
case where the service never actually restarted — and that all of it happens within a deadline. Its exit 1
is what triggers the rollback.
Warming the cache before receiving traffic:
#!/bin/bash
# scripts/calentar_cache.sh -> ValidateService (or BeforeAllowTraffic in blue/green)
set -euo pipefail
curl -sf -m 30 -X POST http://localhost:8080/interno/precargar-catalogo
for sku in FRUT-0012 VERD-0034 PESC-0007 CARN-0021; do
curl -sf -m 5 "http://localhost:8080/productos/${sku}" > /dev/null
done
echo "Cache warm"Without this step, the first wave of traffic finds an empty cache, goes to Aurora, and
TiempoConfirmacionPedido shoots up for two minutes. At the Friday peak with 900 orders/hour, that is
exactly what makes mercadofresco-alb-latencia-alta fire and roll back a deployment that was perfectly
fine.
In-place deployment: AllAtOnce, HalfAtATime and OneAtATime
| Configuration | At a time | Minimum capacity | Duration (4 instances) | When |
|---|---|---|---|---|
AllAtOnce |
All of them | 0 | ~3 min | Development. Never in production |
HalfAtATime |
50 % | 50 % | ~6 min | Production with spare capacity |
OneAtATime |
1 | N-1 | ~12 min | Maximum caution; small fleets |
AllAtOnce in production is an outage, not a deployment. Every instance out at the same time is exactly
what you are trying to avoid; and if it fails, no instance is left with the good version to go back to.
You can define your own configuration with aws deploy create-deployment-config and
--minimum-healthy-hosts type=FLEET_PERCENT,value=90. But be careful: with that 90 % and a fleet of 2
instances, the healthy minimum is 2 (it rounds up) and no deployment can start; with small fleets use
HOST_COUNT instead of percentages.
Blue/green with the ASG and the target groups
In blue/green no existing instance is touched: new ones are created, they are checked, they are given the traffic and only then are the old ones terminated.
flowchart TB
ALB[alb-mercadofresco-tienda] -->|100% before| TGA[tg-mercadofresco-tienda]
ALB -.->|100% after| TGV[tg-mercadofresco-tienda-verde]
TGA --> AZUL[Blue: v1.4.3<br/>i-aaa1, i-aaa2]
TGV --> VERDE[Green: v1.5.0<br/>i-vvv1, i-vvv2]
VERDE --> H[BeforeAllowTraffic:<br/>warm the cache]
H --> S{State OK?}
S -->|No| F[Terminate green<br/>Blue keeps serving]
S -->|Yes| C[Switch the ALB listener]
C --> W[Wait 30 min<br/>watching alarms]
W -->|Alarm| R[ROLLBACK:<br/>back to blue in 90 s]
W -->|All good| T[Terminate blue]
aws deploy update-deployment-group \
--application-name app-mercadofresco-tienda \
--current-deployment-group-name dg-mercadofresco-tienda-produccion \
--deployment-style '{"deploymentType":"BLUE_GREEN",
"deploymentOption":"WITH_TRAFFIC_CONTROL"}' \
--blue-green-deployment-configuration '{
"terminateBlueInstancesOnDeploymentSuccess": {
"action": "TERMINATE", "terminationWaitTimeInMinutes": 30 },
"deploymentReadyOption": { "actionOnTimeout": "CONTINUE_DEPLOYMENT" },
"greenFleetProvisioningOption": { "action": "COPY_AUTO_SCALING_GROUP" }}' \
--profile mercadofresco-dev --region eu-west-1terminationWaitTimeInMinutes: 30 is the window in which the blue environment still exists, shut down
but intact, after the traffic switch: it is what makes the rollback take ninety seconds, because all you
have to do is send the listener back to the blue target group — at 0 you save pennies and going back becomes
a full ten-minute deployment. actionOnTimeout: CONTINUE_DEPLOYMENT makes the traffic switch automatic;
the alternative, STOP_DEPLOYMENT, waits for somebody to press "reroute traffic", useful on day one and
unsustainable as a practice. And COPY_AUTO_SCALING_GROUP creates a new ASG by copying
lt-mercadofresco-tienda, as opposed to DISCOVER_EXISTING with already tagged instances.
| Aspect | In place | Blue/green |
|---|---|---|
| Cost during the deployment | None | Doubled capacity |
| Rollback time | Full deployment (~10 min) | ~90 seconds |
| Risk to the stable version | It is overwritten | Untouched until the end |
| Total duration | 6-12 min | 15-40 min |
| Local state on disk | Preserved | Lost: new instances |
MercadoFresco uses in place with HalfAtATime in development and blue/green in production. Doubling
two t3.medium for half an hour is about nine cents per deployment; ninety seconds of rollback are worth
quite a lot more than that.
The Lambda and ECS appspec
For Lambda and ECS the appspec is a different thing: it does not copy files, it declares which version
should receive the traffic. It accepts YAML or JSON.
# appspec-lambda.yml
version: 0.0
Resources:
- mercadofresco-cobrar-pago:
Type: AWS::Lambda::Function
Properties:
Name: mercadofresco-cobrar-pago
Alias: produccion
CurrentVersion: "7" # the one serving now
TargetVersion: "8" # the new one
Hooks:
- BeforeAllowTraffic: mercadofresco-validar-antes-de-trafico
- AfterAllowTraffic: mercadofresco-validar-despues-de-traficoAn important difference: in Lambda the hooks are Lambda functions, not scripts, and they have an
obligation that is always forgotten: reporting their result with
PutLifecycleEventHookExecutionStatus, or CodeDeploy waits until the timeout and fails.
import boto3, json, os
cd = boto3.client("codedeploy")
lam = boto3.client("lambda")
def handler(event, context):
deployment_id = event["DeploymentId"]
status = "Failed"
try:
# We invoke the NEW version directly, before it receives real traffic
r = lam.invoke(
FunctionName=f"mercadofresco-cobrar-pago:{os.environ['VERSION_NUEVA']}",
Payload=json.dumps({"modo": "prueba_humo", "importe_eur": 1.00,
"clave_idempotencia": f"humo-{deployment_id}"}))
body = json.loads(r["Payload"].read())
if r["StatusCode"] == 200 and body.get("estado") == "cobrado":
status = "Succeeded"
except Exception as e:
print(f"Smoke test failed: {e}")
# WITHOUT this call, CodeDeploy waits until the timeout and fails the deployment
cd.put_lifecycle_event_hook_execution_status(
deploymentId=deployment_id,
lifecycleEventHookExecutionId=event["LifecycleEventHookExecutionId"],
status=status)
return {"status": status}Notice the clave_idempotencia derived from the deployment identifier: the smoke test goes down the same
path as a real charge, and without a unique key every deployment would reuse the previous one. It is the
idempotency of 07-05 applied to the deployment itself. In ECS the appspec references the task
definition and the container; the ECS detail is module 10.
Canary and linear: deploying mercadofresco-cobrar-pago
Here there is percentage traffic shifting, the best safety net CodeDeploy offers.
| Configuration | How it shifts | Duration | When |
|---|---|---|---|
AllAtOnce |
100 % in one go | Seconds | Development |
Canary10Percent5Minutes |
10 %, wait 5 min, the rest | ~5 min | Charges: few bad requests |
Canary10Percent30Minutes |
10 %, wait 30 min, the rest | ~30 min | High-risk changes |
Linear10PercentEvery1Minute |
+10 % every minute | ~10 min | Gradual degradation |
Canary or linear, with judgement. The canary keeps a small percentage for a while and then jumps to
100 %: if the failure is obvious, only 10 % suffer it and only for a short time. The linear one goes up bit
by bit and is better at spotting degradations that depend on load — a memory leak, a pool that runs out —
but it exposes 50 % before the deployment is halfway through. For mercadofresco-cobrar-pago, Marta chooses
Canary10Percent5Minutes: a failure in charging is obvious within seconds, not gradual, and with 900
orders/hour at peak, five minutes at 10 % means about 7 orders exposed.
The mechanism relies on versions and aliases, which we saw in 02-05: mercadofresco-cobrar-pago:produccion
points to version 7, and CodeDeploy adds a weight towards version 8, raising it according to the
configuration. Everything that invokes the function uses the alias and notices nothing.
It is launched with aws deploy create-deployment and
--deployment-config-name CodeDeployDefault.LambdaCanary10Percent5Minutes, passing the appspec in
--revision. The alarm that watches it goes on the errors of the alias, not of the whole function:
aws cloudwatch put-metric-alarm \
--alarm-name mercadofresco-cobrar-pago-errores-canario \
--namespace AWS/Lambda --metric-name Errors --statistic Sum \
--dimensions Name=FunctionName,Value=mercadofresco-cobrar-pago \
Name=Resource,Value=mercadofresco-cobrar-pago:produccion \
--period 60 --evaluation-periods 1 --threshold 3 \
--comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
--profile mercadofresco-dev --region eu-west-1The 60-second period and a single evaluation period are deliberate: with a 5-minute window for the canary, an alarm of 3 periods of 5 minutes would fire when the deployment is already at 100 %. The alarm has to be faster than the deployment it is watching.
Automatic rollback: the piece that solves problem 4
This is the section the lesson exists for. CodeDeploy rolls back on its own in three situations, declared in
autoRollbackConfiguration: DEPLOYMENT_FAILURE when a hook returns non-zero or times out,
DEPLOYMENT_STOP_ON_ALARM when an alarm from alarmConfiguration goes to ALARM during the deployment, and
DEPLOYMENT_STOP_ON_REQUEST when somebody stops it by hand.
What matters is understanding what "rolling back" means in each mode, because they are not the same. In place, CodeDeploy launches a new deployment with the previous revision: a full one, with its hooks and its minutes. In blue/green within the wait window, it sends the ALB listener back to the blue target group: ninety seconds, and the blue instances never even noticed. In Lambda or ECS, it returns the alias weight to the previous version: seconds.
| Alarm | What it detects | Why it is here |
|---|---|---|
mercadofresco-alb-latencia-alta |
p95 > 1,500 ms, 2 periods of 60 s | A slow version is a broken version |
mercadofresco-pedidos-fallidos |
DLQ > 5 messages in 5 min | The new consumer is not processing properly |
mercadofresco-tienda-degradada |
Composite (05-01) | The overall picture |
Choosing the alarms is the design decision of this section, and they must meet three conditions.
Detecting what a bad deployment causes: a cost alarm has no business here; a latency or 5xx one does.
Being faster than the deployment, because one of 3 periods of 5 minutes does not protect a 10-minute
deployment. And not firing for unrelated reasons: if mercadofresco-alb-latencia-alta fires every Friday
at 19:00 because of the normal peak, it will roll back correct deployments and the team will end up disabling
the rollback. Watch them for a month before wiring them up.
The initial state of the alarm matters. If it is already in ALARM when you start, CodeDeploy will not
begin: correct — do not deploy onto a system that is already broken — but baffling the first time.
Comparison with manual rollback and with weighted Route 53
| Mechanism | Rollback time | Automatic | Granularity |
|---|---|---|---|
| Manual over SSH (today's way) | 10-30 min, with nerves | No | Instance |
| CodeDeploy in place | ~10 min | Yes | Instance |
| CodeDeploy blue/green | ~90 s | Yes | Whole environment |
| CodeDeploy canary (Lambda) | Seconds | Yes | % of invocations |
| Weighted Route 53 (03-05) | 60 s + TTL | Not out of the box | % of DNS resolutions |
It is worth comparing with weighted Route 53, which in 03-05 also split traffic and might look like an alternative. It is not, for two reasons. The DNS TTL: even if you set the weight to 0 instantly, resolvers and browsers keep using the cached answer for as long as the TTL lasts, so "rolling back" takes minutes and is never complete. And the fact that the split is of resolutions, not of requests: a client that resolves once and keeps the connection stays where it landed. Weighted Route 53 is excellent for moving traffic between regions; CodeDeploy is the right mechanism for deploying a version, because it operates on the load balancer and is immediate and complete.
Schema migrations: expand and contract
Here is the most important warning in the lesson, and it is not a technical detail but a process rule: the code deployment and the schema change must not go in the same step.
The reasoning is simple. During any gradual deployment two versions of the code coexist against a single
database: if the AfterInstall of the new one runs ALTER TABLE pedidos DROP COLUMN direccion_antigua, the
old one — which is still serving half the traffic — starts to fail. And if the deployment rolls back, the
rollback does not undo the ALTER TABLE: you are left with the old code and the new schema, the worst
possible state. The answer is expand and contract, in three deployments:
| Phase | What is done | Compatible with | When |
|---|---|---|---|
| 1. Expand | Add the new without removing anything: NULLABLE column |
Both versions | Before the code |
| 2. Migrate | Code that writes to both and reads the new one with a fallback | Both versions | Normal deployment |
| 3. Contract | Remove the old | Only the new one | Days or weeks later |
Applied to the delivery slot Luis is about to add:
-- PHASE 1 (expand): BEFORE deploying the code. No NOT NULL, no DEFAULT and no
-- index: it neither rewrites the table nor locks it. 1.4.3 does not know the column and ignores it.
ALTER TABLE pedidos ADD COLUMN franja_entrega VARCHAR(10) NULL;
-- PHASE 3 (contract): weeks later, when NO instance is serving 1.4.3 any more
ALTER TABLE pedidos ALTER COLUMN franja_entrega SET NOT NULL;
CREATE INDEX CONCURRENTLY idx_pedidos_franja ON pedidos(franja_entrega);Four practical rules follow from this. Never run migrations in a CodeDeploy hook: the hook runs on
every instance, so with four instances you get four simultaneous ALTER TABLEs, and if it fails halfway
the deployment rolls back but the schema does not. The migration is a pipeline step of its own, before
the application deployment; that is 08-04. Every expansion migration must be reversible or harmless: a
NULLABLE column is redundant but harmless if the code rolls back. And CREATE INDEX CONCURRENTLY so as
not to lock the table, outside the deployment. The full walkthrough, with the queue consumer and the event
contract, is lesson 08-05.
Deployment observability and diagnosing a failure
CodeDeploy publishes every state change to EventBridge, so everything from 07-03 applies, with a pattern
on source: ["aws.codedeploy"], detail-type: ["CodeDeploy Deployment State-change Notification"] and detail.state: ["FAILURE", "STOP"].
With alertas-mercadofresco as the target and an input transformation that leaves a readable message: which
application, which version, which state and the link to the console. A deployment that rolls back on its
own at 19:15 on a Friday is good news, but only if somebody hears about it. Diagnosing a failure:
# Which deployment failed and why; then, which instance and which specific hook
aws deploy get-deployment --deployment-id d-A1B2C3D4E \
--query 'deploymentInfo.[status,errorInformation.code,errorInformation.message]' --output table
aws deploy get-deployment-target --deployment-id d-A1B2C3D4E --target-id i-0abc123def456 \
--query 'deploymentTarget.instanceTarget.lifecycleEvents[?status==`Failed`]'| Error code | What it means | Where to look |
|---|---|---|
HEALTH_CONSTRAINTS |
Fewer healthy instances than required | A percentage with a small fleet? |
SCRIPT_FAILED |
A hook returned non-zero | The agent log on that instance |
SCRIPT_TIMED_OUT |
A hook went over its timeout |
Raise the timeout or speed the script up |
NO_INSTANCES |
No instance matches | Tags or the name of the ASG |
AGENT_ISSUE_* |
The agent is not responding | Is it alive? Is there network egress? |
ALARM_ACTIVE |
An alarm was in ALARM at the start |
Fix the system before deploying |
Cost and cleanup
| Item | Cost |
|---|---|
| CodeDeploy on EC2, Lambda and ECS | 0 USD (0.02 USD/instance on on-premises servers) |
| Doubled capacity in blue/green | ~0.09 USD per deployment (2 × t3.medium, 30 min) |
| Storage of Lambda versions | 75 GB quota per region |
| MercadoFresco total | ~2 USD/month with 20 deployments |
# Old Lambda versions are NOT deleted on their own and they eat into the quota
aws lambda list-versions-by-function --function-name mercadofresco-cobrar-pago \
--query 'Versions[?Version!=`$LATEST`].[Version,LastModified]' --output table
aws lambda delete-function --function-name mercadofresco-cobrar-pago:3
# ASGs orphaned by a failed blue/green: instances running and billing
aws autoscaling describe-auto-scaling-groups \
--query "AutoScalingGroups[?starts_with(AutoScalingGroupName,'CodeDeploy_')].[AutoScalingGroupName,DesiredCapacity]"Orphaned ASGs are the hidden cost of this lesson. A blue/green interrupted halfway can leave an ASG with
the CodeDeploy_ prefix and its instances alive, serving nobody and billing. Check for them after every
failed deployment.
Common Mistakes and Tips
An ApplicationStop that can fail. It runs from the previous revision and blocks every future
deployment of that instance. Always finish with exit 0.
Forgetting file_exists_behavior: OVERWRITE. The default DISALLOW fails the deployment as soon as a
file already exists at the destination, which is always from the second one onwards.
A ValidateService that only checks that the process is alive. One that cannot reach Aurora passes the
check and receives traffic: validate dependencies and version.
Running database migrations in a hook. It runs on every instance, it is not transactional with respect to the deployment, and the rollback does not undo it. It is the most expensive mistake in the module.
AllAtOnce in production, which is not a deployment but a scheduled outage; and healthy fleet
percentages with small fleets, because FLEET_PERCENT=90 with 2 instances demands 2 healthy and no
deployment can start. Use HOST_COUNT.
terminationWaitTimeInMinutes: 0 in blue/green. You save pennies and lose the 90-second rollback, which
is the reason you chose blue/green in the first place.
Alarms slower than the deployment, which fire when everything is already deployed; or alarms with false positives wired to the rollback, which will roll back good deployments until the team disables the rollback altogether. Watch them for a month before wiring them up.
Forgetting PutLifecycleEventHookExecutionStatus in a Lambda hook, which makes it wait until the timeout
with a message that does not mention the missing call; or missing KMS permissions on the instance role,
which stops the ZIP being decrypted with an error that looks like an S3 one.
Tip: test the rollback on purpose. Deploy a version with a ValidateService that returns 1 and time it:
a rollback nobody has rehearsed is a hypothesis. Have a /salud that tells the truth, with the state of
every dependency and the version being served, because it is the piece everything else rests on. And deploy
in development first: broken hooks are discovered just as well there.
Exercises
Exercise 1: the deployment that got stuck
Luis deploys 1.5.0 to dg-mercadofresco-tienda-produccion (in place, HalfAtATime, 4 instances). The first
half goes well; the second fails with SCRIPT_FAILED on ApplicationStop. The rollback is launched and
also fails on ApplicationStop. Result: 2 instances on 1.5.0, 2 on 1.4.3 and no deployment possible.
Investigating, he sees that the parar_servicio.sh of 1.4.3 does set -e and systemctl stop mercadofresco && rm /var/run/mercadofresco.pid, and that this PID file has not existed since 1.4.2.
Answer: (a) why the rollback fails in the same place; (b) why the first two instances did work; (c) how you unblock the situation right now, with the commands; (d) how you stop it happening again; (e) what would have been different if the group were blue/green.
Exercise 2: choosing a strategy for three components
For each component state the target, the deployment configuration, the alarms wired to the rollback and the
justification. (a) The web shop on asg-mercadofresco-tienda (2-4 instances), with a peak on Fridays
from 17:00 to 21:00 and 900 orders/hour, where a 500 error is a lost sale. (b) The Lambda
mercadofresco-cobrar-pago, around 900 invocations/hour at peak, already idempotent (07-05), where a failure
can charge twice or not charge at all. (c) The workers on asg-mercadofresco-trabajadores, which consume
cola-mercadofresco-pedidos, receive no ALB traffic and, if they stop for 5 minutes, the queue grows and
drains afterwards.
Exercise 3: the migration that broke the rollback
On Tuesday at 11:00, Luis deploys 1.6.0 in blue/green. 1.6.0 renames the column direccion to
direccion_entrega, and the AfterInstall runs
ALTER TABLE pedidos RENAME COLUMN direccion TO direccion_entrega;. Green starts up fine, passes
ValidateService and receives 100 % of the traffic. Eight minutes later, mercadofresco-pedidos-fallidos
fires: the queue consumer, still on 1.5.2, fails on every message. CodeDeploy rolls back to blue in 90
seconds. And then the whole shop stops working.
Answer: (a) why the rollback made things worse; (b) how many times the ALTER TABLE ran and what
happened on the second one; (c) the immediate recovery plan; (d) how the change should have been
made, with the phases and what is deployed in each; (e) what automatic control would have stopped this
reaching production.
Solutions
Solution 1
(a) Because ApplicationStop always runs from the previous revision installed on the instance, and
on instances 3 and 4 the previous revision is 1.4.3, whose script is broken. The rollback is a new deployment
of 1.4.3, and its first hook is again the ApplicationStop of whatever is installed — which is still the
broken 1.4.3. The same faulty script runs and fails in exactly the same way. That loop is precisely why this
hook has to be bulletproof.
(b) By chance, and it is worth understanding: on instances 1 and 2 the PID file existed — they were
older and were dragging one along from 1.4.1 — so rm returned 0 and the script finished fine. On 3 and 4,
recreated by the ASG with a clean 1.4.3, there was no file, rm returned 1 and set -e aborted. A
deployment that works on half the fleet and fails on the other half almost always points to divergent state
between instances, and that is a problem in itself: the instances of an ASG should be indistinguishable.
(c) Unblocking. The quick route, if the shop is serving badly, is to fix the script by hand on the
affected instances through Session Manager, editing the
deployment-root/<id>/deployment-archive/scripts/parar_servicio.sh, and relaunch. The clean one is to skip
the hook just once: in the console you can launch a deployment ticking the option to omit
ApplicationStop, BeforeBlockTraffic and AfterBlockTraffic, designed for exactly this case. The
equivalent alternative through the CLI is deleting and recreating the deployment group, because that removes
the reference to the last known revision and ApplicationStop stops running:
aws deploy delete-deployment-group --application-name app-mercadofresco-tienda \
--deployment-group-name dg-mercadofresco-tienda-produccion \
--profile mercadofresco-dev --region eu-west-1
# ...recreate it, and deploy 1.4.3 again with create-deployment.(d) Three measures. In the script, remove set -e, put an unconditional exit 0 at the end and use
rm -f, which does not fail if the file is not there. In the process, a test of the hooks in CodeBuild:
a container that runs each script against a clean system and checks that they return 0 in the cases "service
does not exist", "service stopped" and "service running". And always deploy in development first, where
this failure would have shown up with no consequences.
(e) With blue/green nothing would have happened: the green instances are new and have no previous
revision, so ApplicationStop does not even run. If the deployment failed for another reason, blue —
untouched, with 1.4.3 working — would keep serving 100 % and the rollback would be sending the ALB listener
back. It is an argument in favour of blue/green that does not appear in the tables: it eliminates a whole
class of failures at the root, the ones inherited from the previous deployment.
Solution 2
(a) The web shop. EC2 with the ASG, blue/green, terminationWaitTimeInMinutes: 30 and
actionOnTimeout: CONTINUE_DEPLOYMENT. Alarms: mercadofresco-alb-latencia-alta and
mercadofresco-tienda-degradada. Justification: with 2-4 instances, an in-place deployment takes half
the capacity out just when it is needed most; blue/green keeps 100 % throughout the whole process, and the
90-second rollback is what lets you deploy without fear. Added operational rule: no deployment of the shop
between 16:00 and 22:00 on a Friday, not out of distrust of the mechanism but because getting it wrong at
the peak is disproportionately expensive. Hooks: BeforeAllowTraffic warms the cache — crucial, or the first
wave sends latency up and rolls back a good deployment — and AfterAllowTraffic runs a smoke test with real
traffic.
(b) The charging Lambda. Canary10Percent5Minutes, with
mercadofresco-cobrar-pago-errores-canario (period 60 s, 1 evaluation, threshold 3) and one on Duration
p99. Justification: a charging failure is obvious within seconds, not gradual, so the canary detects it
sooner and exposes less than the linear one. With 900 invocations/hour, 5 minutes at 10 % means about 7
orders exposed, and the idempotency of 07-05 stops a retry after the rollback duplicating the charge. Plus
a BeforeAllowTraffic that invokes the new version with a €1.00 test payment. The detail not to forget:
the alarm must carry the Resource dimension of the alias, or it would measure the errors of every
version together and be contaminated by invocations of the old one.
(c) The workers. In place with AllAtOnce, with no latency alarms, with
mercadofresco-pedidos-fallidos wired to the rollback. Justification: here the reasoning changes
completely, and that is what makes this part interesting. They receive no ALB traffic, so there are no
requests that can fail: if they stop for five minutes, the queue grows and drains afterwards, which is
exactly the buffering we were after in 07-01. Blue/green would be complexity and cost with no benefit. What
you do have to guarantee is a clean ApplicationStop, one that lets the message in flight finish and
does not delete it halfway, or you will have in-flight messages reappearing when the visibility expires. And
watch ApproximateAgeOfOldestMessage after the deployment: if it does not come down within ten minutes, the
new consumer is not processing even though the process is alive.
Solution 3
(a) Because the rollback returned the code to 1.5.2 but did not undo the schema. 1.5.2 queries
direccion, which no longer exists. Before the rollback at least the shop worked with 1.6.0 and only the
consumer was failing; afterwards everything fails, because 100 % of the traffic is served by code
incompatible with the schema. It is the trap the lesson warns about: the rollback rolls artefacts back,
not databases, and a destructive schema change turns your safety net into an accelerator of the
incident.
(b) It ran once per green instance, that is, twice. The first one succeeded; the second failed with
ERROR: column "direccion" does not exist, because the rename had already been done. Whether the deployment
stopped there depends on whether the script swallowed the error. This behaviour is the demonstration of why
migrations do not go in hooks: they are not idempotent and they run as many times as there are
instances.
(c) Immediate recovery, with the shop down, so quickly and without elegance. Converge forwards, not
backwards: the schema is already in the new state and renaming it back is another migration with risk, so
redeploy 1.6.0, which is compatible, and get the shop back. Then, deploy the compatible version of the
consumer, which is still on 1.5.2 and still failing; meanwhile the messages pile up in
cola-mercadofresco-pedidos and will end up in the DLQ, but they are not lost. With the shop serving,
apply the DLQ runbook from 07-05 — contain, classify, reprocess. And a post-mortem with no hunt for culprits,
with the process rule written down.
(d) How it should have been done, in three deployments separated by days. Deployment 1 (expand),
Monday: schema only, with no new code — ALTER TABLE pedidos ADD COLUMN direccion_entrega VARCHAR(255) NULL; plus an UPDATE copying the values — 1.5.2 keeps working because it does not know the new column and
is not bothered by it. Deployment 2 (migrate), Tuesday: the 1.6.0 code writes to both columns and
reads from direccion_entrega with a fallback to direccion, which makes it compatible in both
directions — it coexists with 1.5.2 during the gradual deployment and works if it has to be rolled back —
and here the queue consumer is deployed before the shop, so that it knows how to read the new field
before it starts arriving, which is the producer/consumer order that closes 08-05. Deployment 3 (contract),
the following week, when no instance is serving 1.5.2 any more: ALTER TABLE pedidos DROP COLUMN direccion;. And each schema phase as a pipeline step of its own, run once from a dedicated task.
(e) Three controls, from cheapest to most expensive. An analysis of the migrations in CodeBuild that
fails the build if it finds DROP COLUMN, RENAME COLUMN, ALTER COLUMN ... NOT NULL or DROP TABLE
without an explicit "approved contraction" tag: that is thirty lines of script and would have stopped this
dead. A backwards compatibility test that starts the previous version of the code against the new
schema and runs the smoke suite — it simulates the rollback directly, and it is surprisingly rare in
practice. And mandatory review of every change under migraciones/ through a code owners rule, so that
no schema change gets in on a distracted approval.
Conclusion
Luis's ritual is over. There is no more ssh, no scp, no forty seconds of errors per instance, no two
versions coexisting with no control. There is an application, app-mercadofresco-tienda, two
deployment groups that make the same revision behave differently in development and in production, an
artefact identified in mercadofresco-artefactos and a plan CodeDeploy executes the same way every time.
And above all, something that did not exist: a way back that does not depend on somebody watching.
You know the three targets and what each one supports — on EC2 there is no canary and no linear; on Lambda
and ECS there is no in-place deployment — and you know the agent is only needed on EC2, installed as a
Systems Manager association so that it updates itself. You have mastered the appspec.yml with its files,
permissions and hooks, including file_exists_behavior: OVERWRITE, the first failure everybody hits. And
you know the hooks one by one, with the two truths that prevent the most incidents: that ApplicationStop
runs from the previous revision — and that is why it must always finish with exit 0 — and that
ValidateService is the hook that justifies everything else, checking dependencies and version instead
of settling for a live process.
You know how to choose the pace: AllAtOnce only in development, HalfAtATime with spare capacity,
OneAtATime when caution matters more than the clock, and why a high FLEET_PERCENT with two instances
stops any deployment starting. You have blue/green with its terminationWaitTimeInMinutes of 30 minutes
— literally the price of the 90-second rollback — and its hidden advantage: new instances eliminate at the
root the failures inherited from the previous deployment. And the canary of mercadofresco-cobrar-pago
with Canary10Percent5Minutes, its alarm on the Resource dimension of the alias and periods shorter than
the deployment itself, because an alarm slower than what it watches protects nothing.
The centrepiece is the automatic rollback, and now you know what it means in each case: a full deployment in place, ninety seconds in blue/green within the window, seconds in Lambda. You know that choosing the alarms is a design decision with three conditions — that they detect what a bad deployment causes, that they are faster than the deployment and that they have no false positives, because a noisy alarm ends with the team disabling the rollback altogether. And why weighted Route 53 is not the tool for this: the DNS TTL makes rolling back slow and incomplete.
And you take away the most important warning of the module, which is not technical but a matter of process:
the application deployment and the schema change do not go in the same step. Because two versions coexist
against a single database, because a hook runs once per instance, and above all because the rollback rolls
artefacts back, not databases — a badly placed RENAME COLUMN turns your safety net into an accelerator of
the incident. The answer is expand and contract in three separate deployments.
And there is what is still missing. Every piece works, but nobody has chained them together. Luis still
has to remember to launch the build when he merges, copy the artefact path, write
aws deploy create-deployment with the right bucket and key, wait, look at whether it went well and repeat
it for the next environment. The schema step is run by hand. Nobody checks that what is deployed in
production is exactly what was validated in pre-production, nor is there a moment when Marta formally
approves that a change goes out. Excellent tools and zero orchestration: we still depend on one person
remembering the sequence on a Friday afternoon.
In 08-04, "AWS CodePipeline", everything gets chained together. We will see the difference between
continuous delivery and continuous deployment and why MercadoFresco chooses the first, with manual approval
before production; the structure of pipelines, stages, actions and the artefacts that flow between them; the
V1 and V2 types, the triggers by branch, tag or file path, and pipeline variables; the manual approval
with the exact information Marta needs to decide in thirty seconds; the development, pre-production and
production stages; the quality gates with a Lambda action that queries the MercadoFresco/Tienda metrics
and stops the pipeline if anything degrades; retries, per-action permissions and cross-account access; and
the notifications towards alertas-mercadofresco and Slack.
AWS Course
Module 1: Introduction to AWS
- What Is AWS?
- Setting Up Your AWS Account
- AWS Global Infrastructure
- The AWS Management Console
- AWS CLI and SDKs
Module 2: Core AWS Services
Module 3: Networking and Content Delivery
Module 4: Security and Identity
- AWS Identity and Access Management (IAM)
- AWS Key Management Service (KMS)
- Secrets Manager and Parameter Store
- AWS Shield
- AWS WAF
Module 5: Monitoring and Management
Module 6: Databases
Module 7: Application Integration
- Amazon SQS
- Amazon SNS
- Amazon EventBridge
- AWS Step Functions
- Integration Patterns: Idempotency, Retries and Dead-Letter Queues
