Module 4 ended with an uncomfortable sentence: it is all built, and nobody is watching. MercadoFresco has a shop that scales on its own, a replicated database, a CDN that serves 96 % of the traffic, minimal identities, encrypted data and two application firewalls. And yet, if on Friday at 19:10 the shop starts returning errors, the first person to find out will be a customer on Twitter.

Amazon CloudWatch is the service that solves that. It is not "the AWS graphs": it is the central repository where every service you have built writes down what happens to it, the place where your application's logs are stored, and the engine that decides when to wake somebody up. Marta has been using it in passing since 02-01 —every time we have looked at CPUUtilization or created an alarm we were in CloudWatch— but we have never set it up properly.

This lesson sets it up properly. By the end, MercadoFresco will have a dashboard that Marta opens every morning with her coffee, its own business metrics, all of its logs in one place queryable with something SQL-ish, and —most importantly— the verified certainty that a notice reaches a phone at four in the morning.

Cost warning. CloudWatch is one of the AWS services that most easily runs out of control, and hardly ever because of metrics: because of logs. A log group with infinite retention, or an application with DEBUG on in production, can cost more than the EC2 instances that generate them. Every section of this lesson states its cost, and there is a whole section devoted to it.

1. What CloudWatch is and what it is not
2. The three pillars
3. Metrics: namespaces and dimensions
4. Standard resolution versus high resolution
5. Statistics: why the average lies
6. Periods, aggregation and retention
7. The metrics that really matter from what is already built
8. Why EC2 does not publish memory usage
9. Publishing custom metrics with boto3
10. Batched PutMetricData and statistic values
11. The CloudWatch unified agent on the ASG instances
12. Embedded Metric Format from Lambda
13. Logs: groups, streams and retention
14. The logs MercadoFresco has scattered around
15. Logs Insights: syntax and real queries
16. Answering "why did that order take eight seconds?"
17. Metric filters: from a log pattern to an alarm
18. Subscriptions and export to S3
19. Alarms: thresholds, evaluation and missing data
20. Composite alarms
21. Anomaly detection
22. Alarm actions
23. Checking that the notice really arrives
24. The mercadofresco-produccion dashboard
25. ServiceLens and Synthetics
26. CloudWatch cost and how it explodes
27. Cleanup

CloudWatch is a regional observability service that does three things: store numeric time series (metrics), store timestamped text (logs), and evaluate conditions over those to trigger actions (alarms).

What it is not, and it is worth settling from the start because it saves a lot of confusion:

The most common confusion is the first one. If Luis is looking for "who deleted the bucket", that is not in CloudWatch: it is in CloudTrail. CloudWatch stores what your application and the services say about themselves, not who talked to the AWS API.

Historically the event bus was called "CloudWatch Events" and you will still see that name in old documentation and in some CLI responses. Today it is EventBridge and it is covered in 07-03; this lesson does not touch it.

Notice the arrow that goes from logs to metrics: a metric filter turns a text pattern into a numeric series. It is the bridge that lets you alarm on something that only appears in a log, such as ERROR pago rechazado. And notice too that the ALB writes its access logs to S3, not to CloudWatch Logs: that will have consequences in the Logs Insights section.

A metric is a time series identified by three things:

• Namespace: the container. The AWS ones start with AWS/ (AWS/EC2, AWS/ApplicationELB, AWS/RDS, AWS/Lambda, AWS/S3, AWS/CloudFront). Yours cannot start with AWS/: MercadoFresco uses MercadoFresco/Tienda.
• Metric name: CPUUtilization, TargetResponseTime, PedidosConfirmados.
• Dimensions: up to 30 key/value pairs identifying what the measurement is of. InstanceId=i-0abc..., LoadBalancer=app/alb-mercadofresco-tienda/..., DBInstanceIdentifier=mercadofresco-pedidos.

The detail that trips most people up at first: every distinct combination of dimensions is a different metric and is billed separately. These are three metrics, not one:

And CloudWatch does not aggregate automatically across them. If you publish only the third and then query the first, there will be no data. This has two practical consequences:

1. If you want the total and the breakdown too, publish both.
2. Never use something of high cardinality as a dimension. Putting PedidoId as a dimension creates a new metric per order: 900 metrics an hour, 650,000 a month, at 0.30 USD each. That is more than 190,000 USD a month for a one-line slip. The identity of an order belongs in a log or in an X-Ray annotation (05-02), never in a dimension.

MercadoFresco rule: dimensions are small, closed categories —environment, component, payment method, province— never identifiers.

High resolution is for processes that change within seconds: a queue filling up, a 20-second latency spike that at minute resolution is diluted into the average. MercadoFresco does not use it: its problems are perfectly visible at one minute, and 10-second alarms generate night-time noise. It is a tool for one-off diagnosis, not a default.

EC2's basic metrics come at 5 minutes unless you enable detailed monitoring (roughly 2.10 USD per instance a month in eu-west-1), which brings them down to 1 minute. For MercadoFresco's ASG it is definitely worth it: with 5-minute data, scaling reacts too late to the peak on Friday evenings.

CloudWatch does not store every individual point: it stores aggregates per period. When you query, you choose which aggregate you want:

The classic mistake —and the one that hides the most incidents— is alarming on the Average of a latency. One real minute of the MercadoFresco shop:

Average = 0.17 s. Perfect. Green. Not a single alarm fires.

p99 = 9.4 s. Ten customers a minute are watching a spinner turn for almost ten seconds while they try to pay. They are exactly the customers who matter: the ones who are buying.

Rule: for any latency metric, alarm on p95 or p99, never on Average. The average tells you how the server is doing; the percentile tells you how the customer lives it.

One important nuance: percentiles are only available if the metric is published with enough data. ALB metrics support them natively. For your own metrics, you have to publish individual values or use StatisticValues carefully (we will see this: StatisticValues do not allow percentiles, because they arrive already aggregated).

The period is the aggregation window of the query or the alarm: 60 s, 300 s, 3600 s… It is independent of how often you publish. If you publish every 10 seconds and query with a period of 300, CloudWatch aggregates 30 points into one.

Retention is automatic, free and not configurable:

Two operational consequences:

• You cannot query with a period of 60 a day from two months ago. The data is already aggregated to 5 minutes. If you need to compare last year's Black Friday minute by minute, you must have exported it beforehand.
• Every month Marta exports the business metrics (PedidosConfirmados) to mercadofresco-informes-analitica with get-metric-data, precisely so that Sara can compare campaigns from different years.

This is the table Marta has printed out. Of the hundreds of metrics that MercadoFresco's services publish, these are the ones that really decide something:

Three observations that separate whoever understands this from whoever copies thresholds:

• HTTPCode_ELB_5XX_Count and HTTPCode_Target_5XX_Count are not the same thing. The first means the load balancer could not deliver the request to anybody (no healthy targets, timeout expired). The second means your application answered 500. Confusing them wastes hours searching in the wrong place.
• RDS's FreeStorageSpace is in bytes. The 20 GB threshold is written 21474836480. One zero too many and the alarm never fires.
• Lambda Throttles with a threshold of > 0, not > 10. A single throttle means you have hit the concurrency ceiling and there are customers seeing errors. There is no "acceptable amount of throttling".

It is the question everybody asks on getting here, and the answer explains how the cloud works.

The AWS hypervisor sees the virtual machine from the outside. It can measure CPU consumed, network traffic, disk operations on the EBS volume: all of that goes through its layer. But free memory is a concept of the guest operating system. Only Linux knows how much RAM is in buff/cache and could be freed, and the hypervisor cannot know that without going inside.

The same happens with free disk space: EBS is a block device, and only the file system inside it knows what is occupied.

Practical consequence: for EC2 memory and disk you need to install an agent inside the instance. We do that further down. And a design consequence: that is why Lambda, Fargate and RDS —where AWS manages the operating system— do publish memory without you doing anything.

Infrastructure metrics tell you how the machine is doing. Business metrics tell you how the company is doing, and they are the ones that really detect incidents. In 04-04 we saw why: if 5,000 requests a second arrive and PedidosPorHora does not move, they are not customers.

MercadoFresco already published PedidosPorHora. Now we add the two that are missing:

The code, in the shop:

Details that matter and are not obvious:

• Timestamp is optional; if you leave it out, CloudWatch uses the time of receipt. You can publish data up to 2 weeks old and up to 2 hours into the future. This lets you recover metrics from a batch process that failed.
• The unit matters little to CloudWatch but a great deal to alarms. If you publish in Milliseconds and the alarm expects Seconds, the alarm finds no data and stays in INSUFFICIENT_DATA for ever. Be consistent.
• Publishing Value: 1 repeatedly is correct. CloudWatch adds up the values in the period when you query with Sum. The application does not need to keep a counter.
• province is not used as a dimension in this example. Spain has 52 provinces: it would be acceptable, but it would multiply the number of metrics by 52 (15.60 USD/month for that alone). Marta decided that this breakdown lives in Sara's reports on mercadofresco-informes-analitica, not in CloudWatch.

The code above makes one HTTPS call to the CloudWatch API inside the critical path of a purchase. With 900 orders/hour it works; with a serious peak, that 40 ms call adds to the response time the customer perceives, and if the CloudWatch API slows down, your shop slows down.

Three solutions, from worst to best:

1. Wrap it in try/except and never fail. The bare minimum: a metric must not bring down a sale.
2. Accumulate in memory and publish in batches every 20 seconds from a separate thread (next section).
3. Write the metric to the log in EMF format and let CloudWatch extract it (the Lambda section). Zero cost in the critical path.

PutMetricData accepts up to 1,000 points per call (with a 1 MB body limit). Publishing in batches reduces API cost (0.01 USD per 1,000 calls) and takes latency out of the critical path.

If you only need Sum, Average, Min and Max, you can pre-aggregate yourself and send a single point with StatisticValues. One call instead of nine hundred:

The price of this optimisation: with StatisticValues you lose percentiles. CloudWatch only receives four numbers; it cannot compute a p99 from them. Since the p95 of TiempoConfirmacionPedido is precisely what Marta wants to watch, MercadoFresco does not use StatisticValues for that metric. It does use it for ImportePedido, where Sum and Maximum are all that matters.

A third, intermediate option is the Values + Counts pair, which sends distinct values with their multiplicity and does keep percentiles:

For memory and disk you need an agent inside the instance. The CloudWatch unified agent (amazon-cloudwatch-agent) does two jobs at once: it publishes system metrics and it ships log files to CloudWatch Logs. It replaces the old Perl scripts and awslogs, which are no longer used.

The rol-mercadofresco-tienda role needs this added to pol-mercadofresco-tienda:

Notice two things. cloudwatch:PutMetricData does not accept a resource ARN —hence "Resource": "*" with the namespace condition, exactly the pattern we studied in 04-01—. And the agent's configuration is stored in Parameter Store (04-03), which is the clean way for all the ASG instances to share the same one without baking it into the AMI.

A review of the decisions taken there, because each one has its reason:

• Its own namespace, MercadoFresco/Sistema, separate from MercadoFresco/Tienda. Machine metrics and business metrics do not mix: it makes permissions and dashboards easier.
• append_dimensions with ${aws:AutoScalingGroupName}: the agent resolves those variables by itself, querying the instance metadata. There is no need to generate one file per instance.
• aggregation_dimensions with [] (an empty list) also publishes the metric aggregated over the whole group. That is the one used on the dashboard: Marta cares about "average ASG memory", not that of instance i-0abc.
• multi_line_start_pattern: without this, a 30-line Python traceback becomes 30 disconnected log events and Insights queries find nothing.
• retention_in_days in the configuration itself: the agent creates the group with finite retention from day one. It is the line in the whole file that saves the most money.
• procstat watches the specific gunicorn process. If the process dies and systemd restarts it in a loop, the instance CPU looks normal but procstat gives it away.

And the fragment added to the user data of the lt-mercadofresco-tienda template (02-01):

After updating the user data you have to create a new version of the launch template and run an instance refresh on the ASG:

Cost of the agent: the software is free; you pay for what it publishes. With 4 system metrics aggregated at ASG level plus those of each instance, MercadoFresco publishes about 20 custom metrics: 6 USD a month. The caveat matters: if you leave the per-instance dimensions unaggregated and the ASG rotates instances all day, every new InstanceId creates new metrics. They are metrics that stop receiving data —and therefore stop being billed after the billing period—, but they clutter up the dashboards.

In Lambda, calling PutMetricData is especially expensive: the function is billed per millisecond, so waiting 40 ms for the CloudWatch API is literally paid for. And if the Lambda is invoked 50,000 times a day, that is 50,000 API calls.

The Embedded Metric Format (EMF) solves this elegantly: you write a JSON with a special structure to stdout, and CloudWatch Logs extracts the metrics automatically when it ingests it. Cost inside the function: that of a print.

The conceptual key is in the last two sections of the document:

In other words: EMF lets you keep the S3 object key and the request ID next to the metric, without paying the price of cardinality. When the p99 of TiempoProceso shoots up, an Insights query will tell you exactly which files were the slow ones. That is impossible with PutMetricData.

A practical note: the AWS aws-embedded-metrics library does all of this for you with a decorator. Here the JSON is written by hand so that the structure is visible, which is what you need to understand.

Cost of EMF: you pay for the log ingestion (about 0.63 USD/GB in eu-west-1) and the metrics extracted (0.30 USD each). No API calls are paid for. For high-frequency functions it is clearly worth it.

The hierarchy is simple and worth being clear about:

• Log group: the container. This is where retention, KMS encryption and subscriptions are configured. Example: /mercadofresco/tienda/aplicacion.
• Log stream: a sequence of events from a single source. One instance, one Lambda execution. Example: i-0abc123def456.
• Event: a line with a timestamp and a message. Maximum 256 KB.

By default, a log group is created with Never expire retention. Nobody changes it. Three years later, MercadoFresco would have 400 GB of 2023 nginx logs costing 0.03 USD/GB/month —12 USD a month for data nobody is ever going to read— and growing.

And there is something worse than the cost: a Logs Insights query scans whatever you ask it to, and at 0.0063 USD per GB scanned, a careless search over three years of logs costs more than the search itself.

MercadoFresco's policy:

And the command that must be run the same day a group is created:

A useful audit that Marta runs once a month, to find the groups that have slipped through without retention:

In 05-04 we will see how AWS Config turns this manual audit into a rule that fires by itself.

CloudWatch Logs has two storage classes:

For flowlogs-mercadofresco, which is only queried when there is a network investigation, the Infrequent Access class saves half. For /mercadofresco/tienda/aplicacion, on which metric filters and alarms depend, you have to stay on Standard.

This was one of the five open questions from module 4: "the WAF, VPC, ALB and Lambda logs pile up in five places with no correlation". Let us look at the real map:

Honesty matters here: CloudWatch Logs does not centralise everything. The ALB and CloudFront access logs go to S3 by design, because their volume would make ingestion into Logs prohibitive. The real centralisation at MercadoFresco happens at two levels:

• Hot diagnosis (the last few days): CloudWatch Logs Insights.
• Historical and forensic analysis: S3 + Athena, covered in 05-03.

Enabling the PostgreSQL logs on the RDS instance, which is indeed missing:

That log_min_duration_statement=1000 is going to be decisive in the eight-second order section.

CloudWatch Logs Insights is a query language over the logs. It is not SQL, although it looks like it: it is a pipeline of commands separated by |.

The fields @timestamp, @message, @logStream and @log always exist. If your log is JSON, Insights breaks it apart on its own and you can refer to nivel, pedido_id, duracion_ms directly. That is reason number one to log in JSON from day one.

bin(5m) groups by five-minute windows: that is what turns a list of errors into a trend graph. When you run it, Insights offers "Visualization" and that result can be added directly to the dashboard.

This is the query that in 04-05 decided the move from Count to Block. Now you know exactly where it is run.

The fields @duration, @billedDuration, @maxMemoryUsed and @initDuration are generated by Lambda automatically in the REPORT line of every invocation. @maxMemoryUsed against the configured memory is the most direct way of right-sizing a function: if it uses 80 MB of the 512 configured, you are paying too much.

Somebody trying to reach port 5432 and being rejected by sg-mercadofresco-basedatos. If that IP is internal, it is a configuration error. If it is external, it is a scan.

parse with a regular expression and named groups turns the nginx access log into queryable fields. It is the rescue for all the software that does not log in JSON.

Cost warning: every run charges for GB scanned, not for results returned. Narrowing the time range is what decides the bill: the same query over 1 hour or over 30 days costs 720 times more in the second case. Always shrink the window before refining the query.

This is the second question module 4 left open. A customer writes: "on Thursday afternoon it took about eight seconds for my order to be confirmed". Marta has their email and the rough time.

Prerequisite: a correlation identifier that crosses every component. Without it, this is impossible. In nginx, MercadoFresco adds an X-Peticion-Id header that the application propagates to the Lambda and writes on every log line.

Step 1. Confirm the problem exists and narrow it down.

34 slow requests show up on Thursday between 18:40 and 19:20. It was not one customer's problem.

Step 2. Find the specific request.

One comes back: peticion_id = 7f3a9b2c, duracion_ms = 8140.

Step 3. Follow that identifier across all the groups at once. Insights can query up to 50 log groups in a single query, and that is the real answer to "the logs are in five places with no correlation":

Step 4. Read the timeline.

Found it. It was not the shop (27 + 57 + 52 + 10 + 34 = 180 ms), nor the Lambda (550 ms): it was 7.4 seconds inside PostgreSQL. The log_min_duration_statement logs we enabled earlier are what gave it away.

Step 5. See which query it was.

And up comes a query that runs a SELECT for the price of each order line, in a loop, instead of a single SELECT ... WHERE id IN (...). It is the N+1 pattern: an order with 38 products runs 39 queries.

What we have just done and what we have not. We have located the problem with correlated logs, and it worked because MercadoFresco had a request identifier propagated by hand. But look at the effort: four queries, a timeline assembled manually, and all of it resting on somebody remembering to propagate the header. There is a tool designed for exactly this, which does step 4 automatically and in a graph: it is X-Ray, and it is lesson 05-02. We will come back to this same 8.14-second order there.

You cannot alarm on a log; you can on a metric. A metric filter is the conversion.

Three details from the field:

• The --filter-pattern syntax is not the Insights one. For JSON logs use { $.field = "value" } with &&, ||, =, !=, >, <. For plain text, quotes and terms: "ERROR" -"ERROR test".
• defaultValue=0 is essential. Without it, when there are no errors no data point is published, the alarm stays in INSUFFICIENT_DATA and the metric has gaps that break anomaly detection. With defaultValue=0 the series is continuous.
• Filters only apply to new events. There is no retroactive effect on what has already been ingested.

You can also extract a numeric value from the log instead of counting:

And testing the pattern before creating it, against real events, which avoids the classic filter that never matches:

Two ways of getting logs out of CloudWatch, with different purposes:

A real-time subscription to a Lambda that forwards the critical entries to Slack:

Export to S3 for cheap archiving of the flow logs, which will be looked at once a year:

The bucket needs a policy allowing logs.eu-west-1.amazonaws.com to write. And watch out: there can only be one active export task per account at a time, and it is not incremental. For continuous archiving, the modern option is a subscription to Data Firehose with S3 as the destination.

An alarm has three states:

And four parameters that almost nobody configures right first time:

• --period: the aggregation window. 60, 300…
• --evaluation-periods: how many periods are looked at.
• --datapoints-to-alarm: how many of those must breach. If omitted, all of them.
• --treat-missing-data: what to do with the gaps.

The evaluation-periods + datapoints-to-alarm combination is called "M out of N" and it is the tool against noise. Compare:

With "2 out of 3", a spike lasting a single minute —a deployment, a garbage collector, a backup— does not wake anybody up, but a real problem that oscillates is detected. The "3 out of 3" configuration misses exactly those intermittent problems.

And --treat-missing-data, four options:

The case that explains everything: the alarm on PedidosConfirmados. If the shop goes down completely, the metric stops being published. With notBreaching, the alarm stays in OK while the shop is dead. It is the most common silent failure in CloudWatch, and that is why that particular alarm carries --treat-missing-data breaching.

The alarms MercadoFresco creates in this lesson:

A note on --ok-actions in alarm 1: send the recovery too. Without it, somebody gets a notice in the small hours with no way of knowing whether it is still happening. With --ok-actions, the second message says "it is fine now". A small detail that changes on-call life a great deal.

And a warning about alarm 8: PedidosConfirmados < 1 in 15 minutes is right for business hours, but at 4 in the morning on a Sunday it may be perfectly normal to have no orders. That alarm generates night-time false positives. The right way to solve it is not to raise the period: it is anomaly detection, two sections further down.

A composite alarm is evaluated over other alarms, with AND, OR and NOT. It serves two different purposes:

1. Reducing noise: only notify if several signals coincide.
2. Suppressing child alarms during a known maintenance window.

MercadoFresco's real problem: when the database saturates on a Friday, mercadofresco-rds-conexiones-altas, mercadofresco-alb-latencia-alta and mercadofresco-pedidos-fallidos all fire at once. Three text messages at 19:15 describing the same incident.

And now the step most people forget: remove the SNS action from the child alarms. Otherwise you carry on receiving four notices instead of three.

The child alarm still exists, still changes state and is still visible on the dashboard: it simply no longer notifies. The notice comes from the composite one, which carries the diagnosis.

Suppression during maintenance: the --actions-suppressor parameter lets you point at another alarm —for example, one Marta puts into ALARM by hand during a deployment— that silences the composite while it is active.

Cost: 0.50 USD per composite alarm a month. It usually pays for itself just by not waking somebody up three times.

Instead of a fixed threshold, CloudWatch trains a model with up to two weeks of history and learns the pattern: the Friday peaks, the small-hours troughs, the weekly cycle. The alarm fires when the value leaves the expected band.

It is exactly what solves the night-time false positive of mercadofresco-sin-pedidos: at 4 in the morning the model expects 3 orders, and 0 is anomalous; at 19:00 it expects 900, and 400 is anomalous too. A fixed threshold cannot express that.

Key details:

• LessThanLowerThreshold: it only notifies if there are fewer orders than expected. More is good news, not an alarm. There are also GreaterThanUpperThreshold and LessThanLowerOrGreaterThanUpperThreshold.
• The 2 in ANOMALY_DETECTION_BAND(m1, 2) is the number of deviations. Higher = wider band = fewer notices. Start at 2 and adjust with real data.
• It needs history. For the first few days the band is enormous and the alarm is useless. You have to create it and wait.
• You can exclude a period from training with --configuration and ExcludedTimeRanges: if there was a 6-hour incident, you do not want the model to learn that that is normal.

Cost: 0.30 USD per analysed metric a month, plus 0.30 USD per anomaly alarm.

And an honest warning: anomaly detection is not magic. It detects deviations from the historical pattern, not problems. A slow, steady degradation over three weeks becomes the new "normal" and stops notifying. It complements fixed thresholds; it does not replace them.

The automatic EC2 actions are useful but dangerous. arn:aws:automate:eu-west-1:ec2:recover on StatusCheckFailed_System migrates the instance to another host when the underlying hardware fails, keeping IP and volumes: that is clearly good. By contrast arn:aws:automate:eu-west-1:ec2:reboot on high memory is a bad idea: you reboot in a loop an instance with a memory leak and hide the problem instead of fixing it.

For the ASG instances, moreover, the right action is almost never to reboot: it is to let them fail the /salud health check and let the ASG itself replace them, as we built in 02-01 and 03-03.

Here is the first open question from module 4, and the most important one in this lesson: nobody has checked that an SNS notice reaches a phone at four in the morning.

An alarm that fires at an SNS topic with no confirmed subscribers is worse than having no alarm: it gives a false sense of coverage.

Step 1. See who is really subscribed.

If the SubscriptionArn column shows PendingConfirmation, that subscription receives nothing. It is the most frequent finding when this check is done for the first time: somebody created the subscription months ago, the confirmation email went to the junk folder, and nobody knew.

Step 2. Subscribe the endpoints that are missing.

A note on SMS. For some years now, sending SMS with SNS in many regions has required leaving the SMS "sandbox" and, for Spanish numbers, registering a sender and a use case. If your test SMS does not arrive, check the sandbox status before writing the alarm off as broken. The full detail of SNS —FIFO topics, delivery policies, subscription filters— is lesson 07-02.

Step 3. The drill. Force the alarm state.

This is the command that closes module 4's question:

This does not touch the metric: it forces the alarm state and fires its actions for real. The SMS and the email go out. It is the only honest way of knowing that the channel works.

Afterwards it is returned to reality:

On the next evaluation period, CloudWatch recomputes the real state from the metric data and corrects it by itself if need be.

Step 4. The protocol MercadoFresco adopts.

And one more check, which usually uncovers the real problem: run the drill at 4 in the morning for real, once. Marta did. She discovered that the on-call mobile was on "do not disturb" and that SMS from short numbers did not break the silence. The fix was not an AWS one: it was to configure an exception on the phone. The notification chain includes the phone, and the phone has to be tested too.

A CloudWatch dashboard is a JSON document with a grid 24 columns wide. Widgets are positioned with x, y, width, height.

The criterion Marta designed it with —and this is what you should copy, rather than the JSON— is: at the top whatever answers "is the business all right?", in the middle "which component is failing?", at the bottom the detail. When the phone rings, you read down and in 30 seconds you know where to go.

And creating it:

Details of the JSON that deserve explanation:

• "start": "-PT3H" sets the default window to the last 3 hours. ISO 8601 duration syntax: -PT1H, -P1D, -P7D.
• ["...", { "stat": "p95" }] repeats the previous metric changing only the statistic. It saves repeating the long ARN three times.
• The edge widget carries "region": "us-east-1". CloudFront's metrics and those of the CloudFront web ACL live there, the same trap as in 03-04 and 04-05. A dashboard can mix regions widget by widget, and that makes it the only place where MercadoFresco sees everything together.
• The horizontal annotations draw the threshold line. Watching the graph approach the red line before the alarm fires is half the value of a dashboard.
• The log widget runs an Insights query every time the dashboard loads. Careful: that costs money every time, and if the dashboard sits on an office screen refreshing every minute, that is 1,440 queries a day. Marta gave it limit 20 and a short window for that reason.

Dashboard cost: the first 3 are free; beyond that, 3 USD per dashboard a month. MercadoFresco has two: mercadofresco-produccion and mercadofresco-negocio (Sara's, with orders and amounts). Cost: 0 USD.

Two CloudWatch features mentioned here and put to use in 05-02:

ServiceLens brings together metrics, logs and X-Ray traces in a single service map. It needs X-Ray to be active, so it is covered in the next lesson.

CloudWatch Synthetics runs "canaries": small scripts that behave like a customer, from outside, every N minutes. It is synthetic monitoring: it does not wait for a real customer to detect that the shop is broken. With 900 orders/hour that may not seem important, but at 3 in the morning on a Sunday two hours can go by without a single purchase, and that is exactly the gap through which an outage slips.

MercadoFresco's canary, canario-mercadofresco-compra, runs every 5 minutes and does the full journey: home page, search for "tomate", open the product page, add to basket, reach the payment form (without paying). If any step fails, it fires alertas-mercadofresco.

Two practical warnings:

• A canary that really buys generates real orders. MercadoFresco's stops before confirming, and the test user is flagged so that Sara excludes it from her reports.
• Cost: 0.0012 USD per run. Every 5 minutes is 8,640 runs a month: 10.37 USD, plus the screenshots in S3 and the traces. It is not negligible; it is the most expensive item in this lesson after the logs.

Reference prices in eu-west-1 (approximate; always check the official calculator):

MercadoFresco's calculation:

The five ways to blow up this bill, in order of how often it really happens:

1. Infinite retention on the log groups. It grows non-stop and you never notice it all at once.
2. DEBUG left on in production. It multiplies ingestion by ten from one day to the next. Ingestion is the expensive item: 0.63 USD/GB.
3. A high-cardinality dimension. PedidoId as a dimension: tens of thousands of USD. It is the most expensive mistake you can make in CloudWatch, and it is made in one line.
4. Insights queries over 30 days for debugging. Every refinement of the query scans everything again.
5. A dashboard with log widgets on an office screen. It runs queries every minute for ever.

And the defence: an AWS Budgets budget filtered by the CloudWatch service (module 11) with a notice at 80 %. In 05-04 we will also see how AWS Config automatically detects log groups without retention.

Two warnings:

• Custom metrics cannot be deleted. They stop being billed when they stop receiving data (after the corresponding period), but they remain visible for a while. It is one more reason not to create dimensions lightly.
• Deleting a log group removes its data irreversibly. If there may be a retention obligation, export it to S3 first and check with whoever handles compliance.

1. Alarming on the average of a latency. It is mistake number one. Average of TargetResponseTime stays flat while p99 is at 9 seconds. Use --extended-statistic p95 or p99 for anything that is a time.

2. Leaving log retention on "Never expire". It is the default value. It is the most expensive mistake in the medium term. Set retention the same day you create the group, and audit monthly with describe-log-groups.

3. Using an identifier as a dimension. PedidoId, UserId, RequestId in a dimension create a new metric every time. Thousands of USD. That data belongs in the log or in an X-Ray annotation.

4. A badly chosen --treat-missing-data. For a metric whose absence is the problem —PedidosConfirmados, HealthyHostCount— you have to set breaching. With notBreaching the alarm sits quietly while the shop is dead.

5. Confusing HTTPCode_ELB_5XX_Count with HTTPCode_Target_5XX_Count. The first belongs to the load balancer (no healthy targets, timeout); the second is your application. Searching in the wrong place costs hours.

6. Thresholds in the wrong units. FreeStorageSpace is in bytes; Lambda's Duration in milliseconds; TargetResponseTime in seconds. One zero too many or too few and the alarm never fires.

7. Creating an alarm without checking the SNS subscription. If the ARN is right but nobody has confirmed the subscription, the alarm fires into the void. Check with list-subscriptions-by-topic and run a drill with set-alarm-state.

8. Not configuring --ok-actions. Whoever gets a notice in the small hours needs to know when it has recovered. Without the notification of the return to OK, somebody gets up for nothing or, worse, does not get up when they should.

9. Looking in CloudWatch for who deleted a resource. That is in CloudTrail (05-03). CloudWatch records what your application says about itself.

10. CloudFront metrics looked for in eu-west-1. They are in us-east-1, always. Just like ACM certificates (03-04) and the CloudFront web ACL (04-05).

11. Insights queries over enormous ranges while debugging. Start with 15 minutes, refine the query, and only then widen the window. You pay per GB scanned on every run.

12. Logging plain text instead of JSON. With JSON, Insights breaks the fields apart on its own and queries are trivial. With plain text you have to write regular expressions with parse. Changing the log format on day one costs an hour; doing it three years later, weeks.

13. A metric filter without defaultValue=0. The series has gaps, the alarm stays in INSUFFICIENT_DATA and anomaly detection does not work.

14. Forgetting to remove the actions from the child alarms when creating a composite one. You end up getting one notice more, not fewer.

Final tip: the most valuable metric in a system is almost never a technical one. PedidosConfirmados detects more real incidents than CPUUtilization, because there can be a thousand ways for the shop to fail with the CPU at 30 %, but there is only one way for orders to drop to zero.

MercadoFresco deploys a new version of the shop on a Tuesday at 11:00. The version has a bug that only affects orders paid with a card from one particular bank: roughly 12 % of purchases. The rest works normally.

Incident data: CPUUtilization unchanged (42 %), HealthyHostCount at 2, TargetResponseTime p95 stable at 0.6 s, HTTPCode_Target_5XX_Count with 3 errors every 5 minutes (before: 1). PedidosConfirmados goes from 900/h to 792/h. In the logs, lines appear reading {"nivel":"ERROR","tipo_error":"PAGO_RECHAZADO","pasarela":"banco-x"}.

No current alarm fires. The bug is found 6 hours later through a complaint on social media.

Design the detection: which metric or metrics you would use, how you would obtain them (metric filter, custom metric, expression), what kind of alarm and with exactly which parameters for period, evaluation and missing data. Write the commands. Justify why your proposal would have fired in less than 30 minutes without generating false positives for the rest of the month.

Sara asks whether the shop's search is working properly. The application writes one JSON line per search:

Write:

• a) An Insights query giving, per hour, the number of searches, the average duration, the p95 and the percentage of searches with no results.
• b) A query listing the 20 most searched terms that return zero results (they are products MercadoFresco ought to have in its catalogue: direct value for Sara).
• c) A metric filter and an alarm that warn if the percentage of searches with no results exceeds 20 % for 15 minutes, which is the sign that the search index has been corrupted. Careful: a metric filter counts, it does not compute percentages. You will need a metric expression in the alarm.

Marta gets the bill and CloudWatch has gone from 44 to 610 USD in a month. The breakdown:

Investigation clues:

• The /mercadofresco/tienda/aplicacion group has gone from 8 GB to 620 GB in the month.
• The MercadoFresco/Tienda namespace now has 320 metrics; last month it had 8.
• There are 11,200 Insights runs, all with a 30-day window.
• There is a new canary, canario-mercadofresco-admin, running every minute.
• Three weeks ago Luis deployed a "recommended orders" function that logs the activity of every customer.

Diagnose each item, say exactly what Luis did wrong in each case, write the corrective actions with the commands, and propose three preventive controls so that it does not happen again. Estimate the resulting bill.

Solution 1

Why nothing fired. All the current alarms look at the infrastructure, and the infrastructure is perfect. A 12 % failure on one particular gateway is invisible to the CPU, to the number of healthy targets and to the latency p95 (a rejected payment answers fast, even faster than a correct one). And the drop in orders from 900 to 792 —12 %— sits within the normal variation of a Tuesday. No reasonable fixed threshold detects 12 %.

This is what is called a partial failure, and it is the kind of incident that takes longest to detect in real systems.

Proposed detection: three layers.

Layer 1 — the direct signal. A metric filter on the specific error, broken down by gateway.

Layer 2 — the ratio alarm, which is the good one. An absolute threshold of "more than N rejections" fails in both directions: at night 3 rejections are a huge number, at the Friday peak they are nothing. The right thing is to alarm on the percentage, with a metric expression:

With 12 % rejections against a threshold of 5 %, and "2 out of 3" periods of 5 minutes, this alarm fires in 10-15 minutes. The requirement is to publish IntentosPago too (a second metric filter or a custom metric); without a denominator there is no ratio.

Layer 3 — the generic safety net: anomaly detection on PedidosConfirmados. A 12 % drop sustained for 6 hours does leave the anomaly band, because the model knows the pattern of a Tuesday morning precisely. It is the mercadofresco-pedidos-anomalos alarm we already created: it would have fired, probably within the first hour. It is slower than layer 2, but it is the only one that detects problems nobody has anticipated.

Why it does not generate false positives. Layer 2 looks at a ratio, and a ratio is independent of volume: it works the same at 4 in the morning as on Friday at 19:00. With "2 out of 3" periods of 5 minutes, an isolated spike of rejections —a bank restarting its gateway for a minute— triggers nothing.

The underlying lesson: infrastructure alarms detect total outages. Partial failures, which are the most frequent and the most expensive, are only detected by business metrics and ratios.

Solution 2

a) Hourly overview.

The key is in sum(resultados = 0): in Insights, a boolean expression is worth 1 or 0, so summing it counts the true cases. It is the idiom for computing percentages in this language.

b) Terms with no results: direct business value.

This is not technical monitoring: it is a list of products customers search for and MercadoFresco does not sell. Marta sends it to Sara every Monday. It is the best example of application logs being worth far more than debugging.

c) Metric filter and ratio alarm.

Two filters, because you need a numerator and a denominator:

And the alarm with an expression:

Three justified decisions:

• MAX([total, 1]) in the denominator avoids the division by zero when there are no searches in the small hours. Without it, the expression produces non-numeric values and the alarm behaves erratically.
• 3 out of 3 periods of 5 minutes = 15 minutes, exactly as the brief asked. Here "3 out of 3" is preferable to "2 out of 3": a corrupted index does not fix itself, so the signal will be sustained, and waiting 15 minutes avoids the noise of a search-index deployment.
• notBreaching because in the small hours there may be no searches, and that is not an incident.

Solution 3

Diagnosis, item by item.

1. Log ingestion: 412 USD (8 GB → 620 GB). Luis's "recommended orders" function logs the activity of every customer on every visit. It probably writes one line per product seen, and very probably with nivel: DEBUG because that is how he left it after debugging. 612 GB extra at 0.63 USD/GB is 386 USD. It is the main cause.

Luis's mistake: leaving DEBUG on in production and writing into the application log a stream of analytics events that is not an application log.

Correction, in two stages:

Underlying fix: customer behaviour events do not go to CloudWatch Logs. They go to mercadofresco-informes-analitica in S3, via Firehose, where storage costs 0.023 USD/GB instead of the 0.63 USD/GB of ingestion. It is 27 times cheaper and it is where Sara wants them anyway.

2. Custom metrics: 96 USD (8 → 320 metrics). 320 × 0.30 = 96 USD. Almost certainly, the recommendations function publishes a metric with a medium-cardinality dimension —CategoriaProducto with 300 values, or ModeloRecomendador—.

Luis's mistake: using as a dimension something that is not a small, closed category.

Correction: remove that dimension and, if the breakdown is needed, put it in as an EMF property, which is queryable with Insights and does not cost 0.30 USD per value. Important note for the mental exam: the metrics bill does not go down until the following month, because metrics already created are billed while they receive data. They cannot be deleted; you have to stop publishing them.

And the real prevention, in the role's IAM policy —the 04-01 pattern—:

This does not limit cardinality —IAM cannot—, but it does stop an application creating new namespaces uncontrolled, which is the other half of the problem.

3. Logs Insights: 71 USD (11,200 runs over 30 days). 11,200 queries scanning enormous volumes. Two overlapping causes: somebody debugging with the window on "30 days" without changing it, and very probably a log widget on a dashboard open on a screen that re-runs the query every time it refreshes.

Correction: reduce the default window of the log widgets, set a low limit, and establish the rule of always starting with 15 minutes. Since the group drops from 620 GB to ~8 GB once point 1 is fixed, this item falls by more than 95 % on its own.

4. Synthetics: 27 USD. canario-mercadofresco-admin every minute is 43,200 runs a month. For an admin dashboard used by three people during office hours, it is absurd.

Correction:

From 43,200 to 2,880 runs: from 51.84 to 3.46 USD. And if it only matters during office hours, it can be scheduled with a cron expression instead of rate.

5. Alarms and dashboards: 4 USD. Correct. Leave it alone.

Estimated bill after the corrections:

Three preventive controls:

1. An AWS Budgets budget filtered by service = CloudWatch, with a threshold of 60 USD and a notice at 80 % to aws-alertas@mercadofresco.example. It spots the deviation in days, not in next month's bill. Covered in module 11.
2. An AWS Config rule flagging as non-compliant any log group without retentionInDays, with automatic remediation setting it to 30 days. It is exactly the use case of lesson 05-04.
3. A review step in the deployment process: nobody deploys code that writes new logs without somebody else reviewing the estimated volume and the log level. It is a checkbox in code review, not a tool. Module 8 integrates it into the pipeline.

And a fourth, cultural one, which is the one that really works: show people the bill. When Luis saw that his debug logs cost 386 USD a month —more than the EC2 instances that generated them— he never left DEBUG on again.

MercadoFresco now has eyes. You know that CloudWatch is three things —metrics, logs and alarms— and you know what it is not: not the record of who called the API (CloudTrail, 05-03), not the request tracer (X-Ray, 05-02), not the compliance evaluator (Config, 05-04), nor the event bus (EventBridge, 07-03).

You have mastered the metrics model: namespace, name and dimensions, with the golden rule that every combination of dimensions is a billable metric and that an identifier in a dimension is the most expensive mistake in this service. You know how to choose the statistic —Sum for counters, Average for utilisation, p95/p99 always for latency— and why the 0.17 s average hid the fact that ten customers a minute were waiting nine seconds to pay. You know the automatic retention that aggregates data to 5 minutes after 15 days, and the table of the metrics that really decide something in every service you have built, with the critical difference between HTTPCode_ELB_5XX_Count and HTTPCode_Target_5XX_Count.

You have published the business metrics PedidosConfirmados and TiempoConfirmacionPedido in MercadoFresco/Tienda, first with direct PutMetricData and then in batches from a separate thread to take the call out of the critical path of a purchase. You know why EC2 does not publish memory —the hypervisor cannot see inside the guest operating system— and you have deployed the unified agent on lt-mercadofresco-tienda with its configuration in Parameter Store, its append_dimensions resolved by themselves and its retention_in_days set from day one. And you know the Embedded Metric Format, which in Lambda gives you free metrics in the critical path and —this is the important part— lets you keep the S3 key and the request ID next to the metric without paying for cardinality.

In logs, you have centralised what could be centralised and you know what cannot: the ALB and CloudFront accesses live in S3 and are queried with Athena. You have the retention policy per group, the Infrequent Access class for the flow logs, and the Logs Insights queries that answer real questions —WAF blocks by rule, the p99 of a Lambda with @duration, the rejections at port 5432, and parse with a regular expression for software that does not log in JSON—. And, above all, you have answered the question of the eight-second order: four log groups queried at once with a correlation identifier, a timeline assembled event by event, and the culprit identified —7,402 ms inside PostgreSQL, an N+1 pattern with 39 queries for an order of 38 products—.

You know how to turn a log pattern into a metric with a metric filter (and why defaultValue=0 is not optional), and how to get logs out with real-time subscriptions or export to S3.

In alarms, you have mastered what separates a useful alarm from a source of noise: the "M out of N" combination with --datapoints-to-alarm 2 --evaluation-periods 3, the treatment of missing data —with breaching for metrics whose absence is the incident, like PedidosConfirmados—, the composite alarms that turn three text messages into one and that require removing the children's actions, and the anomaly detection that knows 0 orders at 4 in the morning is normal and at 19:00 is a catastrophe. You have built eight new alarms —which add to those of modules 2 and 4 up to thirteen in total—, the composite mercadofresco-tienda-degradada and the anomaly detector on PedidosConfirmados.

And you have answered the first question module 4 left open, the most uncomfortable of them all: the notice really arrives. You have checked the subscriptions looking for the dreaded PendingConfirmation, you have subscribed the email and the on-call phone, and you have run the drill with set-alarm-state, which fires the real actions without touching the metric. And you know the chain includes the phone: a mobile's "do not disturb" has silenced more alarms than any AWS configuration error.

All of it comes together on the mercadofresco-produccion dashboard, ordered top to bottom —business, components, detail—, with the CloudFront metrics in their us-east-1 widget, the annotation lines that show how close you are to the threshold before it fires, and the log widget with the latest errors. And with the canario-mercadofresco-compra canary walking the shop every five minutes so that on a Sunday in the small hours, without a single real customer, somebody is still checking that you can buy. All for about 44 USD a month, with the five ways of blowing up that figure identified and retention set the same day each group is created.

One concrete dissatisfaction remains, however. The investigation of the eight-second order worked, but it cost four queries, a timeline assembled by hand in a spreadsheet, and it depended entirely on somebody having remembered to propagate the X-Peticion-Id header across all the components. If tomorrow the bottleneck is inside the Lambda, or in the call to the payment gateway, or in the database connection time, the whole job will have to be repeated and the place to look guessed all over again.

There is a tool that does that work by itself, that draws the timeline in a graph and that says, for a specific request, how long each leg of the journey took: AWS X-Ray. In lesson 05-02, "AWS X-Ray and distributed tracing", we will see what a trace, a segment and a subsegment are, how the identifier is propagated with X-Amzn-Trace-Id, why not every request is traced and how sampling is configured, how to instrument the shop with aws_xray_sdk and enable tracing on mercadofresco-estado-pedido with a checkbox and a permission, how to read the service map and search with filters such as annotation.pedido_id, and we will come back to this same 8.14-second order to see it, this time, in a single graph.