AWS Config closed the previous lesson with an honest limitation: it checks the rules you have told it about. That is precise and it is powerful, but it cannot warn you about what you never thought to ask.

Nobody has written a rule saying "tell me if I have orphan EBS volumes costing money", because to write it you would first have to suspect that they exist. Nobody has written a rule about unassociated elastic IPs, which are billed precisely for not being used. And certainly nobody has written a rule about the account's vCPU limit, which —as we will see in this lesson— is the real reason why asg-mercadofresco-tienda cannot go beyond four instances.

AWS Trusted Advisor performs that review. It compares your account against a catalogue of best practices that AWS has distilled from millions of customers, and tells you what you had not thought to ask about. It is not a real-time detection tool nor a replacement for anything that came before: it is the automatic review, the equivalent of somebody very experienced looking at your account once a month and pointing out the obvious things you have not seen for months because you are too close.

This lesson closes module 5, and with it MercadoFresco's observability layer.

Contents

  1. What Trusted Advisor is and how it differs from Config
  2. And how it differs from Well-Architected
  3. The five categories
  4. What each category checks
  5. What you see depending on your support plan
  6. How to make up for what the Basic plan does not give you
  7. An annotated walkthrough of a MercadoFresco report
  8. Service quotas: the invisible limit
  9. The case of the ASG that will not go beyond four instances
  10. Requesting a quota increase
  11. Alarming before you hit a quota
  12. Automation: API, export and notifications
  13. Integration with EventBridge
  14. Compute Optimizer and Cost Optimization Hub
  15. Marta's monthly review routine
  16. Closing the module: the complete observability layer
  17. The five questions from module 4, answered
  18. Total cost added
  19. What comes next: the bottleneck has moved

What Trusted Advisor is and how it differs from Config

AWS Config (05-04) Trusted Advisor
What it evaluates Your rules AWS best practices
Who defines the criteria You AWS
Customisable Yes, completely Very little (some thresholds)
Scope The resources you record The whole account, always
Latency Minutes Refreshed every 24 h (or manually)
Remediates Yes No
Detects what you did not foresee No Yes
Cost Per CI and evaluation Included in the support plan
History Yes, a timeline No

The underlying difference: Config answers questions; Trusted Advisor asks them.

An example that makes it clear. MercadoFresco has a Config rule that checks that all EBS volumes are encrypted. Perfect. But no rule checks whether a volume is of any use, because it occurred to nobody. Trusted Advisor points it out without being asked: "you have 3 volumes in the available state, attached to nothing, that have been costing 24 USD a month since March".

They do not compete. Trusted Advisor gives you the finding; Config turns it into a permanent rule. The natural flow is: Trusted Advisor discovers a problem you had not foreseen → you write a Config rule so it does not happen again → the remediation fixes it on its own. That is the complete cycle.

And how it differs from Well-Architected

Trusted Advisor Well-Architected (11-01)
Nature Automatic Guided questionnaire
Frequency Continuous, refreshed every 24 h One-off review, quarterly or annual
Level Specific resources Architectural decisions
Example "This volume is orphaned" "How do you handle disaster recovery?"
Who takes part Nobody: it just appears The team, in a session lasting hours
Result A list of findings An improvement plan with prioritised risks
Cost Included in support The tool is free

Trusted Advisor looks downwards, at the resources. Well-Architected looks upwards, at the decisions. Trusted Advisor tells you an instance is underutilised; Well-Architected asks you whether your scaling strategy suits your traffic pattern. Both use the same five or six pillars as their conceptual framework, and in fact the Trusted Advisor categories are almost the Well-Architected pillars, but they operate at different altitudes.

Well-Architected is covered in 11-01, opening the module on best practices and costs.

The five categories

flowchart TD
    TA["AWS Trusted Advisor"]
    TA --> C1["COST OPTIMISATION<br/>Underutilised or<br/>unused resources"]
    TA --> C2["PERFORMANCE<br/>Configurations that<br/>limit speed"]
    TA --> C3["SECURITY<br/>Exposure, permissions,<br/>encryption, MFA"]
    TA --> C4["FAULT TOLERANCE<br/>Single points of failure,<br/>backups, Multi-AZ"]
    TA --> C5["SERVICE LIMITS<br/>Usage against quota,<br/>warning at 80%"]
    C1 --> R["Report with a status per check"]
    C2 --> R
    C3 --> R
    C4 --> R
    C5 --> R
    R --> V["Green: fine"]
    R --> A["Amber: investigate"]
    R --> RJ["Red: action recommended"]

The three possible states of each check:

State Means What to do
Green (no problems) Nothing has been detected Nothing
Amber (investigate) There may be a problem Review it with judgement
Red (action recommended) There is a clear problem Act

And a warning that saves a lot of frustration: amber is not always a problem. Trusted Advisor does not know your context. An instance at 5 % CPU may be a perfectly justified standby server. A bucket without versioning may be a bucket of temporary files. Findings are hypotheses that have to be evaluated, not orders.

What each category checks

Cost optimisation:

Check What it looks for
Low-utilisation EC2 instances CPU < 10 % and low network for 4+ of the last 14 days
Unattached EBS volumes available state: they are paid for and serve no purpose
Unassociated elastic IP addresses They are billed for not being used
Idle load balancers ALB/NLB with no registered targets or no traffic
Idle RDS instances No connections for 7 days
Reserved instance / Savings Plans usage Purchase recommendations (module 11)
Old RDS snapshots Manual ones, very old
Redshift and other idle services Unused clusters

Performance:

Check What it looks for
High-utilisation EC2 instances CPU > 90 % sustained: not enough capacity
EBS volumes with throttled performance IOPS at the volume's maximum
CloudFront cache hits Configuration that prevents caching
CloudFront without compression enabled Unnecessary transfer
Security groups with many rules Evaluation latency
Service limits close to the ceiling It also appears here

Security:

Check What it looks for
Security groups with unrestricted open ports 0.0.0.0/0 on sensitive ports
S3 bucket permissions Public read or write
MFA on the root account The most important check of all
Root account usage Recent root activity
Exposed IAM access keys Searched for in public repositories
Access key rotation Keys older than 90 days
S3 access logging and CloudTrail Enabled or not
ACM certificates close to expiry 30 days' notice
Public RDS and EBS snapshots Shared with "everyone"
IAM password policy Minimum requirements

Fault tolerance:

Check What it looks for
RDS without Multi-AZ Single point of failure in the database
ASG in a single availability zone No tolerance to an AZ failure
Load balancers with targets in a single AZ The same
EBS volumes without recent snapshots No backup
RDS backup retention Insufficient period
Route 53 health checks No health checks configured
S3 bucket versioning Recovery from deletions
Redundant VPN tunnels Only one tunnel

Service limits (quotas):

It checks usage against the quota for dozens of services and warns when 80 % is exceeded. It is the most underrated category and the one that has caused the most outages in the history of AWS. It has its own section further down.

What you see depending on your support plan

This is where complete honesty is called for, because it is where the disappointment lies:

Support plan Cost Trusted Advisor checks
Basic 0 USD A subset: basic security + service limits
Developer 29 USD/month or 3 % The same subset as Basic
Business From 100 USD/month (or ~10 % of spend) All of them (~115 checks) + API + notifications
Enterprise On-Ramp From 5,500 USD/month All of them + a shared technical account manager
Enterprise From 15,000 USD/month All + a dedicated technical account manager + guided Well-Architected

What MercadoFresco sees with the Basic plan (0 USD):

Check Available?
Security groups: specific ports unrestricted Yes
S3 bucket permissions Yes
MFA on the root account Yes
IAM password policy Yes
Public RDS snapshots Yes
Public EBS snapshots Yes
IAM usage (existence of users/roles) Yes
Service limits Yes
Publicly exposed access keys Yes
Underutilised EC2 instances No
Unattached EBS volumes No
Unassociated elastic IPs No
Idle load balancers No
RDS without Multi-AZ No
ASG in a single AZ No
CloudFront cache hits No
ACM certificates close to expiry No
Reserved instance recommendations No
Trusted Advisor API No
Weekly email notifications No

In other words: with the Basic plan, MercadoFresco sees basic security and service limits, which is no small thing —they are the two categories that prevent the most outages— but it sees nothing about cost, performance or fault tolerance, and it has no API to automate with.

Is it worth moving up to Business? The honest calculation for MercadoFresco:

Item Figure
Monthly AWS spend ~600 USD
Cost of the Business plan 100 USD/month (minimum)
Potential saving found by the cost checks 30-60 USD/month
Value of technical support (1 h response for production down) Hard to quantify

With 600 USD of spend, paying 100 USD for Trusted Advisor does not pay for itself on the recommendations alone. But the Business plan includes far more than Trusted Advisor: 24×7 technical support with a one-hour response for production being down, access to solutions architects, and the ability to open technical cases. That is the real reason to buy it, and it has to be decided on that basis, not on the checks.

MercadoFresco's decision: stay on Basic for now and make up for what is missing with what has already been learned, revisiting the decision when spend goes above 2,000 USD a month or when the shop becomes critical enough to the business to need support with an SLA. It is written down in the decision log, with a date, just like the decision on Shield Advanced in 04-04.

How to make up for what the Basic plan does not give you

This is where the whole module pays for itself. Every check that Trusted Advisor does not give you on Basic can be reproduced with what you already know:

Missing check How to cover it Lesson
Unattached EBS volumes Advanced Config query 05-04
Unassociated elastic IPs describe-addresses + a script 01-05
Underutilised instances The CPUUtilization metric + Compute Optimizer 05-01
RDS without Multi-AZ The Config rule rds-multi-az-support 05-04
ASG in a single AZ A custom Config rule with Guard 05-04
Idle load balancers The RequestCount metric = 0 05-01
ACM certificates expiring describe-certificate + an alarm 03-03
Low cache hit rates The mercadofresco-cdn-aciertos-bajos alarm 04-04
Reservation recommendations Cost Explorer 11-03
Volumes without snapshots The DLM policy already in place 02-02

The actual scripts. Orphan volumes, with Config:

aws configservice select-resource-config \
  --expression "
    SELECT resourceId, configuration.size, configuration.createTime, tags
    WHERE resourceType = 'AWS::EC2::Volume'
      AND configuration.state.value = 'available'
  " \
  --profile mercadofresco-dev --region eu-west-1

Unassociated elastic IPs, which are billed for not being used:

aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId,Tags[?Key==`Componente`].Value|[0]]' \
  --output table \
  --profile mercadofresco-dev --region eu-west-1

Load balancers with no traffic in the last 7 days:

"""Detects idle load balancers: the cost finding Basic does not give you."""
import boto3
from datetime import datetime, timedelta, timezone

elb = boto3.client("elbv2", region_name="eu-west-1")
cw = boto3.client("cloudwatch", region_name="eu-west-1")

end = datetime.now(timezone.utc)
start = end - timedelta(days=7)

for balancer in elb.describe_load_balancers()["LoadBalancers"]:
    name = balancer["LoadBalancerName"]
    # The ALB dimension is the part of the ARN from "loadbalancer/" onwards
    dimension = balancer["LoadBalancerArn"].split("loadbalancer/")[1]

    data = cw.get_metric_statistics(
        Namespace="AWS/ApplicationELB",
        MetricName="RequestCount",
        Dimensions=[{"Name": "LoadBalancer", "Value": dimension}],
        StartTime=start, EndTime=end,
        Period=86400, Statistics=["Sum"],
    )
    total = sum(p["Sum"] for p in data["Datapoints"])
    if total == 0:
        print(f"IDLE:   {name} — 0 requests in 7 days (~16 USD/month)")
    else:
        print(f"active: {name} — {int(total):,} requests")

ACM certificates close to expiry (remember: the CloudFront ones are in us-east-1):

for REGION in eu-west-1 us-east-1; do
  aws acm list-certificates --region "$REGION" \
    --query 'CertificateSummaryList[].CertificateArn' --output text \
    --profile mercadofresco-dev \
  | tr '\t' '\n' | while read -r ARN; do
      aws acm describe-certificate --certificate-arn "$ARN" --region "$REGION" \
        --query 'Certificate.[DomainName,NotAfter,Status,RenewalEligibility]' \
        --output text --profile mercadofresco-dev
    done
done

Certificates issued by ACM and validated by DNS renew themselves (03-03), but imported ones do not, and an expired certificate brings the whole shop down. This check deserves a place in the monthly routine.

The conclusion that matters: with Config, CloudWatch and a handful of scripts, MercadoFresco reproduces most of what the Business plan would give it, for free. What it cannot reproduce is technical support with an SLA, and that is the real product you are buying.

An annotated walkthrough of a MercadoFresco report

This is the full report, including the checks MercadoFresco does not see with Basic but which we have reproduced with the techniques from the previous section. Annotated finding by finding.

Security

Check Status Detail
MFA on the root account Green Enabled in 01-02
Root account usage Green No activity since February
IAM password policy Green 04-01
Security groups: unrestricted ports Red 1 group with 22 open to 0.0.0.0/0
S3 bucket permissions Green Public access blocked on all 7 buckets
Public RDS snapshots Green None
Exposed access keys Green None detected
Access key rotation Amber 1 key that is 412 days old
CloudTrail logging Green trail-mercadofresco (05-03)

Comment. The red on port 22 is exactly the same finding as the one the mercadofresco-ssh-restringido Config rule gave in 05-04. Two independent tools pointing at the same thing is the best possible confirmation that it is real. The difference: Config remediates it on its own; Trusted Advisor only points at it.

The 412-day-old key belongs to the integracion-proveedor user from exercise 2 of 05-03. The real fix is not to rotate it: it is to replace it with an assumable role with sts:ExternalId (04-01).

Fault tolerance

Check Status Detail
RDS Multi-AZ Green mercadofresco-pedidos is Multi-AZ (02-04)
Automatic RDS backups Green 7 days of retention
ASG across several AZs Green eu-west-1a and eu-west-1b (03-01)
ALB targets across several AZs Green
EBS volumes with snapshots Amber 2 volumes with no snapshot in 30 days
S3 bucket versioning Amber 2 buckets without versioning
Route 53 health checks Amber No health checks on the main record

Comment. The three ambers are perfect examples of findings that have to be evaluated, not obeyed:

  • The 2 volumes without snapshots are the root volumes of the ASG instances. They do not need backups: they are ephemeral by design, they are recreated from lt-mercadofresco-tienda, and snapshotting them would mean paying to copy something that is already in the AMI. Marked as accepted and documented.
  • The 2 buckets without versioning are mercadofresco-registros-web (ALB logs) and mercadofresco-catalogo-fotos. The first one is correct: logs are not overwritten. The second one is a real finding: if somebody uploads the wrong photo over a good one, there is no way back. Fixed.
  • The Route 53 health checks are a real and valuable finding. MercadoFresco has an alias record pointing at CloudFront (03-05) with no health check. There is no failover configured towards anything. It is a pending decision that deserves discussion, not an immediate fix.

Performance

Check Status Detail
High-utilisation EC2 instances Green The ASG scales before it gets there
EBS volumes with throttled performance Green gp3 with enough IOPS (02-02)
CloudFront cache hits Green 96.6 % (03-04)
Compression in CloudFront Amber No compression on 1 cache behaviour
Security groups with many rules Green

Comment. The amber on compression is money straight away: enabling automatic compression in CloudFront for the API responses reduces outbound transfer, which is the most expensive line item on CloudFront. It is a one-checkbox change. Fixed the same day.

Cost optimisation

Check Status Detail Saving
Unattached EBS volumes Red 3 available volumes, 100 GiB in total 8 USD/month
Unassociated elastic IPs Red 2 unassociated IPs 7.30 USD/month
Underutilised EC2 instances Amber mercadofresco-tienda-01 at 4 % CPU ~30 USD/month
Idle load balancers Green The ALB has traffic
Old RDS snapshots Amber 4 manual ones more than 6 months old 3.50 USD/month
Reserved instances / Savings Plans Amber Purchase recommendation ~90 USD/month

Comment, finding by finding:

  • The 3 orphan volumes were left over from instances terminated by hand during the tests in 02-01 and 02-02. Nobody deleted them because terminating an instance does not delete the additional volumes unless DeleteOnTermination is enabled. It is the most common cost finding in AWS. They are deleted, after checking that they contain nothing.
  • The 2 elastic IPs are from the NAT tests and the initial instance. An elastic IP is free while it is associated with a running instance; when it is not, it is billed precisely to discourage hoarding. They are released.
  • mercadofresco-tienda-01 at 4 % is a particularly interesting finding. That instance is the original one from 02-01, from before the ASG. It is no longer of any use: the traffic goes to the ALB and from there to the group. It has been running ever since because nobody remembered to switch it off. It is exactly the kind of thing only an automatic review picks up.
  • The 4 manual snapshots more than 6 months old are from the tests in 02-04. Judgement is needed here: before deleting them it is worth checking there is no retention obligation, and that check is the custom rule from exercise 2 of 05-04.
  • The Savings Plans recommendation is the highest-impact one in the whole report, and it is deliberately not dealt with here: analysing usage commitments is lesson 11-05. Trusted Advisor points at it; the decision is taken with Cost Explorer in front of you.

Service limits

Service Quota Usage % Status
EC2: on-demand vCPU (standard) 16 8 50 % Green
VPCs per region 5 1 20 % Green
Security groups per VPC 2,500 6 0 % Green
Rules per security group 60 8 13 % Green
Elastic IPs per region 5 5 100 % Red
RDS instances 40 2 5 % Green
Lambda functions: concurrency 1,000 ~40 4 % Green
S3 buckets 100 7 7 % Green
CloudFront distributions 200 1 0 % Green
Route 53 hosted zones 500 1 0 % Green
ACM certificates 2,500 2 0 % Green

The red on elastic IPs at 100 % is the most urgent finding in the report, and it is doubly interesting: it has the same cause as the cost finding. The 2 unassociated IPs are taking up 2 of the 5 slots in the quota. If a third NAT gateway or an IP for something else were needed tomorrow, the API call would fail with AddressLimitExceeded and nobody would understand why. Releasing them solves both things at once.

And the vCPU row is what leads to the next case.

Service quotas: the invisible limit

Every AWS account has quotas (formerly "service limits") on practically everything: how many vCPUs you can have running, how many VPCs, how many rules per security group, how much Lambda concurrency.

They exist to protect AWS from uncontrolled usage —accidental or malicious— and to protect you from a surprise bill. Most of them are adjustable; some are not.

Adjustable quota Non-adjustable quota
Example On-demand vCPU, elastic IPs, VPCs Rules per NACL (20), maximum object size in S3
How it is raised A request from Service Quotas It cannot be
Time Minutes to days

What makes quotas dangerous is when you find out: once you have already hit one, at the worst possible moment. An account that works perfectly for months can fail all at once on the Friday when it needs to scale.

Service Quotas is the console where they are viewed and managed:

# View the on-demand vCPU quota
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-1216C47A \
  --query 'Quota.[QuotaName,Value,Adjustable]' \
  --output table \
  --profile mercadofresco-dev --region eu-west-1

# List every quota for a service
aws service-quotas list-service-quotas \
  --service-code ec2 \
  --query 'Quotas[?Adjustable==`true`].[QuotaCode,QuotaName,Value]' \
  --output table \
  --profile mercadofresco-dev --region eu-west-1

# The available service codes
aws service-quotas list-services \
  --query 'Services[].[ServiceCode,ServiceName]' --output table \
  --profile mercadofresco-dev --region eu-west-1

The quotas MercadoFresco keeps an eye on, and why:

Quota Code Value Why it matters
On-demand vCPU (standard) L-1216C47A 16 The ASG's real ceiling
Elastic IPs per region L-0263D0A3 5 NAT gateways and egress
Lambda concurrency L-B99A9384 1,000 The thumbnail peak
RDS instances L-7B6409FD 40 Read replicas
Target groups per ALB L-B22855CB 100 The shop's routes
Rules per WAF web ACL L-C4144F1D 1,500 WCU 04-05

The case of the ASG that will not go beyond four instances

Here is the specific case we promised, and it is a perfect example of why this category matters so much.

asg-mercadofresco-tienda is configured with a minimum of 2, desired 2, maximum 4. Marta had always assumed that 4 was a design decision. Looking at the quotas she finds out it is not quite so:

  • The instances are t3.large: 2 vCPUs each.
  • The account's standard on-demand vCPU quota is 16.
  • Other workloads in the account consume 8 vCPUs.
  • That leaves 8 vCPUs available = 4 t3.large instances.

The ASG maximum is not limited by design: it is limited by the quota. And that has a very specific and very ugly consequence.

On Friday at 19:00, with 900 orders an hour and a capacity of 600 orders an hour per instance, MercadoFresco needs at least 2 instances just for normal traffic. If a peak of twice that arrives —a campaign, a mention on social media, a bank holiday Friday— the ASG tries to scale. And if something else in the account had consumed vCPUs in the meantime:

Launch failed: You have requested more vCPU capacity than your current
vCPU limit of 16 allows for the instance bucket that the specified
instance type belongs to.

The ASG does not scale. Customers see errors. And the cause is not in any CloudWatch alarm, nor in any Config rule, nor in any X-Ray trace. It is in an account quota that nobody has ever looked at.

Worse still: the failure happens at the worst possible moment, because hitting the limit is precisely what happens when you scale. It is the perfect latent failure.

The mercadofresco-asg-al-maximo alarm from 04-04 warns when the group reaches its maximum, which is a good sign that more capacity is needed. But it warns when you are already at the ceiling; it does not warn you that the ceiling is lower than you thought.

Requesting a quota increase

# 1. See the current value and whether it is adjustable
aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-1216C47A \
  --profile mercadofresco-dev --region eu-west-1

# 2. Request the increase
aws service-quotas request-service-quota-increase \
  --service-code ec2 \
  --quota-code L-1216C47A \
  --desired-value 64 \
  --profile mercadofresco-dev --region eu-west-1

# 3. Track the status of the request
aws service-quotas list-requested-service-quota-change-history \
  --service-code ec2 \
  --query 'RequestedQuotas[].[QuotaName,DesiredValue,Status,Created]' \
  --output table \
  --profile mercadofresco-dev --region eu-west-1

The possible states: PENDING, CASE_OPENED, APPROVED, DENIED, CASE_CLOSED.

Practical tips, learned the hard way:

  1. Request in advance. Small increases are approved automatically in minutes; large ones go to a support case and can take days. Requesting on Friday afternoon because you are hitting the limit right now is exactly the scenario to avoid.
  2. Ask for headroom, but be reasonable. Asking for 64 vCPUs when you use 8 is reasonable if you expect to grow. Asking for 10,000 with no justification gets denied.
  3. Justify it. If the increase goes to support, explaining the use case —"an e-commerce grocery shop with Friday peaks, an ASG that needs to scale to 12 instances"— speeds things up a lot.
  4. Quotas are per region. Raising it in eu-west-1 does not raise it in us-east-1. If you have a disaster recovery plan in another region, raise it there too, or you will discover the problem on the day of the disaster.
  5. Some quotas are per account, not per region: IAM, S3, CloudFront.
  6. Request through Organizations when you have several accounts (09-04): quota templates apply default values to new accounts.

And the arithmetic you have to do before deciding on the value:

Scenario Instances vCPUs needed
Normal (2 instances) 2 × t3.large 4
Friday peak (4) 4 × t3.large 8
Double peak (8) 8 × t3.large 16
During a blue/green deployment (×2) 16 × t3.large 32
Headroom for other workloads +16
Recommended quota 64

Look at the fourth row: a blue/green deployment temporarily doubles the number of instances. It is a case that is systematically forgotten when calculating quotas, and it is why so many first zero-downtime deployments fail. It is covered in 08-03.

Alarming before you hit a quota

Service Quotas publishes usage metrics to CloudWatch in the AWS/Usage namespace. That lets you alarm before you reach the limit, which turns a quota from a trap into a managed number.

aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-cuota-vcpu \
  --alarm-description "vCPU usage above 70% of the account quota" \
  --namespace AWS/Usage \
  --metric-name ResourceCount \
  --dimensions Name=Service,Value=EC2 \
               Name=Resource,Value=vCPU \
               Name=Type,Value=Resource \
               Name=Class,Value=Standard/OnDemand \
  --statistic Maximum --period 300 --evaluation-periods 2 --datapoints-to-alarm 2 \
  --threshold 11 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

That threshold of 11 is 70 % of 16. The elegant way of expressing it, which adjusts itself if the quota changes, is a metric expression using the SERVICE_QUOTA() function:

aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-cuota-vcpu-porcentaje \
  --alarm-description "vCPU usage above 70% of the quota, adjusts itself" \
  --evaluation-periods 2 --datapoints-to-alarm 2 \
  --threshold 70 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --metrics '[
    {
      "Id": "usage",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Usage",
          "MetricName": "ResourceCount",
          "Dimensions": [
            {"Name":"Service","Value":"EC2"},
            {"Name":"Resource","Value":"vCPU"},
            {"Name":"Type","Value":"Resource"},
            {"Name":"Class","Value":"Standard/OnDemand"}
          ]
        },
        "Period": 300,
        "Stat": "Maximum"
      },
      "ReturnData": false
    },
    { "Id": "quota", "Expression": "SERVICE_QUOTA(usage)", "ReturnData": false },
    { "Id": "percentage", "Expression": "100 * (usage / quota)",
      "Label": "% of the vCPU quota used", "ReturnData": true }
  ]' \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

SERVICE_QUOTA() is the function that makes this work properly: it returns the current value of the quota for that metric. If AWS approves the increase to 64 tomorrow, the alarm recalibrates itself without anybody touching anything. It is exactly the kind of detail that separates an alarm that ages well from one that has to be maintained.

The same technique for Lambda concurrency, which is the other quota that could bite MercadoFresco during the peak of photo uploads to the catalogue:

aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-cuota-lambda-concurrencia \
  --namespace AWS/Lambda --metric-name ConcurrentExecutions \
  --statistic Maximum --period 60 --evaluation-periods 3 --datapoints-to-alarm 2 \
  --threshold 700 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

Cost: 0.10 USD per alarm. Two quota alarms, 0.20 USD a month, to avoid a scaling failure during a Friday peak. It is probably the best value-for-money decision in the entire module.

Automation: API, export and notifications

Important: the Trusted Advisor API (support and trustedadvisor) requires the Business plan or higher. On Basic, these commands return a subscription error. They are included because it is what you will use as soon as your organisation has that plan.

# List all the available checks
aws support describe-trusted-advisor-checks \
  --language es \
  --query 'checks[].[id,category,name]' --output table \
  --region us-east-1 --profile mercadofresco-dev

# The result of one particular check
# (0Xc6LMYG8P = underutilised EBS volumes)
aws support describe-trusted-advisor-check-result \
  --check-id 0Xc6LMYG8P --language es \
  --region us-east-1 --profile mercadofresco-dev

# Force a refresh of a check
aws support refresh-trusted-advisor-check \
  --check-id 0Xc6LMYG8P \
  --region us-east-1 --profile mercadofresco-dev

# A summary of the status of all of them
aws support describe-trusted-advisor-check-summaries \
  --check-ids 0Xc6LMYG8P Qch7DwouX1 hjLMh88uM8 \
  --region us-east-1 --profile mercadofresco-dev

The support API is only in us-east-1. It is global and is served from there, just like ACM for CloudFront (03-04) or the CloudFront web ACLs (04-05). It is a recurring AWS pattern that you now recognise.

A script that exports the full report to CSV for the monthly review:

"""Exports the Trusted Advisor findings to CSV. Requires the Business plan."""
import boto3
import csv
from datetime import date

support = boto3.client("support", region_name="us-east-1")

checks = support.describe_trusted_advisor_checks(language="es")["checks"]
filename = f"trusted-advisor-mercadofresco-{date.today()}.csv"

with open(filename, "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["Category", "Check", "Status",
                     "Flagged resources", "Estimated saving USD"])

    for check in checks:
        result = support.describe_trusted_advisor_check_result(
            checkId=check["id"], language="es"
        )["result"]

        if result["status"] == "ok":
            continue   # Only warning and error are of interest

        saving = 0
        if "costOptimizing" in result.get("categorySpecificSummary", {}):
            saving = result["categorySpecificSummary"]["costOptimizing"] \
                           .get("estimatedMonthlySavings", 0)

        writer.writerow([
            check["category"],
            check["name"],
            result["status"],                            # warning / error
            result["resourcesSummary"]["resourcesFlagged"],
            round(saving, 2),
        ])

print(f"Report written to {filename}")

Weekly email notifications: with the Business or Enterprise plan, Trusted Advisor sends a weekly summary to the configured contacts. They are enabled in the console preferences, and the recipients are the account's alternate contacts —the operations one, the security one, the billing one— which we configured in 01-02. If you did not configure them then, now is the moment: aws-alertas@mercadofresco.example.

Integration with EventBridge

Trusted Advisor emits events when the status of a check changes. With EventBridge (07-03) you can react automatically:

{
  "source": ["aws.trustedadvisor"],
  "detail-type": ["Trusted Advisor Check Item Refresh Notification"],
  "detail": {
    "status": ["ERROR", "WARN"],
    "check-name": [
      "Security Groups - Specific Ports Unrestricted",
      "Amazon S3 Bucket Permissions",
      "MFA on Root Account",
      "Service Limits",
      "Exposed Access Keys"
    ]
  }
}
aws events put-rule \
  --name regla-mercadofresco-trusted-advisor \
  --description "Critical Trusted Advisor findings towards alerts" \
  --event-pattern file://patron-trusted-advisor.json \
  --profile mercadofresco-dev --region us-east-1

aws events put-targets \
  --rule regla-mercadofresco-trusted-advisor \
  --targets 'Id=1,Arn=arn:aws:sns:us-east-1:111122223333:alertas-mercadofresco-global' \
  --profile mercadofresco-dev --region us-east-1

Two details: Trusted Advisor events are emitted in us-east-1, so the rule goes there; and filtering by check-name is essential, because without it you would get a notification every time any of the ~115 checks changed.

What EventBridge enables, and this is the pattern to copy: not just notifying, but acting. A rule can invoke a Lambda that, faced with an "exposed access key" finding, disables the key immediately. When a key turns up in a public repository, every minute counts. The full detail of EventBridge —patterns, targets, buses, retries— is lesson 07-03.

Compute Optimizer and Cost Optimization Hub

Two complementary services that are mentioned here and developed in module 11:

AWS Compute Optimizer analyses the CloudWatch metrics of the last 14 days —more if you enable the agent's memory metrics that we installed in 05-01— and recommends the optimal instance type for each workload. It is free for the basic recommendations.

aws compute-optimizer get-ec2-instance-recommendations \
  --query 'instanceRecommendations[].[instanceName,currentInstanceType,finding,recommendationOptions[0].instanceType,recommendationOptions[0].performanceRisk]' \
  --output table \
  --profile mercadofresco-dev --region eu-west-1

Typical output for MercadoFresco:

Instance Current type Finding Recommended Risk
mercadofresco-tienda-01 t3.large Over-provisioned t3.small Very low
The ASG instances t3.large Optimized

And here an important warning is in order: Compute Optimizer does not know that mercadofresco-tienda-01 should not exist. It recommends shrinking it to t3.small and saving 20 USD. The right answer is to switch it off, and save 60. Automatic tools optimise what is there; the question of whether it should be there is still asked by a person.

Compute Optimizer also analyses EBS volumes, Lambda functions —recommending the optimal memory, which in Lambda also determines the CPU— and ECS services on Fargate (module 10).

Cost Optimization Hub consolidates in a single place the cost recommendations from Compute Optimizer, Trusted Advisor, Cost Explorer and the reservation recommendations, with the estimated saving aggregated and without duplication. It is lesson 11-03.

This lesson does not go into costs. The whole of module 11 is devoted to it: tagging and allocation (11-02), Cost Explorer (11-03), Budgets (11-04) and Savings Plans (11-05).

Marta's monthly review routine

This is the concrete checklist that Marta runs on the first Monday of every month, in an hour. It is the operational product of the whole of module 5.

Security (15 minutes)

  • [ ] Trusted Advisor, security category: no reds.
  • [ ] Config compliance dashboard: number of non-compliant rules against last month.
  • [ ] aws cloudtrail validate-logs over the previous month; save the output.
  • [ ] Athena query: unknown source IPs and AccessDenied grouped together (05-03).
  • [ ] Review IAM users with keys more than 90 days old.
  • [ ] Check that there is no root account activity.

Cost (10 minutes)

  • [ ] available EBS volumes (Config query).
  • [ ] Unassociated elastic IPs.
  • [ ] Load balancers with no traffic in 7 days.
  • [ ] Compute Optimizer recommendations.
  • [ ] Compare this month's bill with the previous one; investigate any deviation greater than 20 %.
  • [ ] Manual RDS and EBS snapshots more than 6 months old.

Reliability (10 minutes)

  • [ ] Trusted Advisor, fault tolerance.
  • [ ] Service quotas: nothing above 70 %.
  • [ ] ACM certificates: none less than 60 days from expiry (in both regions).
  • [ ] Check that the canario-mercadofresco-compra canary has not failed.
  • [ ] Check that the DLM policies have created the expected snapshots (02-02).

Observability (15 minutes) — the most important one and the one most often skipped

  • [ ] Quarterly drill: set-alarm-state on a critical alarm and confirm that the notice reaches the on-call phone.
  • [ ] SNS subscriptions: none in PendingConfirmation.
  • [ ] Log groups without retention (Config rule, should give zero).
  • [ ] CloudWatch cost for the month: has it grown? why?
  • [ ] Alarms that fired this month: was any of them a false positive? Adjust or delete it.
  • [ ] Alarms that did not fire when they should have: add the missing one.

That penultimate point is what keeps the system alive. An alarm that produces repeated false positives ends up being ignored, and on the day it is real nobody will look at it. An alarm that gets ignored is worse than no alarm at all, because it gives a false sense of coverage. Deleting alarms is as important as creating them.

Technical debt (10 minutes)

  • [ ] Review the decision log: any of them with an overdue review date? (Shield Advanced in 04-04, the support plan in this lesson, the migration to ADOT in 05-02).
  • [ ] WAF rules marked as temporary during an incident (04-05).
  • [ ] Accepted Config and Trusted Advisor exceptions: are they still valid?

Closing the module: the complete observability layer

flowchart TD
    subgraph INFRA["MercadoFresco infrastructure - modules 1 to 4"]
        A["ASG + EC2"]
        B["ALB + CloudFront"]
        C["RDS + replica"]
        D["Lambdas + S3"]
        E["VPC + SG + WAF + KMS"]
    end

    INFRA --> M["05-01 CLOUDWATCH<br/>metrics, logs, alarms<br/>mercadofresco-produccion dashboard"]
    INFRA --> X["05-02 X-RAY<br/>end-to-end traces<br/>service map"]
    INFRA --> T["05-03 CLOUDTRAIL<br/>who called the API<br/>trail-mercadofresco"]
    INFRA --> G["05-04 AWS CONFIG<br/>state and compliance<br/>+ automatic remediation"]
    INFRA --> V["05-05 TRUSTED ADVISOR<br/>best practices<br/>and service quotas"]

    M --> SNS["alertas-mercadofresco<br/>email + SMS VERIFIED"]
    X --> SNS
    T --> SNS
    G --> SNS
    V --> SNS

    M -.->|"ServiceLens"| X
    T -.->|"triggers the recording"| G
    G -.->|"finding -> rule"| V
    V -.->|"new rule"| G

    SNS --> P["Marta, Luis and Sara"]

The five pieces and what each one answers, which is the summary to take away:

Lesson Service Question it answers
05-01 CloudWatch Is it working? How much? Is it going well? Will I hear if it fails?
05-02 X-Ray Where did the time go? Which component is to blame?
05-03 CloudTrail Who did what? When, from where, with what result?
05-04 Config Is it configured properly? Is it compliant? Does it fix itself?
05-05 Trusted Advisor What am I missing? Am I about to hit some limit?

And the connections between them, which are what turn five services into a system:

  • CloudTrail triggers Config's recording: without CloudTrail, Config never sees the changes.
  • CloudWatch receives CloudTrail's events and turns them into security alarms.
  • ServiceLens joins the CloudWatch metrics to the X-Ray traces and the logs.
  • Trusted Advisor discovers what you did not think of; Config turns it into a permanent rule.
  • Config remediates; CloudTrail records the remediation; CloudWatch warns it happened.

The full path of an incident, which MercadoFresco now has end to end:

CloudWatch alarm → ServiceLens → X-Ray trace → the guilty subsegment → correlated log → CloudTrail if there was human intervention → a Config rule so that it does not happen again.

The five questions from module 4, answered

Module 4 closed with five open questions. These are the answers, with names and surnames:

1. "Nobody has checked that an SNS notice reaches a phone at 4 in the morning."

Answered in 05-01. The alertas-mercadofresco subscriptions were checked, looking for the dreaded PendingConfirmation, the aws-alertas@mercadofresco.example address and the on-call phone were subscribed, and the drill with set-alarm-state was run, which fires the real actions without touching the metric. It also emerged that the phone's "do not disturb" was silencing SMS from short numbers: the notification chain includes the phone, and the phone has to be tested too. It stays as a quarterly drill in the monthly routine.

2. "The WAF, VPC, ALB and Lambda logs pile up in five places without being correlated."

Answered in 05-01. Centralisation in CloudWatch Logs of what can be centralised —the application, nginx, the Lambdas, WAF (aws-waf-logs-mercadofresco), flow logs (flowlogs-mercadofresco) and PostgreSQL— with retention per group from day one, and with the honesty to acknowledge what does not go there: the ALB and CloudFront access logs live in S3 and are queried with Athena. Logs Insights queries up to 50 groups at a time, and that is what makes the correlation possible.

3. "If a customer says their order takes 8 seconds, Marta does not know whether the problem is in the shop, in the Lambda or in the database."

Answered twice. First in 05-01, with four Logs Insights queries over four groups and a timeline assembled by hand: 7,402 ms inside PostgreSQL. Then in 05-02, in a single search —annotation.pedido_id = "48213"— and a waterfall view where the problem reads in two seconds: 39 subsegments of 190 ms in series, 91 % of the time, an N+1 pattern. Fixed with WHERE id = ANY(...): p95 from 6.8 s to 0.44 s, and DatabaseConnections from 185 to 96 at the Friday peak.

4. "Nobody knows who decrypted the last backup."

Answered in 05-03. Three kms:Decrypt events on mercadofresco-pedidos: RDS encrypting its automatic backup, an ASG instance from 10.0.11.24, and rol-restauracion-copias at 03:42 from 198.51.100.77, without MFA, on snapshot-2026-07-27. The full investigation —freezing the evidence, finding the real AssumeRole, reconstructing the session by accessKeyId— led to Luis testing a restore in the middle of the night. Right intention, wrong procedure. With corrective actions and dates, and an alarm for next time.

5. "Nothing warns if somebody disables the encryption on a bucket or opens an SG to the world."

Answered in 05-04. And it does not just warn: it fixes it. Full timeline: T+0 somebody disables the encryption, T+3 min Config creates the CI and evaluates NON_COMPLIANT, T+4 min the remediation runs AWS-EnableS3BucketEncryption, T+8 min compliant again. No human intervention, with the full trail in CloudTrail. And the initial compliance dashboard found fourteen resources non-compliant for months with nothing detecting them, because they were not events: they were states.

And a sixth, which nobody had formulated and which appears in this lesson: the ASG cannot go beyond four instances because of an account vCPU quota that nobody had ever looked at. That is exactly the kind of problem that only an automatic review finds, and it is the best argument in favour of Trusted Advisor and Service Quotas.

Total cost added

Lesson Service Monthly cost
05-01 CloudWatch (metrics, logs, alarms, dashboard, canary) 43.68 USD
05-02 X-Ray (with optimised sampling) 10.60 USD
05-03 CloudTrail (trail, scoped data events, Insights, Athena) 1.59 USD
05-04 AWS Config (optimised recorder, 29 rules, remediation) 6.40 USD
05-05 Trusted Advisor (Basic plan) + 2 quota alarms 0.20 USD
Total for module 5 ~62.47 USD/month

Put in context:

Item Monthly cost
MercadoFresco infrastructure (modules 1-3) ~540 USD
Security (module 4) 28.36 USD
Observability (module 5) 62.47 USD
Total ~631 USD

Observability is 9.9 % of the total spend. It is a high proportion compared with security, and it is worth saying why it is reasonable: 70 % of that figure is CloudWatch and the canary, that is, log ingestion and the continuous check that people can buy. And against what it costs: the N+1 fix that X-Ray found, for 10.60 USD a month, halved the pressure on the database, avoiding an RDS instance upgrade that would have cost 80 USD a month. Observability paid for itself in the first month.

And the five decisions that keep that figure at 62 USD instead of 600:

  1. Retention on every log group the day it is created (05-01).
  2. No high-cardinality dimensions in custom metrics (05-01).
  3. X-Ray sampling at 0 % on /salud and static files, 100 % on orders (05-02).
  4. Data event selectors by prefix and readOnly: false (05-03).
  5. recordingFrequency: DAILY for instances and volumes in Config (05-04).

Five lines of configuration that separate 62 USD from more than 600.

Common Mistakes and Tips

1. Expecting Trusted Advisor to see everything on the Basic plan. On Basic you see basic security and service limits. Nothing on cost, performance or fault tolerance, and no API.

2. Obeying the ambers without judgement. Trusted Advisor does not know your context. The root volume of an ephemeral instance does not need snapshots. Every finding is a hypothesis; document the ones you accept and why.

3. Ignoring the service limits category. It is the one that has caused the most outages, and it is one of the few available on the free plan. Always review it.

4. Requesting a quota increase when you are already hitting the limit. Large increases go to a support case and take days. Alarm at 70 % and ask in advance.

5. Forgetting that quotas are per region. Raising it in eu-west-1 does not raise it in eu-central-1. If you have a recovery plan in another region, raise it there too.

6. Not counting the blue/green deployment when calculating vCPUs. It temporarily doubles the instances. It is the most frequent cause of a first zero-downtime deployment failing (08-03).

7. Confusing Trusted Advisor with Config. Config checks your rules and remediates; Trusted Advisor checks AWS's and only points them out. The good cycle is: Trusted Advisor discovers → Config turns it into a rule → the remediation fixes it.

8. Refreshing checks by hand all the time. They refresh themselves every 24 hours. Refreshing manually before a particular review is fine; doing it every hour adds nothing.

9. Buying the Business plan just for Trusted Advisor. Do the sums: if the saving detected does not cover the cost, the reason to buy it is technical support with an SLA, not the checks. Decide it on that basis.

10. Letting findings pile up. A report with 40 permanent findings stops being read. Resolve them, formally accept them with a justification, or take them out of scope. A clean report is a report that somebody looks at.

11. Not reviewing the alarms that produce false positives. An ignored alarm is worse than no alarm. The monthly review includes deleting alarms, not just creating them.

12. Optimising an instance that should not exist. Compute Optimizer recommends shrinking mercadofresco-tienda-01 to t3.small. The right answer is to switch it off. Tools optimise what is there; the question of whether it should be there is asked by a person.

Final tip for the module: observability is never "finished". It is maintained. The system you have built over these five lessons degrades on its own if nobody looks after it: noisy alarms appear, logs without retention, non-compliant rules that get normalised, quotas creeping closer. Marta's monthly hour is what keeps everything else alive. Without it, in six months you have a dashboard nobody looks at and a pile of services billing away.

Exercises

Exercise 1: the support plan decision

MercadoFresco is growing. Current data:

Item Value
Monthly AWS spend 2,400 USD
Daily orders 6,500
Monthly revenue 185,000 EUR
Gross margin 22 %
People on the technical team 3 (Marta, Luis and a new hire)
Out-of-hours on-call Marta, with no formal rota
Outages in the last year 2, of 40 and 95 minutes

Available plans: Basic (0 USD), Developer (29 USD/month or 3 % of spend, whichever is greater), Business (100 USD/month or 3 to 10 % of spend depending on the band, whichever is greater).

Work out the real cost of each plan at that level of spend. Estimate the cost of a 95-minute outage on a Friday at peak time. Decide which plan you would buy, justifying it with numbers and not just with the Trusted Advisor checks. Say what you would do about what the chosen plan does not cover.

Exercise 2: the quota plan for Black Friday

For Black Friday MercadoFresco expects a peak of 5 times the usual traffic for 6 hours. The current situation:

Element Current value
ASG instances 2-4 × t3.large (2 vCPUs)
Capacity per instance 600 orders/hour
Usual Friday peak 900 orders/hour
On-demand vCPU quota 16
Other workloads in the account 8 vCPUs
Lambda concurrency Quota 1,000, current peak ~40
Elastic IPs Quota 5, 5 in use
RDS connections Limit 200, current peak 96
Read replicas 1

On top of that, the team wants to do a blue/green deployment the week before in order to publish the campaign.

Work out: how many instances are needed at the peak, how many vCPUs, which quotas fall short and by how much. Write the increase requests with the values you would ask for and their justification. Design the quota alarms you would set up. Say which other non-AWS limits you would review, and how far in advance you would do each thing.

Exercise 3: the monthly review with real findings

It is the first Monday in October. Marta runs her routine and finds this:

Security

  • Trusted Advisor: 1 red — "Security groups: specific ports unrestricted".
  • Config: 3 non-compliant rules (last month there were 0).
  • validate-logs: a digest file from 14 September is missing.
  • Athena: 2,400 AccessDenied from rol-mercadofresco-tienda on s3:GetObject.

Cost

  • CloudWatch bill: from 44 to 71 USD.
  • 1 available EBS volume of 200 GiB, created on 22 September.
  • Compute Optimizer: the ASG instances marked as under-provisioned.

Reliability

  • vCPU quota at 75 %.
  • The canario-mercadofresco-compra canary has failed 14 times, all between 03:00 and 03:20.

Observability

  • The mercadofresco-pedidos-fallidos alarm has fired 23 times this month; 21 were false positives.
  • One SNS subscription in PendingConfirmation: the new person's email address.

For each of the eleven findings: say what it most probably means, what priority you give it (critical, high, medium, low), what specific action you would take, and which lesson of the course the necessary knowledge comes from. Identify which of these findings are related to each other, which is the part that separates a mechanical review from a useful one.

Solutions

Solution 1

Real cost of each plan with 2,400 USD of monthly spend:

Plan Calculation Real cost
Basic 0 USD
Developer max(29, 3 % of 2,400 = 72) 72 USD/month
Business max(100, 10 % of the first 10,000 = 240) 240 USD/month

Watch out for the Developer and Business calculation: it is the greater of the minimum and the percentage. With 2,400 USD of spend, Business costs 240 USD, not 100.

Cost of a 95-minute outage on a Friday at peak time:

Item Calculation
Monthly revenue 185,000 EUR
Revenue per hour (average) 185,000 / 30 / 24 = 257 EUR/h
Friday peak-time factor ×4
Revenue at peak time ~1,028 EUR/h
95 minutes ~1,628 EUR of lost revenue
Lost margin (22 %) ~358 EUR
Orders never recovered (an estimated 40 % are lost) ~143 EUR of margin
Reputational and customer service cost Not quantifiable, but real

Two outages a year ≈ 700-900 EUR of lost margin, plus the damage to the image of a fresh produce shop, where confidence in delivery within 24 h is the product.

The decision: Business, 240 USD/month. And the justification is not the Trusted Advisor checks:

Argument Weight
1-hour response for production down, 24×7 Decisive
Access to solutions architects to review decisions High
Being able to open technical cases instead of searching forums High
The full Trusted Advisor Medium
The Trusted Advisor API and weekly notifications Medium
Saving found by the cost checks 50-100 USD/month

The number that settles the decision: 240 USD/month is 2,880 USD a year. If support with an SLA cuts the duration of one single outage from 95 to 30 minutes, that saves around 240 EUR of margin. With two outages a year, it does not pay for itself on that alone.

But the team is three people, an informal on-call arrangement and no way of escalating to anybody at 4 in the morning. The real argument is one of risk, not of saving: with 185,000 EUR a month depending on the platform, having nobody to call when something serious breaks is a risk that is wholly disproportionate against 2,880 USD a year. It is the same kind of reasoning that in 04-04 led to not buying Shield Advanced —there the cost was 100 times higher and the risk far lower—, applied in the opposite direction. Reaching a different conclusion with the same method is what shows the method is a good one.

What Business does not cover and you have to keep doing:

Not covered How it is solved
Dedicated technical account manager Enterprise only; not needed at this scale
Guided Well-Architected review Done self-service with the free tool (11-01)
Automatic remediation AWS Config (05-04): Trusted Advisor does not remediate
Your own compliance rules AWS Config with Guard
Intrusion detection GuardDuty, if it is bought
Somebody actually reading the report Marta's monthly routine. No tool replaces this

And a condition for the decision to be a good one: buying Business and changing nothing else would be throwing money away. The decision includes setting up formal on-call rotas among the three people, with the set-alarm-state drill verified for every phone (05-01). AWS support answers within the hour; somebody at MercadoFresco has to be awake to read that answer.

Solution 2

Capacity calculation.

Item Value
Usual Friday peak 900 orders/hour
Black Friday peak (×5) 4,500 orders/hour
Capacity per instance 600 orders/hour
Instances needed 4,500 / 600 = 7.5 → 8
Safety margin (+50 %) 12 instances
vCPUs needed (12 × 2) 24 vCPUs
During the blue/green deployment (×2) 48 vCPUs
Other workloads 8 vCPUs
Total at the worst moment 56 vCPUs
Current quota 16

The 50 % margin is not paranoia: the figure of 600 orders/hour per instance was measured under normal conditions. On Black Friday the average basket is bigger, there are more searches per order and more abandonments with retries. Capacity per instance will go down, not up.

Quotas that fall short:

Quota Current Needed Adjustable?
On-demand vCPU (standard) 16 64 Yes
Elastic IPs 5 (5 in use) 10 Yes
Lambda concurrency 1,000 1,000 (peak ×5 = 200) Enough
RDS instances 40 40 Enough
RDS connections 200 ~480 Not an AWS quota
Target groups per ALB 100 100 Enough

Increase requests:

# vCPU: from 16 to 64. Headroom for the peak, the blue/green and future growth.
aws service-quotas request-service-quota-increase \
  --service-code ec2 --quota-code L-1216C47A --desired-value 64 \
  --profile mercadofresco-dev --region eu-west-1

# Elastic IPs: from 5 to 10. The 2 unassociated ones must be released first.
aws service-quotas request-service-quota-increase \
  --service-code ec2 --quota-code L-0263D0A3 --desired-value 10 \
  --profile mercadofresco-dev --region eu-west-1

Justification for the support case, if it escalates: "A grocery e-commerce shop. Black Friday campaign with an expected peak of 5× the usual traffic for 6 hours on 27 November. We need to scale the auto scaling group to 12 t3.large instances and to perform a prior blue/green deployment that temporarily doubles the capacity. We are requesting 64 vCPUs with headroom."

The most important row in the quota table is the one that is not an AWS quota: the RDS connections. With 12 instances × 40 pool connections = 480 connections, against a max_connections of 200. The database becomes the bottleneck before EC2 does. And that one is not fixed with a request to AWS: it is fixed with architecture.

Other limits that are not AWS quotas and have to be reviewed:

Limit Risk Action
RDS max_connections Critical Connection pooling, RDS Proxy, or a bigger instance
Read replicas High Add a second replica for the searches
Payment gateway limit Critical Warn the provider weeks in advance
Email sending quota (SES) High Order confirmations: sandbox and daily limit
Warehouse and delivery capacity Critical Not a technical problem. 4,500 orders/hour have to be fulfillable
Third-party API limits Medium Couriers, maps, geocoding

The fifth row is the one a technical profile forgets and the one that costs the most: scaling the shop to accept 4,500 orders an hour that the warehouse cannot prepare turns a commercial success into a customer service catastrophe. MercadoFresco's real limit may be in the warehouse, not in AWS.

Quota alarms to set up:

# vCPU at 70% of the quota, adjusting itself if the quota changes
aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-cuota-vcpu-porcentaje \
  --evaluation-periods 2 --datapoints-to-alarm 2 \
  --threshold 70 --comparison-operator GreaterThanThreshold \
  --metrics '[
    {"Id":"usage","MetricStat":{"Metric":{"Namespace":"AWS/Usage",
      "MetricName":"ResourceCount","Dimensions":[
        {"Name":"Service","Value":"EC2"},{"Name":"Resource","Value":"vCPU"},
        {"Name":"Type","Value":"Resource"},{"Name":"Class","Value":"Standard/OnDemand"}]},
      "Period":300,"Stat":"Maximum"},"ReturnData":false},
    {"Id":"quota","Expression":"SERVICE_QUOTA(usage)","ReturnData":false},
    {"Id":"pct","Expression":"100*(usage/quota)","ReturnData":true}
  ]' \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

# RDS connections at 70% of 200: the limit that will really bite
aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-rds-conexiones-criticas \
  --namespace AWS/RDS --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=mercadofresco-pedidos \
  --statistic Maximum --period 60 --evaluation-periods 2 \
  --threshold 140 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

# ASG launch failures: the direct signal of having hit the quota
aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-asg-fallos-lanzamiento \
  --namespace AWS/AutoScaling --metric-name GroupPendingInstances \
  --dimensions Name=AutoScalingGroupName,Value=asg-mercadofresco-tienda \
  --statistic Maximum --period 300 --evaluation-periods 3 --datapoints-to-alarm 3 \
  --threshold 0 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

The third one is particularly clever: instances stuck in Pending for 15 minutes is exactly what you see when launches are failing because of a quota.

Timeline, with realistic notice:

When What
8 weeks before Request the quota increases. The big ones take time
6 weeks before Release the 2 elastic IPs and the orphan volumes
6 weeks before Warn the payment gateway and SES
5 weeks before Set up RDS Proxy or connection pooling; add a second replica
4 weeks before A real load test with 5× the traffic. Verify that it scales to 12
3 weeks before Tune the ASG: maximum 12, a more aggressive scaling policy, warm pool
2 weeks before Blue/green deployment of the campaign. Check that the quota takes it
1 week before Change freeze. Critical fixes only
On the day The mercadofresco-produccion dashboard on screen; two people on call
Afterwards Post-mortem with data, and lowering the quotas is not necessary: they cost nothing

The load test in the fourth row is what turns everything above it into a plan rather than a hope. If you have not tested it, you do not know whether it scales.

Solution 3

Findings, with priority and relationships.

# Finding Priority Probable meaning Action Lesson
1 TA: SG with an unrestricted port Critical Somebody opened a port Identify it with CloudTrail and close it 03-02, 05-03
2 Config: 3 non-compliant rules (were 0) Critical Something changed in September See which rules and since when 05-04
3 A CloudTrail digest file is missing UTTERLY CRITICAL Possible tampering with the audit trail Immediate formal investigation 05-03
4 2,400 AccessDenied from the shop role High A missing permission: something is broken Review pol-mercadofresco-tienda 04-01, 05-03
5 CloudWatch from 44 to 71 USD Medium More log ingestion See which group grew 05-01
6 Orphan 200 GiB EBS volume Medium Instance terminated on 22/9 Check the contents and delete it 02-02
7 ASG under-provisioned High The instances are falling short Review the type and the scaling policy 02-01, 05-05
8 vCPU quota at 75 % High Close to the ceiling Request the increase now 05-05
9 Canary fails 14 times, 03:00-03:20 Medium Maintenance window See what happens at that hour 05-01
10 21 of 23 alarms were false positives High The alarm is badly calibrated Adjust the threshold or redesign it 05-01
11 Subscription in PendingConfirmation High The new person gets no notices Resend it and run a drill 05-01

Finding 3 is the most serious of all and it is worth saying why. A missing digest file means that CloudTrail's integrity chain is broken on 14 September. It may be an AWS delivery problem —it happens, rarely— or it may mean that somebody deleted log files to hide activity. You cannot tell the difference without investigating, and until it is ruled out it has to be treated as a possible security incident:

# 1. Verify the exact scope of what is missing
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/trail-mercadofresco \
  --start-time 2026-09-13T00:00:00Z --end-time 2026-09-16T00:00:00Z \
  --verbose --profile mercadofresco-dev --region eu-west-1

# 2. See whether there are deleted versions in the bucket (versioning is on)
aws s3api list-object-versions \
  --bucket mercadofresco-auditoria-cloudtrail \
  --prefix AWSLogs/111122223333/CloudTrail/eu-west-1/2026/09/14/ \
  --query 'DeleteMarkers[].[Key,LastModified,Owner.DisplayName]' \
  --profile mercadofresco-dev

# 3. Find who touched the bucket or the trail that day
SELECT eventtime, eventname,
       COALESCE(userIdentity.sessionContext.sessionIssuer.userName,
                userIdentity.userName) AS identity,
       sourceipaddress, errorcode
FROM auditoria_mercadofresco.cloudtrail_mercadofresco
WHERE anio='2026' AND mes='09' AND dia IN ('13','14','15')
  AND (eventname IN ('StopLogging','UpdateTrail','DeleteTrail','PutEventSelectors')
       OR json_extract_scalar(requestParameters,'$.bucketName')
          = 'mercadofresco-auditoria-cloudtrail')
ORDER BY eventtime;

If the COMPLIANCE-mode Object Lock from 05-04 was enabled, nobody could have deleted anything and the explanation is an AWS delivery problem: you open a support case and document it. If it was not, there is serious investigating to do. This is the best practical demonstration of why Object Lock mattered.

Relationships between findings, which is the part that makes a review a useful one:

Group A: findings 1, 2 and 3 — probably the same incident.

Three Config rules that went from compliant to non-compliant, an open port, and a hole in the audit trail. That is too much of a coincidence. Config's timeline will say exactly when the rules became non-compliant, and if that date is 14 September, it is no longer a coincidence: it is an incident that has to be reconstructed in full.

aws configservice get-resource-config-history \
  --resource-type AWS::EC2::SecurityGroup \
  --resource-id <sg-id> \
  --earlier-time 2026-09-10T00:00:00Z \
  --later-time 2026-09-20T00:00:00Z \
  --profile mercadofresco-dev --region eu-west-1

Group B: findings 4 and 5 — the same deployment.

2,400 AccessDenied on s3:GetObject means there is a broken feature that no customer has reported. And every failure generates an error line in the log, which largely explains the CloudWatch increase from 44 to 71 USD. A permissions problem showed up as a cost problem. Fixing the permission fixes both.

Group C: findings 7 and 8 — the same capacity problem.

Compute Optimizer says the instances are falling short, and the vCPU quota is at 75 %. They are two sides of the same thing: MercadoFresco has grown and capacity has not followed. And there is a trap: if 7 is solved by scaling to more instances, 8 gets worse. The correct sequence is request the quota increase first, scale afterwards. The other way round, the ASG will try to launch and fail.

Group D: findings 9 and 6 — the maintenance window.

The canary fails between 03:00 and 03:20. What happens at that hour? The RDS maintenance window and the automatic backup window (02-04). During a Multi-AZ failover there are a few seconds of unavailability, and the canary detects it. And this is good news disguised as a problem: the canary is working and it is measuring a real impact that customers suffer too, even if in the small hours it affects few people. Actions: check whether the application retries lost connections properly (the pattern from 07-05), and consider moving the window to an even quieter hour.

The orphan volume from 22 September probably comes from an instance terminated during that same maintenance work.

Group E: findings 10 and 11 — the health of the notification system, and they are more urgent than they look.

  • 21 false positives out of 23 means that nobody looks at mercadofresco-pedidos-fallidos any more. Its fixed threshold of 10 errors in 5 minutes does not survive the growth in traffic: with more orders there are more absolute errors even if the proportion is the same. The fix is the one from solution 1 of 05-01: alarm on the percentage, not on the absolute number, with a metric expression.
  • The pending subscription means the new person has not had a single notice since they joined. If they were on call some night, nobody was on call. The confirmation is resent and a drill with set-alarm-state is run against their phone, exactly the onboarding protocol of 05-01.

Recommended order of action:

Order Action Why first
1 Investigate the missing CloudTrail digest A possible security incident in progress
2 Close the open port Active exposure
3 Confirm the subscription and run the drill Without notices, none of the rest matters
4 Request the vCPU quota increase It takes days; start now
5 Fix the S3 permission Broken feature + cost
6 Recalibrate mercadofresco-pedidos-fallidos An ignored alarm is worse than none
7 Scale the capacity (once the quota is approved) It depends on 4
8 Investigate the canary failures Real but bounded impact
9 Delete the orphan volume Cost, no urgency
10 Review the CloudWatch growth Partly resolved by 5
11 Document the non-compliant Config rules that are accepted Closure

The lesson of the exercise: eleven isolated findings look like a to-do list. Grouped into five related groups, they are five real problems: a possible security incident, a deployment with incomplete permissions, a capacity problem with a trap in the order of operations, a maintenance window with measurable impact, and a notification system that has quietly degraded. That difference is exactly what you get from having the review done by a person with context rather than by an automatic report.

Conclusion

MercadoFresco has the last piece. You know what Trusted Advisor is and how it differs from everything that came before: Config checks your rules, Trusted Advisor checks AWS's; Config answers questions, Trusted Advisor asks them. And you know that the good cycle is to chain them: Trusted Advisor discovers what you had not foreseen, you write a Config rule so it does not come back, and the remediation fixes it on its own. Against Well-Architected (11-01), the distinction is one of altitude: Trusted Advisor looks down, at the resources; Well-Architected looks up, at the decisions.

You know the five categories —cost optimisation, performance, security, fault tolerance and service limits— and what each one checks, with the three states and the warning that an amber is a hypothesis, not an order: the root volumes of the ASG instances do not need snapshots, and formally accepting that finding with its justification is the right answer.

You have the honest table of what MercadoFresco sees with the Basic plan: basic security and service limits, which is no small thing, and nothing on cost, performance, fault tolerance or the API. And you know how to make up for what is missing with what you have already learned: the orphan volumes with an advanced Config query, the elastic IPs with describe-addresses, the idle load balancers with RequestCount, the underutilised instances with Compute Optimizer, Multi-AZ with a Config rule. Almost everything the Business plan would give you, for free. What cannot be reproduced is technical support with an SLA, and that is the real reason why it gets bought.

You have walked through a complete report of MercadoFresco's architecture and found what had been sitting there for months: 3 orphan EBS volumes from the 02-01 tests, 2 unassociated elastic IPs billed precisely for not being used and which were also taking up 2 of the 5 slots in the quota, mercadofresco-tienda-01 at 4 % CPU ever since we built the ASG and which nobody switched off, the same open port 22 that Config had already flagged —two independent tools confirming each other— and the CloudFront compression finding that is fixed with a checkbox.

And you have discovered the quota nobody had looked at: asg-mercadofresco-tienda cannot go beyond four instances because the account's vCPU quota is 16 and other workloads consume 8. The ASG's maximum was not a design decision: it was an invisible limit. A perfect latent failure, which would show up at exactly the worst moment —the Friday when scaling was needed— and would appear in no CloudWatch alarm, no Config rule and no X-Ray trace. You know how to manage Service Quotas: see the usage, request increases in advance because the big ones take days, remember that they are per region, count the blue/green deployment that doubles the instances, and above all alarm at 70 % with SERVICE_QUOTA(), the function that makes the alarm recalibrate itself when the quota changes. Two alarms, 0.20 USD a month, so that you are not left unable to scale on a Friday.

You know the Trusted Advisor API (Business, and only in us-east-1, the same pattern as ACM and the CloudFront web ACLs), exporting the recommendations to CSV, the weekly notifications to the alternate contacts from 01-02, and the integration with EventBridge (07-03) that lets you not just notify but act —automatically disabling an access key that has turned up in a public repository. And you know where Compute Optimizer and Cost Optimization Hub fit, with the warning that sums up the relationship between tools and people: Compute Optimizer recommends shrinking mercadofresco-tienda-01 to t3.small; the right answer is to switch it off. Tools optimise what is there; the question of whether it should be there is asked by a person.

And you have Marta's monthly routine: one hour, on the first Monday, with its checklist for security, cost, reliability, observability and technical debt. Including the point that is most often skipped and matters most: reviewing the alarms that produced false positives and deleting or recalibrating them, because an alarm that gets ignored is worse than no alarm at all.

This closes module 5. MercadoFresco's observability layer is complete: metrics and dashboards with mercadofresco-produccion, MercadoFresco/Tienda and thirteen alarms verified by drill; end-to-end traces with sampling at 100 % on orders and 0 % on the health check; unalterable auditing in trail-mercadofresco with seven years of retention and SQL queries over every API call; continuous compliance with 29 Config rules and automatic remediation that restores a bucket's encryption in eight minutes; and the automatic review that finds what nobody thought to ask about. The five questions from module 4 have answers —the notice arrives, the logs are correlated, the eight-second order was an N+1, the backup was decrypted by Luis testing a restore in the middle of the night, and the encryption restores itself— and all for 62.47 USD a month, 9.9 % of the bill, with five lines of configuration separating that figure from more than 600.

And now, with the house finally under watch, the watching starts to show something uncomfortable. Marta's dashboard has been saying it ever since we built it, and this module's data confirms it from three different angles: the next bottleneck is the database.

mercadofresco-pedidos is a single PostgreSQL instance that does everything, and it does it badly for different reasons in each case. The basket hammers a session table with constant writes of data that has no need whatsoever to be transactional or to live forever. Sara's reports, with their aggregations over months of history, block production when she runs them during working hours —and the read replica only moves the problem, it does not solve it. The catalogue is queried thousands of times a minute to return always the same thing: prices and descriptions that change once a day. And X-Ray itself showed that even after fixing the N+1, the shop's latency is still dominated by database queries.

All of that sits on a single instance because, when we started in 02-04, PostgreSQL was the obvious answer. And it was. But a relational database is not the right tool for every job, and forcing it to be is the most common cause of architectures that do not scale.

In module 6, "Databases", starting with 06-01, "How to choose the right database", we will look at the decision model: relational versus key-value, document, in-memory, time-series and graph, with the real criteria —access pattern, consistency, latency, volume and cost— for choosing between them. After that, DynamoDB for the session table and the basket, Aurora as the natural evolution of PostgreSQL when more is needed, Redshift so that Sara's reports stop touching production, and ElastiCache so that the catalogue is served from memory in microseconds. And all of it with the advantage that, for the first time in the course, every change we make can be measured: we have the metrics, the traces and the dashboards to prove whether a given architecture decision was a good one or not.

© Copyright 2026. All rights reserved