In 03-02, when a single IP address started making thousands of requests per minute against the shop, the solution was elegant and took a minute: a DENY rule numbered 50 in the NACL of the public subnets, with a low number so it would be evaluated before the general ALLOW. It worked perfectly.

It worked because it was one address.

If tomorrow the requests arrive from ten thousand addresses spread across sixty countries, that rule is useless. And it is not just that you would have to write ten thousand rules: it is that you cannot. A NACL takes 20 rules by default and a hard maximum of 40. A security group takes 60. The tool that solved the small problem does not scale to the big one, and that is not a defect of the tool: they are simply different problems.

Worse still, there is collateral damage that almost nobody anticipates the first time. The Auto Scaling group asg-mercadofresco-tienda that we set up in 02-01 to solve the Friday outages will do exactly what it was designed to do: see a lot of load and launch instances. A denial-of-service attack against an elastic architecture does not always bring the service down; sometimes it simply turns it into a five-figure bill.

AWS Shield is the protection service against distributed denial-of-service attacks. In this lesson Marta understands what she is protecting herself against, discovers that most of the protection is already enabled and free, and takes a reasoned —and negative— decision about the 3,000 dollars a month of Shield Advanced.

Warning. The content of this lesson is strictly defensive: it describes how to detect and mitigate attacks against your own infrastructure, never how to carry them out. Running load or stress tests against systems that are not yours is illegal, and against your own on AWS it requires following the AWS simulated testing policy. The examples are teaching material: any security configuration, and in particular any incident response plan affecting customer data under the GDPR, must be reviewed by a security professional before being applied to a real environment.

Contents

  1. What a distributed denial-of-service attack is
  2. Why the single-IP rule does not work
  3. Taxonomy: layers 3/4 versus layer 7
  4. Volumetric attacks
  5. Application-layer attacks
  6. The economic damage: when scaling plays against you
  7. AWS Shield Standard: what you already have
  8. AWS Shield Advanced: what it adds
  9. Cost protection against scaling
  10. The incident response team (SRT)
  11. Real cost and an honest assessment for MercadoFresco
  12. Resilient architecture: the first line of defence
  13. Reducing the exposed surface
  14. Caching aggressively and over-provisioning
  15. Metrics and detection
  16. Alarms towards alertas-mercadofresco
  17. Response plan: what Marta looks at first
  18. Telling an attack from a very good Friday
  19. What to do in the heat of the moment and what to document afterwards
  20. Cost, cleanup and what comes next

What a distributed denial-of-service attack is

A denial-of-service (DoS) attack aims to make a service unreachable for its legitimate users by consuming a finite resource: bandwidth, connections, CPU, memory or database connections. Distributed (DDoS) means the traffic comes from many simultaneous sources —typically compromised machines or abused third-party services—, which makes telling them apart and blocking them one by one unworkable.

The difference from other attacks matters:

Intrusion attack Denial-of-service attack
Goal Steal or alter data Make the service stop responding
Stealth Maximum: it wants to go unnoticed None: it wants to be noticed
Defence IAM, encryption, WAF, patches Capacity, filtering, absorption
Duration Months without being detected Minutes or hours
Damage Data breach, GDPR Lost sales, reputation, bill

For MercadoFresco, a two-hour attack on a Friday afternoon means losing around 1,800 orders —900 an hour at the peak— plus the damage of customers who wanted to buy and could not remembering it the following week.

Who launches them: extortion ("pay up or we carry on"), unfair competition, activism, or simply automated background noise sweeping the internet for easy targets. That last category is the most frequent and the reason even a mid-sized shop needs to think about this.

Why the single-IP rule does not work

It is worth seeing the contrast with numbers:

The 03-02 incident Distributed attack
Sources 1 IP address 10,000-1,000,000 addresses
Solution 1 rule in the NACL Impossible with a NACL
Limit of the tool 40 rules per NACL Nowhere near enough
Where it is filtered In the subnet, already inside your VPC It must be filtered earlier, at the edge
Who decides You, by hand, in a minute An automatic system, in seconds

There is also a deeper conceptual problem. Filtering in the NACL means the packet has already reached your VPC: it has consumed your inbound bandwidth and made it to the door. If the attack saturates the link, filtering inside does not help, because the link is already full.

Hence the principle that organises the whole defence against DDoS:

Malicious traffic must be filtered as far away from your infrastructure as possible, ideally in the provider's network, spread across hundreds of points of presence, before it converges on a single destination.

That is exactly what Shield does, and it explains why the protection is associated with CloudFront and Route 53: they are services that live at the edge.

Taxonomy: layers 3/4 versus layer 7

Attacks are classified by the OSI model layer they operate at, and that classification determines the defence:

Layer 3/4 (network and transport) Layer 7 (application)
What it saturates Bandwidth, connection table CPU, memory, database
Volume Very high: Gbps, Mpps Low: it looks like normal traffic
Appearance Malformed packets or floods Valid HTTP requests
Detection Easy: the volume gives it away Hard: it looks like legitimate traffic
Defence Absorption at the edge, signature filtering WAF, rate-based rules, CAPTCHA
AWS service Shield WAF (04-05)

The operational conclusion is the one that structures this module: Shield for the volumetric stuff, WAF for the clever stuff. Neither replaces the other, and the next lesson covers the second.

Volumetric attacks

The layer 3 and 4 ones. They have existed for decades and today they are almost always launched from botnets or by abusing badly configured services on the internet.

Technique How it saturates Symptom in your metrics
UDP flood Sends enormous amounts of UDP to random ports Massive inbound traffic; CPU spent on filtering
Reflection and amplification Spoofs your IP as the source and asks badly configured DNS/NTP/memcached servers to reply; the reply is hundreds of times bigger than the request Enormous volume from legitimate servers' IPs
SYN flood Opens TCP connections halfway and never completes them, exhausting the connection table Many connections in SYN_RECV; the server accepts no new ones
ICMP flood Massive ping High inbound traffic, not very effective today
Fragmentation Fragmented packets that consume resources on reassembly High CPU in the operating system

Reflection amplification is what produces the largest attacks on record: an attacker with little bandwidth can generate hundreds of times that amount against their target. The mitigation is a network one and happens before it reaches you: it is not something you solve on your server.

The good news for MercadoFresco is that AWS absorbs nearly all of this by default. The aggregate capacity of the AWS network and of the CloudFront points of presence is of an order of magnitude no volumetric attack aimed at a mid-sized shop is going to exhaust. Against this family, MercadoFresco is already protected without having done anything.

Application-layer attacks

This is where a company the size of MercadoFresco has a real problem, because these attacks do not need volume.

Technique How it works Why it hurts
HTTP GET flood Thousands of requests per second to ordinary pages Every request consumes CPU and a connection
HTTP POST flood Requests that force writes or processing Even more expensive than GETs
Slowloris Opens many connections and sends the headers very slowly, without closing them Exhausts the server's connections with very little traffic
Attack on search Complex, uncacheable queries against the search engine One request can cost seconds of database time
Credential stuffing Tries leaked user/password pairs against /login As well as load, it is looking for access
Basket abuse Adds and removes products, reserving stock Consumes database and locks up inventory

The specific case that worries Marta most: product search. A request to /buscar?q=tomate&filtros=ecologico,granel&orden=precio is not served from cache, runs a complex query against mercadofresco-pedidos and takes about 200 ms. Fifty requests per second from fifty different addresses —a ridiculous volume, indistinguishable from normal traffic on a bandwidth chart— can saturate the database without a single network alarm batting an eyelid.

That is why Shield alone is not enough and why lesson 04-05 exists.

The economic damage: when scaling plays against you

This section deserves a calculation, because it is the argument that makes a company's management understand the problem.

Suppose a layer 7 attack sustained for 6 hours against MercadoFresco, with 5,000 requests per second reaching the ALB:

Effect Detail Cost
The ASG scales to the maximum From 2 to 20 t3.medium instances for 6 h 18 × 0.0456 × 6 ≈ 4.92 USD
Outbound transfer If the attack asks for uncached content, 500 GB 500 × 0.085 ≈ 42.50 USD
ALB capacity units Thousands of new connections per second Tens of USD
CloudFront requests 108 million requests in 6 h 108 × 0.0075 ≈ 0.81 USD
RDS reads May require scaling the instance up Variable
Lost sales 6 h of a Friday × 900 orders/h × margin By far the most expensive

The infrastructure cost of an attack like that on MercadoFresco is tens or a few hundred dollars: annoying, not catastrophic. The real damage is the lost sales and the trust. This calculation matters because it is what prevents the emotional decision of buying Shield Advanced at 3,000 USD a month to protect against a 200 USD risk.

In bigger architectures the calculation changes radically, and that is why the cost protection we will see two sections from now exists.

AWS Shield Standard: what you already have

Shield Standard is on, it is free and there is nothing to do to have it. It is applied automatically to every AWS customer.

What it includes:

  • Automatic detection and mitigation of the most common volumetric and connection-state attacks at layers 3 and 4, in real time and continuously.
  • Inline mitigation, without diverting the traffic or introducing perceptible latency.
  • Protection built into CloudFront, Route 53, Global Accelerator and Elastic Load Balancing, which are precisely the services where MercadoFresco's edge lives.
  • Network defences applied in the AWS infrastructure: filtering of malformed packets, per-source limits, protection against SYN floods.

What matters is where it acts:

flowchart LR
    A["Internet traffic<br/>legitimate + attack"] --> B["AWS network<br/>Shield Standard"]
    B -->|"malformed packets,<br/>L3/L4 floods"| X["Dropped at the edge"]
    B -->|"clean traffic"| C["CloudFront<br/>E2QWERTY123ABC"]
    C --> D["ALB<br/>alb-mercadofresco-tienda"]
    D --> E["ASG<br/>asg-mercadofresco-tienda"]
    E --> F["RDS<br/>mercadofresco-pedidos"]

The filtering happens before CloudFront, that is, before the traffic reaches anything of yours and before it generates cost. What Shield Standard does not do is tell a legitimate HTTP request from a malicious one: that is layer 7 and it is WAF's job.

A practical check: since MercadoFresco already serves all its traffic through CloudFront (03-04) and resolves DNS with Route 53 (03-05), it is already using Shield Standard in its most effective form without having consciously decided to. That is one of the non-obvious reasons for putting a CDN in front, even when the transfer saving was not the main argument.

AWS Shield Advanced: what it adds

Feature Standard Advanced
Automatic L3/L4 mitigation Yes Yes, more aggressive
Automatic layer 7 protections No Yes, it writes WAF rules on its own
Per-resource specific detection No Yes, it learns your normal traffic pattern
Attack metrics and diagnostics No Yes, AWS/DDoSProtection
Cost protection No Yes, credits for the scaling during the attack
Response team (SRT) No Yes, 24/7
WAF included at no extra cost No Yes, on the protected resources
Cross-account Firewall Manager Paid separately Included
Global event dashboard No Yes
Cost 0 USD 3,000 USD/month + transfer

The three features that genuinely justify the price:

1. Per-resource specific detection. Shield Advanced establishes a baseline of the normal traffic of your application —not of the AWS average— and detects deviations. It can identify a 200 Mbps attack against a resource that normally receives 20 Mbps, a volume that in absolute terms would not catch anybody's attention.

2. Automatic application-layer protections. With this enabled, Shield Advanced writes and applies WAF rules on its own during an attack, based on the detected pattern, and withdraws them when it passes. It is the only way to respond to a layer 7 attack in seconds in the middle of the night with nobody awake.

3. Cost protection, which deserves its own section.

Cost protection against scaling

It is the least known feature and the one that has justified the purchase most often.

During an attack, your elastic infrastructure reacts: the ASG launches instances, CloudFront serves requests, the ALB processes connections, outbound transfer shoots up. All of that is billed.

Shield Advanced offers service credits for the charges attributable to a verified attack on the protected resources:

Service What is covered
CloudFront The attack's outbound transfer and requests
Route 53 The attack's queries
ELB Capacity units consumed
EC2 Instances launched by the scaling during the attack
Global Accelerator Transfer

How it works in practice: you open a support case during or after the attack, AWS verifies there was a DDoS event on that resource in that window, and applies a credit to the bill. It is not automatic: you have to ask for it.

For a company whose bill during an attack can go from 5,000 to 80,000 dollars in one night, this feature alone pays for the service. For MercadoFresco, whose most expensive imaginable attack costs around 200 dollars, it does not.

The incident response team (SRT)

The Shield Response Team is a team of AWS engineers specialising in DDoS, available 24/7 to Shield Advanced customers with a Business or Enterprise support plan.

What it does:

  • During an attack: it investigates with you, writes bespoke mitigations, applies WAF rules in your Web ACLs if you have given it permission.
  • Proactively: if you authorise it, it can step in without waiting for you to call, as soon as its alarms fire. This is what is known as proactive engagement and it has to be enabled with the contacts configured.
  • Beforehand: it reviews your architecture and recommends changes.

The part to weigh up is that giving it access to your Web ACLs means authorising a third party to modify your filtering rules in production without your prior involvement. For many organisations that is exactly what they want at four in the morning; for others it is a governance issue that has to be documented.

Real cost and an honest assessment for MercadoFresco

Item Price
Shield Standard 0 USD
Shield Advanced 3,000 USD a month, with a 12-month commitment
Data transfer from protected resources An additional per-GB rate (CloudFront, ELB, EC2)
Requirement for the proactive SRT Business support (minimum 100 USD/month) or Enterprise
WAF on protected resources Included

That is 36,000 USD a year, plus transfer, plus support. The annual commitment means you cannot buy it "just while the scare lasts".

Let us do the honest assessment for MercadoFresco:

Factor MercadoFresco's situation Does it push towards Advanced?
Annual turnover Spanish SME, a single country No
Cost of a 6 h attack ~200 USD of infrastructure + lost sales No
Exposed surface Everything behind CloudFront, private origins No
Target profile Grocery shop, no political profile No
Contractual or regulatory requirement None No
24/7 on-call team Does not exist Yes, a little
Regulated data Personal data, but DDoS does not expose it Neutral

Conclusion: MercadoFresco should not buy Shield Advanced today. The cost is roughly equivalent to its entire AWS bill multiplied several times over, to protect against a risk whose direct impact is two orders of magnitude smaller.

What it should do, and this is the rest of the lesson:

  1. Make the most of Shield Standard, which it already has, by serving everything through CloudFront.
  2. Design the architecture to absorb, not to resist.
  3. Set up WAF with rate-based rules (04-05), which costs a few dollars a month and covers 90 % of what worries it.
  4. Have alarms and a written plan.

And when it would make sense to reconsider:

  • If MercadoFresco grew to turn over millions and an hour of downtime cost tens of thousands.
  • If it received an extortion attack with a threat to repeat.
  • If a corporate customer or a regulation required it contractually.
  • If it operated in a sector with a high target profile (banking, gambling, media, public sector).

It is a business decision with numbers, not a technical decision. And knowing how to argue the "no" with data is as valuable as knowing how to configure the "yes".

Resilient architecture: the first line of defence

Ahead of any protection service, the best defence against DDoS is an architecture that absorbs. These are the four principles, and MercadoFresco already meets almost all of them:

flowchart TD
    subgraph L1["Layer 1: the edge absorbs"]
        A["Route 53<br/>Shield Standard"] --> B["CloudFront E2QWERTY123ABC<br/>~700 points of presence<br/>Shield Standard + cache"]
    end
    subgraph L2["Layer 2: intelligent filtering"]
        B --> C["AWS WAF<br/>managed + rate-based rules<br/>Covered in 04-05"]
    end
    subgraph L3["Layer 3: minimum surface"]
        C --> D["ALB alb-mercadofresco-tienda<br/>sg-mercadofresco-alb: 443 only"]
        D --> E["ASG asg-mercadofresco-tienda<br/>private subnets app-a/-b<br/>sg accepts the ALB only"]
    end
    subgraph L4["Layer 4: isolated data"]
        E --> F["RDS mercadofresco-pedidos<br/>subnets datos-a/-b<br/>no internet egress"]
        B -.->|"OAC oac-mercadofresco-catalogo"| G["S3 mercadofresco-catalogo-fotos<br/>public access block"]
    end

What makes this architecture strong is not any security product, but four decisions taken back in modules 2 and 3:

Principle How MercadoFresco meets it Lesson
Serve from the edge All traffic comes in through CloudFront 03-04
Reduce the surface S3 closed with OAC, instances in private subnets, RDS with no egress 03-01, 03-04
Over-provision and autoscale ASG from 2 to 20 instances across two AZs 02-01
Cache aggressively Photos and catalogue pages in the CloudFront cache 03-04

Reducing the exposed surface

The rule is simple: what is not exposed cannot be attacked. A review of MercadoFresco's surface:

Resource Exposed? Comment
CloudFront E2QWERTY123ABC Yes, on purpose It is the entry point; it is designed to absorb
alb-mercadofresco-tienda Yes, with public DNS Improvable: see below
ASG instances No Private subnets, no public IP
mercadofresco-pedidos No Data subnets, no route to the internet
mercadofresco-catalogo-fotos Not directly Closed with OAC since 03-04
API endpoint Yes, via CloudFront Protected by WAF in 04-05

The only outstanding improvement is the second row. Even though the public DNS points at CloudFront, the ALB still has a resolvable public DNS name, and an attacker who discovers it can bypass CloudFront and hit it directly, avoiding the cache and the WAF. The standard mitigation has two parts:

  1. Have CloudFront add a secret header to every request towards the origin.
  2. Have the ALB reject any request that does not carry it, through a listener rule.
# 1. CloudFront sends a custom header to the origin 'origen-tienda-alb'
#    (configured in the distribution's origin definition)
#    X-Origen-Verificado: <random value stored in Secrets Manager>

# 2. The ALB listener only lets through the ones that carry it
aws elbv2 create-rule \
  --listener-arn arn:aws:elasticloadbalancing:eu-west-1:111122223333:listener/app/alb-mercadofresco-tienda/50dc6c495c0c9188/abc \
  --priority 1 \
  --conditions '[{"Field":"http-header",
                  "HttpHeaderConfig":{"HttpHeaderName":"X-Origen-Verificado",
                                      "Values":["fictitious-rotatable-value"]}}]' \
  --actions '[{"Type":"forward","TargetGroupArn":"arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/tg-mercadofresco-tienda/abc123"}]' \
  --profile mercadofresco-dev

# 3. The listener's default rule returns 403
aws elbv2 modify-listener \
  --listener-arn arn:aws:elasticloadbalancing:eu-west-1:111122223333:listener/app/alb-mercadofresco-tienda/50dc6c495c0c9188/abc \
  --default-actions '[{"Type":"fixed-response",
                       "FixedResponseConfig":{"StatusCode":"403",
                                              "ContentType":"text/plain",
                                              "MessageBody":"Direct access not allowed"}}]' \
  --profile mercadofresco-dev

That header value is exactly the sort of rotatable secret that goes to Secrets Manager, as we saw in 04-03. And to complete the lockdown you can restrict the inbound rules of sg-mercadofresco-alb to CloudFront's published IP ranges, using the managed prefix list com.amazonaws.global.cloudfront.origin-facing:

aws ec2 describe-managed-prefix-lists \
  --filters "Name=prefix-list-name,Values=com.amazonaws.global.cloudfront.origin-facing" \
  --query 'PrefixLists[0].PrefixListId' --output text --profile mercadofresco-dev

With those two measures, the ALB stops being a reachable target and all the traffic is forced to go through the edge, which is where the protection lives.

Caching aggressively and over-provisioning

Caching. A request served from the CloudFront cache does not reach your infrastructure, does not consume CPU, does not touch the database and does not count towards scaling. With the hit rate we measured in 03-04, the vast majority of the catalogue traffic stays at the edge. Specific recommendations against DDoS:

  • Cache error responses too (404, 403) for a few seconds. Otherwise an attack asking for non-existent routes is an attack that reaches the origin in full.
  • Define a sensible minimum TTL so an attacker cannot invalidate the cache by adding random query parameters: the cache key must include only the parameters that really change the response, as we configured with the cache policies.
  • Serve a static maintenance page from S3 as the Route 53 failover target, already prepared in 03-05: if everything fails, customers see something.

Over-provisioning. Absorbing is cheaper than falling over:

  • Make the ASG's minimum capacity more than strictly necessary. MercadoFresco has 2 instances for 900 orders/hour when one serves 600.
  • Make the scaling react fast: a short warm-up period and low thresholds.
  • Give the maximum a deliberate ceiling. This point is counter-intuitive but important: an ASG with a maximum of 100 during an attack is a bill. A maximum of 20 means the service degrades, but in a bounded and predictable way. Choose the number deliberately, do not leave it at the default.
  • Use instance types with guaranteed network performance if the traffic is high: the families with network credits can be throttled exactly when they are needed most.

Metrics and detection

With Shield Advanced you get the AWS/DDoSProtection namespace in CloudWatch:

Metric What it measures
DDoSDetected 1 if there is an attack in progress against the resource, 0 if not
DDoSAttackBitsPerSecond Volume of the attack in bits per second (L3/L4)
DDoSAttackPacketsPerSecond Packets per second
DDoSAttackRequestsPerSecond Requests per second (layer 7)

Without Shield Advanced —MercadoFresco's case— those metrics do not exist and you have to detect through indirect signals. These are the ones Marta watches:

Metric Service Alarm signal
RequestCount ALB A sudden 5× rise over the hourly average
TargetResponseTime ALB It rises while RequestCount rises
HTTPCode_ELB_5XX_Count ALB Errors from the balancer itself: saturation
HTTPCode_Target_5XX_Count ALB The targets cannot keep up
ActiveConnectionCount ALB Many connections with few requests: slowloris
NewConnectionCount ALB An anomalous rate of new connections
CPUUtilization ASG At 100 % on every instance at once
GroupInServiceInstances ASG Scaled to the maximum outside the usual hours
DatabaseConnections RDS Close to the parameter group's limit
Requests and 4xxErrorRate CloudFront A spike with a low cache hit rate
CacheHitRate CloudFront A sharp drop: somebody is dodging the cache

That last row is especially revealing. A legitimate traffic spike maintains or improves the hit rate, because a lot of people ask for the same popular pages. An attack designed to do damage asks for random routes or ones with unique parameters, and sinks the hit rate. A chart of CacheHitRate falling while Requests rises is one of the cleanest signals there is.

Alarms towards alertas-mercadofresco

These are the alarms MercadoFresco sets up today, without Shield Advanced. CloudWatch is covered in depth in 05-01; here the mechanics are enough:

# 1. Anomalous request spike on the ALB
aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-alb-peticiones-anomalas \
  --alarm-description "Possible attack: requests far above normal" \
  --namespace AWS/ApplicationELB \
  --metric-name RequestCount \
  --dimensions Name=LoadBalancer,Value=app/alb-mercadofresco-tienda/50dc6c495c0c9188 \
  --statistic Sum --period 60 --evaluation-periods 3 \
  --threshold 60000 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --treat-missing-data notBreaching \
  --profile mercadofresco-dev

# 2. Cache hit rate drop: somebody is dodging CloudFront
#    (CloudFront metric: always in us-east-1)
aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-cdn-aciertos-bajos \
  --alarm-description "The hit rate is falling: possible attack with random routes" \
  --namespace AWS/CloudFront --metric-name CacheHitRate \
  --dimensions Name=DistributionId,Value=E2QWERTY123ABC Name=Region,Value=Global \
  --statistic Average --period 300 --evaluation-periods 2 \
  --threshold 50 --comparison-operator LessThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --region us-east-1 --profile mercadofresco-dev

# 3. Scaled to the maximum: the sign that something is very wrong (or very right)
aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-asg-al-maximo \
  --alarm-description "The ASG has reached 18 instances or more" \
  --namespace AWS/AutoScaling --metric-name GroupInServiceInstances \
  --dimensions Name=AutoScalingGroupName,Value=asg-mercadofresco-tienda \
  --statistic Maximum --period 300 --evaluation-periods 1 \
  --threshold 18 --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev

Two details from the field:

  • Alarm 2 is created in us-east-1, because CloudFront always publishes its metrics there, exactly as happened with the ACM certificates in 03-04.
  • --treat-missing-data notBreaching avoids false alarms in the small hours when there is no traffic.

And one alarm that is not a security one but saves the bill: the budget with notifications we created in 01-02. If daily spend shoots up, somebody finds out even if nobody is watching the panels.

Response plan: what Marta looks at first

A response plan exists so that you do not have to think at three in the morning. This is MercadoFresco's, in order:

flowchart TD
    A["Alarm on alertas-mercadofresco"] --> B["1. Confirm: does the site<br/>work from outside?"]
    B --> C["2. Dashboard: RequestCount,<br/>CacheHitRate, 5XX, CPU"]
    C --> D{"3. Attack or<br/>legitimate spike?"}
    D -->|"Legitimate spike"| E["Raise the ASG maximum,<br/>tell the business, enjoy"]
    D -->|"Attack"| F["4. Which layer?<br/>High volume = L3/L4<br/>Normal volume = L7"]
    F -->|"L3/L4"| G["Shield Standard is already acting.<br/>Check CloudFront and wait"]
    F -->|"L7"| H["5. Identify the pattern:<br/>routes, countries, user agents"]
    H --> I["6. Apply WAF rules<br/>in Block mode (04-05)"]
    I --> J["7. Monitor the effect"]
    J --> K["8. Withdraw the temporary<br/>rules when it is over"]
    K --> L["9. Written post-mortem"]

Step 1: confirm from outside. Before anything else, check the problem is real and not a monitoring failure:

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://mercadofresco.example/salud
dig +short mercadofresco.example

Step 2: the four charts. The ALB's RequestCount, CloudFront's CacheHitRate, HTTPCode_Target_5XX_Count and the ASG's CPUUtilization. With those four you almost always know what is going on.

Step 5: identify the pattern. The access logs we store in mercadofresco-registros-web since 03-03 are the source:

aws s3 sync s3://mercadofresco-registros-web/alb/2026/08/02/ /tmp/registros/ \
  --profile mercadofresco-dev

# The 20 IPs with the most requests
zcat /tmp/registros/*.gz | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

# The most requested routes
zcat /tmp/registros/*.gz | awk '{print $13}' | sort | uniq -c | sort -rn | head -20

# The most frequent user agents
zcat /tmp/registros/*.gz | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -10

That analysis produces the three answers you need in order to write a WAF rule: where it comes from, what it asks for and what identifies it.

Telling an attack from a very good Friday

This is the hard part, and the one that separates a professional response from a self-inflicted blackout. MercadoFresco has legitimate, predictable traffic peaks on Friday afternoons. Blocking real traffic on a Friday at 19:00 is worse than the attack.

Signal Legitimate spike Attack
Timing Friday afternoon, campaigns, public holidays Any time, typically the small hours
Curve Rises over minutes or hours Rises in seconds, vertically
Cache hit rate It holds or improves It collapses
Geographic distribution Spain, a little Portugal Countries with no customers, very spread out
Routes requested Home page, categories, popular product pages Odd, random routes, always the same face
Conversion to order Normal, ~2 % Close to zero
User agents Real browsers, varied A few repeated values, or absent
Referrer Search engines, social, direct Empty or spoofed
Requests/session ratio 10-30 pages Thousands from the same IP
Effect on the database Rises proportionally Shoots up with no orders to justify it

The most reliable signal of all is conversion. If 5,000 requests per second arrive and the number of orders per hour is still 900, they are not customers. A legitimate spike moves both metrics at once; an attack only moves one. Marta has that business metric published in the MercadoFresco/Tienda namespace thanks to the cloudwatch:PutMetricData permission we granted in 04-01, and it is the one she looks at before deciding to block anything.

And the operational golden rule:

When in doubt, start by counting, not blocking. WAF's Count mode lets you see who a rule would affect before applying it. It is the central topic of 04-05 and the most expensive mistake in this discipline is skipping it.

What to do in the heat of the moment and what to document afterwards

Measures in the heat of the moment, from least to most intrusive:

Measure Impact on customers When
Raise the ASG maximum None; cost Whenever the load is absorbable
Increase the cache TTLs Slightly older content Immediate, very effective
WAF rule in Count on the pattern None Always first
Rate-based rule in Block Blocks whoever exceeds the threshold If the pattern is clear
Geolocation blocking Blocks entire countries If you have no customers there
CAPTCHA or Challenge on expensive routes Friction for humans An alternative to blocking
Static maintenance page Degraded service Last resort

Never: switch off CloudFront or point the DNS at the origin. That removes the only layer protecting you and turns an incident into a total outage.

Post-mortem. When it is over, you write it up. No culprits, with data:

  1. Timeline with exact times: first sign, first alarm, first action, mitigation, recovery.
  2. Characterisation: peak volume, number of sources, countries, target routes, layer.
  3. Impact: minutes of degradation, estimated lost orders, extra infrastructure cost.
  4. What worked and what did not: did the right alarm fire? in time? was anybody there?
  5. Concrete actions with an owner and a date: WAF rules that become permanent, new alarms, adjusted thresholds.
  6. Credit request if you had Shield Advanced.

Point 5 is what turns an incident into an improvement. A post-mortem with no dated actions is a document nobody will ever read again.

Cost, cleanup and what comes next

Item Cost for MercadoFresco
Shield Standard 0 USD — already on
Shield Advanced 3,000 USD/month — not purchased, reasoned decision
CloudWatch alarms 3 × 0.10 USD = 0.30 USD/month
Storage of the ALB logs in S3 Already accounted for in 03-03
ALB listener rule No extra cost
Total for this lesson 0.30 USD/month

If you had enabled Shield Advanced to try it out, bear firmly in mind that the commitment is 12 months: it is not cancelled just like that. It is managed from the Shield console or with aws shield disassociate-drt-role and the subscription cancellation process, which requires opening a support case.

To undo what was created in this lesson:

aws cloudwatch delete-alarms \
  --alarm-names mercadofresco-alb-peticiones-anomalas mercadofresco-asg-al-maximo \
  --profile mercadofresco-dev

aws cloudwatch delete-alarms --alarm-names mercadofresco-cdn-aciertos-bajos \
  --region us-east-1 --profile mercadofresco-dev

The listener rule with the verified header is not deleted: it is a permanent improvement to the security posture and it should stay.

Common Mistakes and Tips

Believing Shield Standard has to be enabled. It is already on, in every account, free. What is your decision is making the most of it, by serving the traffic through CloudFront and Route 53 instead of exposing the origin directly.

Thinking Shield protects against everything. Shield is layers 3 and 4. Layer 7 attacks —the ones that really threaten a shop— are filtered by WAF. Buying Shield Advanced and not configuring WAF is spending 3,000 dollars and staying exposed to what is most likely to happen to you.

Leaving the ALB directly reachable. If the balancer's DNS name is resolvable and it accepts traffic from any source, an attacker can bypass CloudFront, the cache and the WAF in one hop. The verified header plus the CloudFront prefix list close that door.

Leaving the ASG maximum at a very high number "just in case". During an attack you will scale without limit and the result will be a bill, not an available service. Set a deliberate ceiling.

Blocking by IP during a distributed attack. It is exhausting, useless, and you end up blocking real customers sharing a corporate NAT. Block by behaviour pattern, not by source.

Blocking without having counted first. The most expensive mistake. A badly calibrated Block rule on a Friday at 19:00 does more damage than the attack. Count first, always.

Confusing a legitimate spike with an attack. Look at the conversion to order and the cache hit rate before anything else. If the orders rise with the traffic, it is business, not an attack.

Switching off CloudFront "to rule it out". It removes the protection and exposes the origin. Never.

Tip: rehearse the plan. A quarterly half-hour drill —"the alarm goes off, what do you do?"— is worth more than a perfect document nobody has read. And check that the notifications from alertas-mercadofresco really do reach a phone, not just an inbox nobody looks at overnight.

Tip: keep a baseline. Have the normal traffic per hour and day of the week to hand. Without a baseline you cannot say whether 5,000 requests per second is a lot.

Tip: separate the expensive routes. Search, /login and /api/pedidos in different target groups —tg-mercadofresco-api has existed since 03-03— lets you protect them with specific rules and stops their saturation dragging the rest of the catalogue down with them.

Exercises

Exercise 1: deciding about Shield Advanced with numbers

A concert ticketing company, with an architecture identical to MercadoFresco's, turns over 40 million euros a year concentrated in the hours following each on-sale. An hour of downtime during an on-sale costs around 350,000 euros. Last year it suffered two layer 7 attacks that degraded the service for 40 minutes each, and it received an extortion email threatening to repeat them. It has no night-time on-call team.

Assess whether it should buy Shield Advanced. Structure the answer as: annual cost of the service, expected loss without it, specific features that add value in this case, additional requirements that have to be budgeted for, and the decision with its justification.

Exercise 2: characterising an incident

On a Tuesday at 04:12 MercadoFresco's alarms fire. The data for the first ten minutes:

  • The ALB's RequestCount: from 200/min to 45,000/min in 30 seconds.
  • CloudFront's CacheHitRate: from 89 % to 11 %.
  • The ASG's CPUUtilization: 97 % on both instances; the ASG scales to 12.
  • RDS's DatabaseConnections: from 25 to 190 (the limit is 200).
  • Metric MercadoFresco/Tienda/PedidosPorHora: 3 (normal at that hour is 5-10).
  • ALB logs: 8,400 distinct IPs, 61 countries, 78 % of them asking for /buscar?q=<random string>&pagina=<random number>.
  • User agent: 91 % declare Mozilla/5.0 (compatible; Baiduspider/2.0).

Answer: (a) attack or legitimate spike, and which three signals justify it?; (b) which layer and which technique?; (c) why is it especially effective against this architecture?; (d) which three measures would you apply, in order, and what is the risk of each?; (e) what would Shield Advanced have done that cannot be done without it?

Exercise 3: closing off direct access to the origin

Write the complete procedure, with commands, to stop anybody hitting alb-mercadofresco-tienda while bypassing CloudFront. It must cover: where the secret value is stored, how CloudFront sends it, how the ALB verifies it, what happens to requests that do not carry it, how it is further reinforced with security groups, and how the value is rotated without causing an outage.

Solutions

Solution 1

Annual cost of the service:

Item Amount
Shield Advanced (3,000 USD × 12, annual commitment) 36,000 USD
Business support (needed for the proactive SRT) from 1,200 USD
Data transfer from protected resources Variable, a few thousand
Approximate total ~40,000 USD/year

Expected loss without it: two 40-minute incidents last year × 350,000 €/hour × 0.67 h ≈ 470,000 € a year in direct loss, without counting reputation or the fact that there is an explicit threat to repeat, which raises the future probability.

Features that add value in this specific case:

  1. Automatic layer 7 protections. There is no night-time on-call and the attacks happen in the sales windows, which can be in the small hours. Shield Advanced writes and applies rules on its own, in seconds. It is the decisive feature here.
  2. SRT with proactive engagement. It literally makes up for the absence of an on-call team.
  3. Cost protection. With that concentration of traffic, the scaling during an attack can generate charges of tens of thousands.
  4. Per-resource detection. This company's traffic is extremely spiky by nature; a generic baseline would not tell an attack from an on-sale. Only the resource-specific baseline can.

Additional requirements to budget for: the Business or Enterprise support plan, the work of configuring Web ACLs and permissions for the SRT, defining emergency contacts and a rehearsal of the procedure.

Decision: buy it, without hesitation. 40,000 USD against an exposure of 470,000 € a year is a ratio of more than ten to one, with an explicit threat to repeat and no internal capacity for a night-time response. It is exactly the profile the product exists for. And it is worth underlining the contrast with MercadoFresco: the same service, the same architecture and the opposite decision, because the deciding variable is not technical but the cost of a minute of downtime.

Solution 2

(a) An attack, with three conclusive signals:

  1. Conversion has collapsed in relative terms. 45,000 requests per minute produce 3 orders an hour, when normal at that hour with 200 requests a minute is 5-10. If they were real customers, the orders would have risen with the traffic.
  2. The cache hit rate collapses from 89 % to 11 %. A legitimate spike asks for popular content and holds or improves it. Here unique URLs are being requested on purpose.
  3. The curve is vertical: from 200 to 45,000 in 30 seconds, and at 04:12 on a Tuesday, which is the absolute traffic trough of a Spanish grocery shop.

As additional confirmation: 8,400 IPs in 61 countries do not correspond to MercadoFresco's customer geography, and 91 % of identical user agents declaring themselves a Chinese crawler is an obvious forgery —real crawlers identify themselves and respect robots.txt, and none of them generates 45,000 requests a minute against a search.

(b) Layer 7, an HTTP GET flood aimed at the search endpoint with random parameters. The volume in bits per second is modest; the damage does not come from the bandwidth.

(c) It is especially effective for three chained reasons that attack precisely the strong points of the architecture:

  1. It nullifies the CloudFront cache. Each q=<random string> generates a different cache key, so 100 % of the requests are cache misses and reach the origin. The CDN, which is the first line of defence, is neutralised.
  2. It attacks the most expensive route. Search is not cached, runs a complex query and touches the database on every request.
  3. It saturates the resource that does not scale. The ASG scales to 12 instances, but each instance opens connections to mercadofresco-pedidos, which has a limit of 200. The scaling makes the problem worse: more instances mean more connections against a database that does not grow. At 190 of 200 connections, the next step is the whole shop being unable to query anything.

(d) Three measures, in order:

  1. Rate-based WAF rule on /buscar, in Count for 2-3 minutes. Risk: none, it blocks nothing; it only costs those minutes. It serves to confirm how many legitimate requests would fall.
  2. Move the rule to Block with a per-IP threshold, and add a block on the spoofed user agent. Risk: low, because at 04:12 legitimate traffic is minimal and the pattern is very well characterised. A legitimate crawler getting blocked does no real damage at that hour.
  3. Increase the minimum TTL of search responses and cache the errors. Risk: slightly stale search results for a few hours, acceptable. As a fourth measure, if the database is still at its limit, divert the reads to mercadofresco-pedidos-lectura.

What you must not do: block the 8,400 IPs one by one —they will come back with others—, or turn off search entirely, or point the DNS at the origin.

(e) Shield Advanced would have done three things that are impossible without it: detect the anomaly against this resource's specific baseline rather than against a fixed threshold; automatically apply WAF rules at 04:12 without Marta waking up, which is exactly the window in which it happened; and provide the DDoSDetected and DDoSAttackRequestsPerSecond metrics that characterise the attack without having to download and analyse gigabytes of logs by hand. On top of that, it would allow a credit request for the cost of scaling to 12 instances.

Solution 3

Complete procedure.

1. Generate and store the secret value. It goes to Secrets Manager (04-03), not to a file or the console:

aws secretsmanager create-secret \
  --name "mercadofresco/produccion/cdn/cabecera-origen" \
  --description "Value of the X-Origen-Verificado header between CloudFront and the ALB" \
  --kms-key-id alias/mercadofresco-datos \
  --secret-string "$(openssl rand -hex 32)" \
  --tags Key=Proyecto,Value=mercadofresco Key=Componente,Value=cdn \
  --profile mercadofresco-dev

2. Have CloudFront send it. In the definition of the origen-tienda-alb origin of distribution E2QWERTY123ABC you add a custom header X-Origen-Verificado with that value. It is sent on every request to the origin and it is not visible to the client.

3. Have the ALB verify it. A priority 1 rule on the HTTPS listener that forwards to the target group only if the header matches:

aws elbv2 create-rule \
  --listener-arn <listener-443-arn> \
  --priority 1 \
  --conditions '[{"Field":"http-header",
                  "HttpHeaderConfig":{"HttpHeaderName":"X-Origen-Verificado",
                                      "Values":["<current-value>"]}}]' \
  --actions '[{"Type":"forward","TargetGroupArn":"<tg-mercadofresco-tienda-arn>"}]' \
  --profile mercadofresco-dev

4. What happens to the rest. The listener's default action becomes a fixed 403 response, so that any request without the header —that is, any request not coming from CloudFront— is rejected at the balancer, without reaching the instances and without consuming anything:

aws elbv2 modify-listener --listener-arn <listener-443-arn> \
  --default-actions '[{"Type":"fixed-response",
                       "FixedResponseConfig":{"StatusCode":"403","ContentType":"text/plain",
                                              "MessageBody":"Direct access not allowed"}}]' \
  --profile mercadofresco-dev

5. Reinforcement with security groups. You restrict the inbound rule of sg-mercadofresco-alb on port 443 to CloudFront's managed prefix list, instead of 0.0.0.0/0:

LIST=$(aws ec2 describe-managed-prefix-lists \
  --filters "Name=prefix-list-name,Values=com.amazonaws.global.cloudfront.origin-facing" \
  --query 'PrefixLists[0].PrefixListId' --output text --profile mercadofresco-dev)

aws ec2 authorize-security-group-ingress \
  --group-id sg-mercadofresco-alb \
  --ip-permissions "IpProtocol=tcp,FromPort=443,ToPort=443,PrefixListIds=[{PrefixListId=$LIST}]" \
  --profile mercadofresco-dev

aws ec2 revoke-security-group-ingress \
  --group-id sg-mercadofresco-alb --protocol tcp --port 443 --cidr 0.0.0.0/0 \
  --profile mercadofresco-dev

They are two independent layers: even if somebody worked out the header value, they would also have to originate the traffic from a CloudFront range.

6. Rotation without an outage. The key is to accept two values simultaneously during the transition, exactly the same principle as AWSCURRENT/AWSPENDING from 04-03:

  1. Generate the new value and store it in the secret as the AWSPENDING version.
  2. Change the ALB rule so it accepts both values in the Values array.
  3. Update the CloudFront origin with the new value and wait for the distribution to finish deploying to every point of presence (a few minutes).
  4. Verify in the logs that no request with the old value is arriving any more.
  5. Remove the old value from the ALB rule and promote the version to AWSCURRENT.

Reversing the order —changing CloudFront before the ALB accepts the new value— causes a total service outage for the whole deployment of the distribution. It is the classic mistake in this operation, and the reason it is done on a Tuesday morning and not a Friday afternoon.

Conclusion

You now know what you are protecting yourself against. A distributed denial-of-service attack is not out to steal anything: it is out to exhaust a finite resource, and that is why the defence is not a permission or a key but capacity, distance and filtering. You understand why the DENY rule numbered 50 from 03-02 worked against one IP and is useless against ten thousand: a NACL takes 40 entries, and filtering inside your VPC means the packet has already consumed your bandwidth. Malicious traffic has to be dropped at the edge, far away.

You can tell the two families and their defences apart: the volumetric attacks at layers 3 and 4 —UDP flood, reflection and amplification, SYN flood, fragmentation—, measured in gigabits and absorbed by the AWS network; and the layer 7 attacks —HTTP flood, slowloris, search abuse, credential stuffing—, which arrive with ridiculous volume, look like legitimate traffic and are the ones that really threaten a shop like MercadoFresco. Shield for the first, WAF for the second.

You know that Shield Standard is already on and free, that it acts on CloudFront, Route 53 and ELB, and that MercadoFresco has been using it since 03-04 without having decided to: putting a CDN in front was a security decision too. And you know what Shield Advanced adds —per-resource detection, automatic layer 7 protections, the DDoSDetected and DDoSAttackBitsPerSecond metrics, cost protection against scaling and access to the 24/7 SRT— along with its real price of 3,000 USD a month with an annual commitment. You have done the assessment with numbers and the answer for MercadoFresco is no: 36,000 dollars a year to cover an infrastructure risk of two hundred. And you know exactly what would have to change for the answer to be the opposite, because the ticketing company exercise is the same architecture with the opposite decision.

What you have done is what genuinely protects a company of this size: architecture. Serving everything from the edge, reducing the surface by closing off direct access to the ALB with a verified header stored in Secrets Manager and CloudFront's prefix list on sg-mercadofresco-alb, caching aggressively including the errors, and over-provisioning with a deliberate ceiling on the ASG so the scaling does not become the attacker's bill. You have set up three alarms towards alertas-mercadofresco for 0.30 USD a month, knowing the CloudFront one goes in us-east-1. And you have a response plan with a clear order and, above all, the table that tells an attack from a very good Friday: the conversion to order and the cache hit rate ahead of anything else.

But the plan always ends at the same point. When Marta identifies the pattern —those routes, those countries, that spoofed user agent, that request rate per IP— she needs a tool that looks inside the HTTP request and decides based on its content, not on its source. Neither Shield nor a security group can do that: one works with packets and the other with IPs and ports. In lesson 04-05, "AWS WAF", we will close the module with that tool: the Web ACL associated in front of the distribution E2QWERTY123ABC and of alb-mercadofresco-tienda, the AWS managed rule groups, the match statements and the text transformations that prevent trivial evasions, the rate-based rules to protect /login and /api/pedidos, and —above all— the deployment method that avoids disaster: start in Count, read the logs, adjust the exceptions and only then move to Block.

© Copyright 2026. All rights reserved