As the previous lesson closed, one uncomfortable sentence was left hanging: the network of vpc-mercadofresco is well designed, but it filters nothing. The local route AWS puts in every route table means any instance in the VPC can try to open a connection to port 5432 of mercadofresco-pedidos, and the public subnet accepts whatever arrives through the internet gateway. Addressing decides where a packet may go; what is missing is deciding whether it may pass.

AWS offers two firewalls for that, and they differ in almost everything: security groups, which protect the network interface of each resource, and network access control lists (NACLs), which protect the perimeter of each subnet. They do not compete: they complement each other, and a packet entering the VPC crosses both, in a specific order worth committing to memory.

In this lesson Marta configures MercadoFresco's complete filtering by applying the pattern that separates a well-built network from a network full of hand-copied IP ranges: referencing security groups across layers. By the end, the orders database will accept connections only from the shop's security group, and not even Marta from her own laptop will be able to reach it directly.

1. The two filtering layers of a VPC
2. Security groups: stateful and allow-only
3. Anatomy of a rule: protocol, port and source
4. The key pattern: referencing security groups across layers
5. MercadoFresco's three layers with their security groups
6. Security group limits and quotas
7. NACLs: stateless, numbered and able to deny
8. The ephemeral-port trap
9. Comparison table: security groups versus NACLs
10. The complete path of a packet
11. Case study: opening port 22 to the office only (and why not to do it)
12. Case study: blocking an abusive IP with a NACL
13. Debugging: which error you see when each thing fails
14. Confirming it with VPC Flow Logs
15. Creation and auditing from the CLI
16. A different layer altogether: identity policies

First of all, the mental map. When a packet enters the VPC from the internet, it goes through this:

• The NACL acts at the subnet boundary. A packet going from one instance to another within the same subnet never crosses it.
• The security group acts on the ENI, that is, right up against the resource. All traffic aimed at an instance crosses it, wherever it comes from.

The practical consequence is that 95 % of the daily work is done with security groups, and NACLs are kept for coarse subnet-wide rules, above all denials, which is the one thing security groups cannot do.

A security group is a virtual firewall attached to an ENI. It has three properties you need to understand properly, because they explain almost all of its behaviour:

1. It only accepts allow rules. There is no "deny" rule. Anything not explicitly permitted is denied by default. This simplifies the reasoning enormously —there is no rule order to think about— but it means that with a security group you cannot block a specific IP: that is what NACLs are for.

2. It is stateful. If you allow an inbound connection, the reply goes back out automatically, even if no outbound rule covers it. And the other way round: if an instance starts an outbound connection, the reply comes in even if there is no inbound rule. The security group remembers established connections.

This is what makes an instance with these rules work perfectly:

A browser connects from ephemeral port 51234 to port 443 on the instance; the reply goes out from 443 to 51234 without needing a rule, because the security group knows that conversation was already open.

3. All rules are evaluated together, with no ordering. If an ENI has three security groups attached, the union of all their rules is evaluated. It is enough for one rule in any of them to allow the traffic. There are no priorities and no "first match wins".

By default, a newly created security group has:

• Inbound: no rules (nothing comes in).
• Outbound: 0.0.0.0/0 on every protocol and port (everything goes out).

Every rule has four fields:

The three possible source types:

About the description: it is optional, but a rule with no description is a rule nobody will dare delete a year from now. Marta imposes the standard that every rule must carry one explaining why it exists, not what it does: "payment provider access", not "port 443 open".

One detail about /32: it is the mask that designates a single IP address. 81.45.20.7/32 means exactly that IP and no other. It is the correct way of expressing "only this machine".

Here is the most important idea in the lesson. Compare two ways of writing the same intent, "the database only accepts connections from the shop's instances":

The naive way, with IP ranges:

The correct way, with a security group reference:

The differences are substantive, not stylistic:

The case that makes it obvious: if Luis launches a test instance in snet-mercadofresco-app-a with a different security group, with the naive approach that instance can connect to the production database, because it sits inside the permitted range. With the group-to-group reference it cannot, because it does not carry sg-mercadofresco-tienda.

How it works underneath: when a rule references sg-mercadofresco-tienda, AWS dynamically resolves the set of private IPs of every ENI carrying that group, and keeps it up to date in real time. When the ASG launches instance number 3 and number 4 on Friday at 17:00, they are authorised the instant they receive their IP.

Two subtleties that catch people out:

• The reference resolves to private IPs. If the traffic arrived via the public IP, the rule would not match. Inside the VPC that does not happen, but it explains why you should always use internal DNS names.
• A group can reference itself (sg-x allows inbound from sg-x). That is the pattern for clusters whose nodes talk to each other, like the one ElastiCache will need in 06-05.

The chain of references is the elegant part: nobody writes a single private IP range. Only the load balancer's group, which is the front door, mentions 0.0.0.0/0.

The complete rules, group by group:

Notice that there is no inbound rule for port 22. No office IP, nothing at all. We come back to that in the case study.

That "no outbound" is deliberate: the default 0.0.0.0/0 outbound rule has to be explicitly deleted. Since the group is stateful, replies to queries still go out anyway.

Numbers worth keeping in mind, because they are reached sooner than you would think:

The real constraint is the last one: rules per group × groups per ENI cannot exceed 1,000. And an accounting detail: a rule with a port range counts as one, but a rule with several CIDRs counts as one per CIDR. A group with "443 from these 40 IPs" consumes 40 rules. That is exactly the scenario customer-managed prefix lists exist for: you define a list with the 40 IPs and reference it with a single rule.

A NACL (Network Access Control List) is a firewall at the subnet level. Each subnet is associated with exactly one NACL; if you do not associate one, it uses the VPC's default NACL, which allows absolutely everything in both directions.

Its properties are almost the opposite of a security group's:

1. It allows and denies. Every rule is either allow or deny. This is the one thing that makes NACLs indispensable.

2. It is stateless. It remembers nothing. If a request comes in through an inbound rule, the reply needs its own outbound rule. This is the source of 90 % of the problems people have with NACLs.

3. Rules are evaluated in numerical order, and the first match wins. Rules are numbered from 1 to 32,766. AWS walks them from lowest to highest and stops at the first one that matches, ignoring everything after it.

4. There is a * rule. At the end of every NACL there is a non-editable rule, numbered *, that denies anything which has not matched earlier. It is the final safety net.

The default NACL of a new VPC:

(and the same on outbound). In other words: by default a NACL filters nothing. That is intentional, and it is why many perfectly correct architectures leave NACLs as they are and trust all filtering to security groups.

Numbering tip: use steps of 100 (100, 200, 300…). When you need to insert a rule between two, you will have room. Renumbering a NACL in production is an unpleasant operation.

This deserves its own section, because it is where everybody comes unstuck.

When a browser connects to the shop, it opens the connection from a high random port —an ephemeral port— to port 443 on the server. The server replies from 443 to that ephemeral port. Since the NACL is stateless, that reply is evaluated against the outbound rules, and its destination port is not 443: it is 51234, or 62890, or whatever it happens to be.

That is why a restrictive NACL always needs an outbound rule opening the ephemeral port range. And the range depends on the client:

AWS's practical recommendation: open 1024–65535 outbound, because you do not know which client you will get. Yes, it is an enormous range, and it is why NACLs are a blunt instrument: filtering return traffic finely is practically impossible.

A working NACL for a public subnet looks like this:

Four of the six rules exist purely because of the lack of state. Compare that with the equivalent security group, which needed one.

A practical rule that sums up the division of labour: security groups define the permitted architecture; NACLs block what has to be blocked.

This is the diagram to memorise. It follows an HTTPS request from a client to the instance and its reply on the way back:

Four checkpoints in total, and the asymmetric one is the third: the security group evaluates nothing on the way out because it remembers the connection, whereas the NACL examines it again as if it were new traffic. The symptom when rule 110 is missing is particularly cruel: the connection is established, the request arrives, the application processes it correctly… and the client sits there waiting until the timeout expires.

Luis asks for SSH access to the instances so he can debug. The "correct by the book" answer would be to open port 22 only to the office's fixed IP:

Never 0.0.0.0/0. A port 22 open to the internet receives automated access attempts within minutes.

But for MercadoFresco not even this is the right solution, for four reasons:

1. The office IP changes. The day the provider renews it, nobody gets in, and somebody will "fix" the problem by putting 0.0.0.0/0 in at eleven at night.
2. It does not work for remote working. Marta at home has a different IP.
3. There is no record of who came in. SSH with a shared key does not identify the person.
4. ASG instances are ephemeral. SSHing into a machine that will be destroyed in two hours to "fix it" by hand is exactly what the ASG is trying to avoid.

The alternative, already used in 02-01, is Session Manager: the SSM agent on the instance opens an outbound connection to the AWS service, and the session travels down that tunnel.

That is why sg-mercadofresco-tienda has no inbound rule for port 22, and that absence is a design decision, not an oversight.

One Tuesday, Sara spots in the logs that the IP 198.51.100.77 is walking the entire catalogue at a rate of 40 requests per second. It is not a distributed attack, it is a single IP scraping prices. Security groups are no help: they cannot deny.

The number 50 is the whole lesson: had we used 150, rule 100 (the general ALLOW) would be evaluated first, it would match, and 150 would never be read. First match wins.

To remove it once the abuse stops:

Two honest warnings:

• Blocking IPs by hand does not scale. If tomorrow there are 300 IPs, this is unworkable and you will need the right tool: AWS WAF, covered in 04-05, with its rate-limit rules. The NACL is the emergency answer, not the solution.
• The NACL blocks traffic as it enters the subnet. If the traffic arrives through CloudFront (lesson 03-04) or a load balancer, the IP the NACL sees is the intermediary's, not the attacker's, and blocking it would shut everybody out.

The symptoms are different, and knowing how to read them saves hours:

The distinction between the first two rows is the most useful thing in the whole table: a firewall drops the packet silently and the client waits; a service that is down replies immediately with a refusal. If telnet mercadofresco-pedidos... 5432 returns "connection refused" instantly, the problem is not in the security group.

The Flow Logs we switched on in 03-01 are the only way to be certain. Recalling the format:

Here 10.0.32.15 (a shop instance) tried to reach 10.0.64.30 (the database) on port 5432, and the result was REJECT. That confirms the block is happening at the network layer.

How to tell which of the two layers did the blocking:

That third row is especially valuable: the absence of records is information. Analysing this data with CloudWatch Logs Insights and its queries is covered in 05-01.

There is a chicken-and-egg problem: sg-mercadofresco-tienda references sg-mercadofresco-alb and vice versa. The solution is to create the empty groups first and the rules afterwards.

Now the rules. --ip-permissions in its long form lets you include the description, something the short form --port / --cidr does not support:

UserIdGroupPairs is the key to the pattern: it is the field that expresses "from this other security group" instead of an IP range.

This is the command Marta runs every Monday. It looks for any security group in the account with an administration or database port open to the whole internet:

A breakdown of the JMESPath expression, which is dense but worth understanding:

• SecurityGroups[? ... ] filters the list of groups.
• IpPermissions[? ... ] looks inside each group's inbound rules.
• The condition combines two things with &&: that the port is one of the sensitive ones (22 SSH, 3389 RDP, 3306 MySQL, 5432 PostgreSQL, 6379 Redis) and that its ranges include a 0.0.0.0/0.
• Backticks (`) mark numeric and string literals in JMESPath, as we saw in 01-05.

If this returns anything, there is a problem to fix today. And a useful companion, looking for orphan groups nobody uses:

And the same check from Python, handy for wiring into a pipeline (module 8):

An important detail in the code: it checks the whole range (first <= p <= last), not just the starting port. A rule saying "TCP 1-65535 from 0.0.0.0/0" does not have FromPort == 22, but it does include 22 and is every bit as dangerous. The simpler CLI command above would miss it.

It is worth closing by clearing up a frequent confusion. Security groups and NACLs control network packets: who may open a TCP connection to which port. They have no opinion whatsoever about who is making the request or what action it is asking for.

When the shop instance requests an object from mercadofresco-catalogo-fotos, there are two independent questions:

1. Is there a path, and is the packet allowed? → routes (03-01), security group, NACL, VPC endpoint.
2. Does that identity have permission to perform s3:GetObject on that object? → IAM policies, the rol-mercadofresco-tienda role, and bucket policies. That is the subject of lesson 04-01, and an AccessDenied from there is never fixed by touching a security group.

Telling these two layers apart saves entire debugging sessions spent in the wrong direction. And protection against application-level attacks —injections, bots, denial of service— is a third, different layer again: Shield (04-04) and WAF (04-05).

Using IP ranges where there should be group references. It works, which is why it spreads. But it opens the door to any future resource in those subnets and forces you to edit rules every time the network changes. If the source is a resource in the VPC, the answer is always UserIdGroupPairs.

Opening port 22 or 3389 to 0.0.0.0/0 "just for a moment, to test". Automated scanners find the port within minutes. If you really must, use a /32 for a specific IP; better still, Session Manager.

Forgetting ephemeral ports in the outbound NACL. The symptom —the connection is established and then hangs— does not point at the NACL at all. If you touch a NACL, open 1024-65535 outbound.

Numbering NACL rules badly. A DENY with a higher number than an ALLOW that already matches is a dead rule. Denials go with low numbers. And number in steps of 100 so you can insert.

Believing a NACL filters traffic between two instances in the same subnet. It does not: the NACL only acts at the subnet boundary. Only security groups see that traffic.

Leaving the 0.0.0.0/0 outbound rule on the database group. It is not an immediate risk, but if the database is compromised, that rule is its route for exfiltrating data. Remove it: the security group's state means replies still go out.

Confusing "timeout" with "connection refused". The first is a firewall, the second is a service that is down. Diagnosing in the wrong direction costs hours.

Not putting a description on rules. A rule with no explanation becomes permanent out of fear. Insist on a description on every rule and review the list every quarter.

The golden tip: draw the diagram of the layers with the security group arrows before writing a single rule. If the arrow is not in the drawing, the rule should not exist.

MercadoFresco needs an internal administration panel on its own instances, which Sara will use from the office. The panel must: receive traffic from the load balancer on port 443, query the mercadofresco-pedidos-lectura read replica on 5432, and write reports to mercadofresco-informes-analitica. It must not be able to reach the primary database.

Design the sg-mercadofresco-panel group and the changes needed in the existing groups, respecting the referencing pattern.

A colleague has configured this NACL on the application subnet and the shop has stopped responding:

Identify all the problems, explain the exact symptom of each and write the corrected NACL.

Write a shell script that, for a given security group, prints a readable report of all its inbound rules distinguishing the source type, flags the dangerous ones, and generates the revoke-security-group-ingress commands needed to remove them without running them.

Solution 1

New group sg-mercadofresco-panel:

Changes to the existing groups: a new group for the replica is needed, sg-mercadofresco-lectura, separate from sg-mercadofresco-basedatos. That is the core of the exercise: if the replica shared a group with the primary, authorising the panel on the replica would also authorise it on the primary, breaking the requirement.

And sg-mercadofresco-basedatos is left untouched: it still accepts only from sg-mercadofresco-tienda and the Lambda, so the panel cannot reach the primary.

On Sara's access from the office: no port is opened for her. She comes in through the same load balancer as the customers, and the office restriction is done by application authentication, or with WAF filtering by IP on /admin (04-05). Adding a rule with the office IP to the panel's group would be going back to the changing-IP problem.

On access to mercadofresco-informes-analitica: no new rule is required beyond the outbound 443, and if the S3 gateway endpoint exists (03-01), it does not even go through the NAT. Permission to write to the bucket is a matter for the IAM role (04-01), not the firewall.

Solution 2

There are three problems:

Problem 1 — the ephemeral port range is missing on outbound. HTTPS requests come in through rule 100 and reach the application, but the reply is addressed to the client's ephemeral port (51234, for example). On outbound, rule 100 only covers 5432 and rule 200 only 443, so the reply falls through to the * DENY. Symptom: the connection is established, the server processes the request and the client hangs until the timeout expires. In the Flow Logs: inbound ACCEPT and outbound REJECT.

Problem 2 — the ephemeral port range is missing on inbound. When the instance queries RDS (going out on 5432, allowed by outbound rule 100), RDS's reply arrives at an ephemeral port on the instance. On inbound only 443 is permitted, so the reply is denied. Symptom: database queries hang and end in a timeout. The same happens with outbound HTTPS calls to S3 or to the payment gateway.

Problem 3 — the inbound source is too restrictive in one sense and not precise enough in the other. 10.0.0.0/16 covers the entire VPC, which is fine if the traffic only comes from the load balancer, but it also lets any subnet in the VPC in on 443, something the NACL should not be deciding. It breaks nothing, but it is a badly framed rule: fine-grained source filtering is the security group's job.

Corrected NACL:

The moral of the exercise: if the fine-grained filtering is already being done by security groups, this NACL could perfectly well have been left on the default blanket ALLOW without losing any real security, and the incident would have been avoided.

Solution 3

Design points of the script: it distinguishes the three source types (CIDR, SG, PL) because a reference to another group is never dangerous in itself; it checks the full port range rather than just the starting port; it uses // 0 and // 65535 in jq for the IpProtocol: "-1" case, where FromPort and ToPort do not exist; and it runs nothing, it only prints the commands, because revoking rules blindly in production is worse than the vulnerability.

MercadoFresco now has firewalls. You know that security groups act on the ENI, are stateful —the reply to an allowed connection goes out on its own, with no rule—, allow only, and are evaluated together with no ordering, and that by default they let nothing in and everything out. You know the anatomy of a rule and, above all, you have mastered the pattern that defines a well-built network: referencing security groups from other security groups with UserIdGroupPairs instead of writing IP ranges. Thanks to that, sg-mercadofresco-basedatos accepts 5432 only from sg-mercadofresco-tienda, and the instances the ASG launches on Friday at 17:00 are authorised the instant they start, with nothing to edit.

You know that NACLs are the opposite in almost everything: they act at the subnet boundary, they are stateless, they are evaluated by number and stop at the first match, they always end in the * deny rule, and they are the only one of the two layers able to deny. You have understood why that forces them to open the ephemeral port range 1024-65535 in the return direction, and why the symptom of forgetting it —the connection is established and then hangs— is so hard to attribute. You have followed the complete path of a packet through the four checkpoints and you know which one of them is asymmetric and why.

In practice you have decided not to open port 22 even to the office IP, with four concrete reasons and Session Manager as the alternative; you have blocked an abusive IP with a DENY rule numbered 50, understanding that the low number is what makes it effective, and knowing that this is an emergency answer and not a solution that scales —WAF, in 04-05, is what that is for. You can tell a timeout (firewall) from a connection refused (service down), and confirm which of the two layers blocked a packet by reading the ACCEPT/REJECT in the VPC Flow Logs. And you have automated the weekly audit that looks for sensitive ports open to 0.0.0.0/0, both in JMESPath and in boto3 checking the full port range. Finally, you are clear that identity policies are a separate layer: an IAM AccessDenied is never fixed by touching a security group, and that is the subject of module 4.

With the network designed and filtered, we can now put in the piece MercadoFresco has been waiting for since the first lesson. The Auto Scaling group asg-mercadofresco-tienda knows how to launch four instances on Friday afternoon, but nothing distributes traffic between them: today they still hang off a single IP. In lesson 03-03, "Elastic Load Balancing", we will build the application load balancer in the public subnets with the sg-mercadofresco-alb group we have just created, connect it to the ASG so instances are registered and deregistered automatically, configure health checks that do not take the application down instead of saving it, and finish solving problem 1: the Friday outages.