The previous lesson ended with a decision still pending. EFS solved the problem of Friday's four instances sharing /var/www/fotos, but at the wrong price: some thirteen times more expensive than the alternative, with all the image traffic going through the instances and without taking advantage of AWS's global scale. That alternative is Amazon S3 (Simple Storage Service), and this lesson is where MercadoFresco makes the final decision about its product photos.

S3 is probably the most important service in AWS. It was the first one launched, back in 2006, and today it is where the data of almost any architecture ends up: static files, backups, logs, data lakes, deployment artefacts. Understanding it properly — not just "it is where you upload files" — changes the way you design systems, because S3 is not a disk: it is an object store with an API, and that difference has consequences everywhere.

1. Object storage: bucket, object, key and prefix
2. Why S3 is not a file system
3. Bucket names, regions and the global namespace
4. Durability and availability: the eleven nines
5. Storage classes and how to choose
6. Lifecycle rules for the old catalogue photos
7. Versioning and protection against accidental deletion
8. Migrating /var/www/fotos with aws s3 sync
9. aws s3 versus aws s3api
10. Permissions: Block Public Access, bucket policies and ACLs
11. Pre-signed URLs: Sara's reports without credentials
12. Encryption at rest and in transit
13. Hosting a static site
14. S3 events: the thumbnail trigger
15. Multipart upload and Transfer Acceleration
16. S3 costs and how to avoid surprises

The S3 model has only three pieces, and it is worth naming them precisely because that is how all the documentation uses them.

An object can weigh anything from 0 bytes to 5 TiB. A bucket can hold an unlimited number of objects and grow without anyone provisioning anything.

This is the idea you have to internalise, because almost every surprise that catches beginners out follows from it.

There are no folders in S3. The console draws them, aws s3 ls lists them, but they are not there. All that exists is a flat list of keys, and the slash / is just one more character in the name.

The practical consequences of this are concrete:

And one enormous benefit: because there is no real hierarchical structure, there is no limit to scale. A bucket with a billion objects works just as well as one with ten, and S3 supports at least 3,500 writes and 5,500 reads per second per prefix, with practically unlimited parallelisation across several prefixes.

That is why key design matters. For MercadoFresco:

A bucket name is unique across the whole of AWS, in every account and every region in the world. If somebody in Australia has fotos, you cannot have it. It is the only AWS resource with a truly global namespace.

Naming rules:

• Between 3 and 63 characters.
• Lower case letters, numbers, hyphens and dots only.
• It must start and end with a letter or a number.
• It cannot look like an IP address.
• Avoid dots: they break TLS certificate validation with virtual-hosted-style names and cause errors that are hard to diagnose.

MercadoFresco's convention, which we will apply throughout:

Even though the name is global, the data reside in one specific region and never leave it unless you explicitly ask them to. For MercadoFresco this is not a minor detail: the photos and the reports stay in eu-west-1, inside the EU, which is what sustains the GDPR argument we went through in lesson 01-03.

The LocationConstraint trap. In us-east-1 (and only there) that parameter must not be passed: the command fails if you include it. In any other region it is mandatory. It is a historical legacy of S3 having been born in Virginia.

S3 Standard promises 99.999999999 % annual durability: the famous eleven nines. The figure sounds like marketing, but it has a precise operational meaning.

If you store 10 million objects, the statistical expectation is to lose one every 10,000 years. Put another way: it is far more likely that you lose the data through an accidental deletion or a badly written policy than through an S3 failure.

How it is achieved: every object is automatically replicated across at least three Availability Zones in the region, with continuous integrity checking and automatic repair. You configure nothing.

Two different metrics must not be confused:

One nuance that matters a great deal for MercadoFresco: S3's durability does not protect you from yourself. If Luis runs an aws s3 rm --recursive on the wrong bucket, S3 will delete it with eleven nines of reliability. The protection against that is versioning, which we cover below.

Not all data are accessed in the same way. The photos of seasonal tomatoes are requested a thousand times a day in June and not once in January. S3 offers classes with different trade-offs between storage cost and access cost/latency.

Between Standard and Deep Archive there is a factor of 23 times in storage cost. That is the prize for classifying your data well.

Three decision criteria:

1. If you do not know the access pattern, use Intelligent-Tiering. It moves objects between tiers automatically, with no retrieval cost, charging a small monitoring fee per object. It is the safe default for heterogeneous data.
2. Watch out for the minimum duration. Upload an object to Standard-IA, delete it after 10 days and you are charged for 30. For short-lived data this class works out dearer than Standard.
3. One Zone-IA only for reproducible data. It lives in a single AZ: if that AZ is lost, the data are lost. It is perfect for MercadoFresco's thumbnails, because they can always be regenerated from the original.

A lifecycle rule applies transitions and expirations automatically according to the age of the objects. It is how the saving described above happens without anyone intervening.

MercadoFresco's analysis: product photos are consulted heavily for the first few months after publication, then interest drops off, and those of discontinued products are only needed for accounting reasons. Translated into policy, in ciclo-vida-fotos.json:

Rule by rule:

• fotos-productos-enfriamiento: a three-step staircase. After 3 months it drops to IA, after a year to Glacier Instant (which still returns in milliseconds), after 3 years to Deep Archive.
• miniaturas-una-sola-az: exploits the fact that they are regenerable. Saving with no real risk.
• informes-analitica-caducan-a-2-anos: as well as moving, it deletes. Sara's reports have no value after two years, and deleting is the ultimate saving.
• limpiar-cargas-multiparte-incompletas: the most forgotten one and the one that quietly saves the most money. An interrupted large upload leaves parts behind that are billed and do not show up when you list the bucket. This rule should be in absolutely every bucket you own.
• versiones-antiguas-caducan-a-90-dias: without this one, versioning makes the bucket grow forever.

Estimated saving for the 40,000 photos of 500 KB each (some 20 GB) with a typical distribution of ages:

At 20 GB the difference is small change. At the 20 TB MercadoFresco will have after opening in three more cities (problem 3), it is more than 250 USD a month of difference. The rules go in now, when they cost a minute, not when they hurt.

With versioning enabled, S3 never really overwrites or deletes:

• When you upload an object with an existing key, a new version is created and the previous one is kept.
• When you delete, a delete marker is placed: the object disappears from listings but all its versions are still there.
• To recover it, all you have to do is delete the delete marker.

Recovering an accidental deletion, the real-world scenario:

Once enabled, versioning cannot be turned off, only suspended (the versions already created are kept). And mind the cost: every version is billed separately, hence the NoncurrentVersionExpiration lifecycle rule in the previous section.

Two additional protections, mentioned briefly because they are for specific cases:

• MFA Delete: requires a multi-factor authentication code from the root user in order to delete versions or change the versioning state. Very strong, but it can only be enabled with root credentials and cannot be managed from the console, which makes it awkward day to day.
• Object Lock (WORM: write once, read many): prevents an object being deleted or modified for a fixed period, even by the account administrator. It has two modes, Governance (a role with a special permission can bypass it) and Compliance (nobody can, not even root). It is for legal retention. It can only be enabled when the bucket is created. For MercadoFresco's database backups it would make sense; for the photos, it would not.

The moment has come. The 40,000 photos on the office server are going to S3.

What each option does:

• sync is incremental: it compares size and modification date, and only uploads what has changed. It can be interrupted and relaunched without duplicating work, which is exactly what you need to migrate 40,000 files over an office connection.
• --dryrun: essential before any sync to a new destination.
• --exclude "*" followed by several --include: order matters. First everything is excluded and then the wanted extensions are let back in. That way no .DS_Store, Thumbs.db or .bak copies sneak through.
• --cache-control: it is stored as metadata and will be honoured by browsers and, later on, by CloudFront (03-04). One day of cache for photos that hardly ever change.

Verification after the migration:

For very large volumes there are specific options worth knowing about even if we do not use them: AWS DataSync for massive, recurring transfers over the network, and the AWS Snowball family for when moving the data over the internet would take longer than shipping them physically on a device. For MercadoFresco's 20 GB, sync is more than enough.

The CLI offers two different interfaces for S3, and confusing them causes a lot of frustration.

A practical rule: move data with aws s3, configure with aws s3api.

Data leaks caused by badly configured S3 buckets have been front-page news many times. AWS responded by changing the defaults: today every new bucket is born completely private and with Block Public Access enabled at both account and bucket level.

The four switches, which a lot of people turn on or off without knowing what each one does:

MercadoFresco's stance is to leave Block Public Access enabled on every bucket, without exception. The photos will be served publicly through CloudFront with an origin access control, not by opening the bucket. That is set up in lesson 03-04.

A bucket policy is an IAM JSON document attached to the bucket saying who can do what. A real MercadoFresco example: allow only the storefront instances' role to read the photos, and force all traffic to use HTTPS.

Two fundamental details:

• s3:ListBucket is granted on the bucket (arn:aws:s3:::bucket), whereas s3:GetObject is granted on the objects (arn:aws:s3:::bucket/*). They are different ARNs and confusing them is the most frequent policy mistake of all.
• An explicit Deny always wins over any Allow, wherever it comes from. That is why the second statement is a solid defence: nobody, not even the administrator, can get in over unencrypted HTTP. The full permission evaluation logic is lesson 04-01.

ACLs (Access Control Lists) are the original mechanism from 2006, predating IAM. They grant permissions object by object to predefined accounts or groups.

AWS recommends not using them. Since 2023, new buckets are born with Object Ownership set to "Bucket owner enforced", which disables them entirely. You only need to know about them for two things: recognising them when you inherit an old account, and knowing that if a tutorial tells you --acl public-read, that tutorial is out of date. Use bucket policies.

Sara, the business analyst, needs to download the monthly sales report. She has no AWS user, she should not have one, and the bucket is private. The solution is a pre-signed URL: a temporary link that carries a cryptographic signature and expires.

The essentials of this mechanism:

• Whoever holds the link can use it. There is no additional authentication: the link is the credential. That is why the windows must be short.
• The URL inherits the permissions of whoever signs it. If the signing profile cannot read the object, neither will the URL.
• Maximum expiry: 7 days with IAM user credentials; 1 hour if it is signed from a role with temporary credentials, as would be the case inside a Lambda.
• generate_presigned_post with Conditions is the correct way to accept third-party uploads: it limits type and size before the file arrives.

Every new object in S3 has been encrypted at rest automatically since January 2023. There is nothing to do to have encryption; the only decision is who manages the key.

MercadoFresco's decision: SSE-S3 for the photos (in practice they are public data, so KMS's per-request cost is not worth it) and SSE-KMS for the sales reports and the database backups, which hold business data and where a record of who decrypts is valuable.

BucketKeyEnabled: true cuts calls to KMS, and their associated cost, by up to 99 %; turn it on whenever you use SSE-KMS. Key management is covered in depth in lesson 04-02.

In transit, all traffic with S3 goes over HTTPS. To guarantee that nobody uses plain HTTP there is the DenegarTodoLoQueNoSeaHTTPS statement in the bucket policy above.

S3 can serve a static website directly: HTML, CSS, JavaScript and images, without any server whatsoever.

The result is a URL of the form http://mercadofresco-tienda-web-desarrollo.s3-website-eu-west-1.amazonaws.com.

Limitations you need to know before getting excited:

• HTTP only, no HTTPS. The static website endpoint does not support TLS.
• No custom domain with a certificate.
• It requires a public bucket, which clashes with the Block Public Access policy.

That is why it is not used this way in production. The correct architecture is a private bucket + CloudFront in front with Origin Access Control: it gives you HTTPS, a custom domain, caching in more than 400 edge locations and protection. That is exactly lesson 03-04. S3's website mode is for Luis's quick tests.

S3 can notify when something happens in a bucket. It is the gateway to event-driven architectures, and for MercadoFresco it solves a concrete problem: the original photos are too heavy for the product listing and thumbnails are needed.

Available events and possible destinations:

The flow we will build:

Configuring the notification (the function is written in 02-05):

The filter by prefix and suffix is essential. Without it, the Lambda would also fire when writing the thumbnails into the same bucket, which would in turn generate another event: an infinite recursive loop that is one of the fastest and most expensive ways to go wrong in AWS.

Two warnings about the semantics of these events: delivery is at least once (the function may receive the same event twice, so it must be idempotent) and ordering is not guaranteed. Here we only set up the trigger; the code is written in lesson 02-05.

Multipart upload slices a large object into pieces and uploads the parts in parallel. It is mandatory above 5 GB and advisable from 100 MB upwards.

Advantages: better throughput through parallelism, retry of a single part if the network fails, and the ability to pause and resume. aws s3 cp and aws s3 sync do it automatically; you only need s3api if you program the process by hand.

The risk: if a multipart upload is interrupted, the parts already uploaded stay behind and are billed, and they do not show up in a normal aws s3 ls. That is why we included the AbortIncompleteMultipartUpload rule in the lifecycle. To see them:

Transfer Acceleration routes uploads through the CloudFront edge location nearest the client, and from there over the AWS backbone to the bucket. It speeds things up significantly when whoever is uploading is far from the region. It has an extra cost (≈0.04 USD/GB) and for MercadoFresco, which uploads from Spain to Ireland, it is not worth it. It would be useful if a supplier in South America were uploading photo catalogues.

The S3 bill has four components, and almost everybody only looks at the first one.

A realistic calculation for MercadoFresco: 20 GB of photos, 500,000 visits a month with 10 photos viewed per visit (5,000,000 GET requests), average photo 200 KB:

97 % of the cost is outbound transfer. And that figure alone justifies lesson 03-04: by putting CloudFront in front, most requests are served from the edge cache, transfer from S3 to CloudFront is free, and the price per GB to the internet is lower. The bill can drop to less than a third, and the site will be faster into the bargain.

Tools for keeping it under control:

• S3 Storage Lens offers a free dashboard with usage metrics and saving recommendations.
• S3 Inventory generates a periodic report in CSV or Parquet with every object, its class and its size: the right way to audit buckets with millions of objects without listing them one by one.
• The presupuesto-mensual-mercadofresco budget from lesson 01-02 is still the safety net that catches anything the other two miss.

Cleaning up after the lesson. If you have created test buckets, delete them. A bucket with versioning also requires deleting the versions and the markers:

aws s3 rm s3://mi-bucket-de-pruebas --recursive --profile mercadofresco-dev
aws s3api delete-objects --bucket mi-bucket-de-pruebas \\
  --delete "$(aws s3api list-object-versions --bucket mi-bucket-de-pruebas \\
    --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json)" \\
  --profile mercadofresco-dev
aws s3 rb s3://mi-bucket-de-pruebas --profile mercadofresco-dev

• Treating S3 like a disk. Renaming a prefix with 40,000 objects means 80,000 operations and minutes of waiting. Design your keys well from the outset.
• Making a bucket public "to get it working quickly". It is the origin of most published leaks. Leave Block Public Access enabled and use CloudFront with OAC.
• Forgetting LocationConstraint outside us-east-1, or including it inside us-east-1. Both fail, with unhelpful messages.
• Not configuring AbortIncompleteMultipartUpload. An invisible cost that grows forever.
• Enabling versioning and not expiring the old versions. The bucket swells without limit and nobody understands why the bill goes up when "nothing new is being uploaded".
• Confusing the bucket ARN with the object ARN in policies: ListBucket goes on the bucket, GetObject on bucket/*.
• Putting a Lambda to listen to events from the same bucket it writes to. Infinite loop. Always filter by prefix, or use different buckets.
• Using --acl public-read because an old tutorial says so. ACLs are disabled by default on new buckets and AWS advises against using them.
• Uploading short-lived data to Standard-IA. The 30-day billing minimum makes it work out dearer than Standard.
• Looking only at the storage cost. Outbound transfer is usually 90 % of what you end up paying for S3.
• Tip: run sync with --dryrun every first time. A mistyped destination can copy 40,000 files to the wrong prefix, and undoing it is more work than doing it.

MercadoFresco will keep the daily dumps of the orders database in mercadofresco-copias-basedatos, under the prefix copias/base-datos/. Each dump takes up 2 GB. The requirements are:

• The last 30 days must be restorable in minutes.
• Between 30 days and 1 year, a restore may take hours.
• Between 1 and 7 years (legal tax retention) they are kept at the lowest possible cost.
• After 7 years they are deleted automatically.

Write the lifecycle rule in JSON and work out the monthly cost in steady state, comparing it with leaving everything in Standard. Use: Standard 0.023; Glacier Flexible 0.0036; Deep Archive 0.00099 USD/GB-month.

Sara must be able to read the reports under informes/, but not delete them or touch other prefixes. The role rol-mercadofresco-analitica must be able to write in informes/. Nobody should get in without HTTPS or upload objects that are not encrypted with KMS.

Write the complete bucket policy and explain why the last condition needs a Deny and not an Allow.

For each MercadoFresco piece of data, state the bucket, prefix, initial storage class, whether you enable versioning and which lifecycle rule you would apply:

• A) The 40,000 original catalogue photos, in high resolution.
• B) The 200×200 thumbnails generated automatically from the ones above.
• C) The web server's access logs, about 3 GB a day, only consulted if there is an incident and mandatory for 90 days.
• D) The monthly sales report Sara downloads through a pre-signed URL.

Solution 1.

Volume in steady state (2 GB/day):

Design notes: 2,555 days is 7 years. Deep Archive has a 180-day minimum duration, no problem here because the objects arrive with 365 days of life and stay for more than six years. And the requirement "restore the last 30 days in minutes" is what stops you dropping to Glacier earlier: Glacier Flexible takes from minutes to 12 hours depending on the retrieval mode.

Solution 2.

Why the last two need Deny and not Allow. An Allow only grants: if you simply "allowed HTTPS", any other policy — an IAM identity policy attached to the user, for instance — could still grant access over HTTP, because permissions are added together. An explicit Deny, by contrast, takes absolute precedence: no policy at any layer can override it. For security requirements that must hold always and without exception, the correct formulation is to forbid the unwanted condition, not to allow the wanted one.

One more detail: without the s3:prefix condition on ListBucket, Sara could list the whole bucket and see the object names under other prefixes, even though she could not download them. Names that sometimes reveal more than they appear to.

Solution 3.

A cross-cutting observation: the thumbnail prefix must sit outside the prefix that triggers the event (productos/), or the notification filter must exclude them. It is the protection against the recursive loop mentioned earlier.

MercadoFresco now has its photos where they belong. You have understood that S3 is an object store and not a file system: there are no folders, only flat keys where the slash is one more character; objects are immutable; renaming a prefix means copying and deleting. In exchange for those constraints you get practically unlimited scale, eleven nines of durability through automatic replication across three or more Availability Zones, and a cost per GB some thirteen times lower than EFS.

You know how to build valid bucket names in a global namespace, LocationConstraint trap included, and you have a convention of your own (mercadofresco-<componente>-<proposito>). You have mastered the seven storage classes and their trade-offs — latency, retrieval cost, minimum duration, number of AZs — and you have written lifecycle rules that cool the catalogue photos automatically, move the thumbnails to One Zone-IA because they are regenerable, expire the reports after two years, delete the old versions and — the rule everybody forgets — abort the incomplete multipart uploads that are billed in silence.

You have enabled versioning and practised the real recovery of an accidental deletion by removing the delete marker, and you know about MFA Delete and Object Lock for the cases that demand inviolable retention. You have migrated /var/www/fotos with aws s3 sync, with its mandatory --dryrun and its inclusion filters, and you are clear about the division of labour between aws s3 for moving data and aws s3api for configuring.

On security, you have seen why buckets are born closed today: Block Public Access with its four switches, bucket policies in JSON with the critical distinction between the bucket ARN and the object ARN, the absolute precedence of an explicit Deny, and ACLs as a legacy you should not use. You have generated pre-signed URLs with boto3 so that Sara can download her reports without having AWS credentials, and also to accept third-party uploads with type and size limits. You know that everything is encrypted at rest by default and when it pays to move from SSE-S3 to SSE-KMS with BucketKeyEnabled.

You have seen static hosting and why it is not used that way in production, you have left configured the event notification that will trigger thumbnail generation — with the prefix filter that avoids the infinite loop — you know about multipart upload and Transfer Acceleration, and you have done the calculation that governs everything else: in MercadoFresco's S3 bill, 97 % is outbound transfer, not storage.

The photos are settled and the heart of the business is still missing. MercadoFresco's orders are still in the database on the office server, with its unreliable backup and its inability to grow. In lesson 02-04, "Amazon RDS", we will migrate that database to a managed service: we will see what it brings over installing PostgreSQL on an EC2, we will set up Multi-AZ with a synchronous standby replica, we will create read replicas so that Sara's reports do not punish production, and we will finally close problem 2 with automated backups and point-in-time recovery.