In the previous lesson we toured the console and created our first bucket with the mouse. It worked, but we left one question open: how do you repeat that exact thing a hundred times, or how do you prove six months from now what you actually did? The answer is to stop clicking and start writing commands and code.

This lesson is the gateway to all serious work on AWS. The AWS CLI is the tool you will use every day to query, create and diagnose; the SDKs are the libraries your application uses to talk to AWS from the inside. Both call the same APIs as the console, so everything you learn here applies to any service in the rest of the course.

1. Why automate: the problem with the console
2. Installing AWS CLI v2 on Linux, macOS and Windows
3. Anatomy of a command
4. Configuring credentials: aws configure, files and profiles
5. Credential precedence
6. Access key security
7. Output formats and filtering with --query
8. Pagination, --dry-run and other useful options
9. Essential exploration commands
10. SDKs: what they are and which languages have one
11. boto3: the Python SDK step by step
12. An equivalent look in JavaScript
13. CloudShell as a no-install alternative

Marta created the test bucket in the console. Now Luis needs to create three identical buckets (one per environment) with the same tags and the same configuration. With the console he would have to repeat about fifteen clicks three times over, and any slip would produce a silent difference between environments.

The four structural problems with the console:

The CLI solves all four: a command is text, and text can be saved, versioned, reviewed, repeated and chained. And it is the natural stepping stone towards the infrastructure as code of module 9, where we will not even write commands any more, but declarations of the desired state.

Always use version 2. v1 is in maintenance mode, it used to be installed with pip (which caused dependency conflicts) and it lacks important features such as signing in with IAM Identity Center.

What each line does:

• curl ... -o "awscliv2.zip": downloads the package and saves it under that name. The URL is the official AWS one; be wary of third-party installers.
• unzip: extracts an aws/ folder containing the installer.
• sudo ./aws/install: copies the binaries to /usr/local/aws-cli and creates the /usr/local/bin/aws link. It needs sudo because it writes outside your home directory.
• rm -rf: deletes what you downloaded, which is no longer needed.

If your machine is ARM (a Graviton instance, for example), swap x86_64 for aarch64.

To upgrade an existing installation, add the update parameters:

An alternative with Homebrew, if you already use it: brew install awscli.

Download and run the official MSI installer:

Or from PowerShell as administrator:

Afterwards close and reopen the terminal so that the PATH variable is reloaded.

On any system:

Expected output (the numbers will vary):

Read it like this: CLI version, embedded Python version (it does not use your Python, it ships with its own), operating system and build type. If you see aws-cli/1.x, you have v1 and you should uninstall it before carrying on.

Every command follows the same structure:

A real example, line by line:

The verbs follow very predictable conventions, and recognising them speeds up learning enormously:

It is the best documentation you have and it works offline:

It opens in a pager: navigate with the arrow keys and quit with q.

The CLI needs to know who you are. We are going to create access keys for the mercadofresco-admin user we created in lesson 01-02.

In the console: IAM → Users → mercadofresco-admin → Security credentials → Create access key. Choose the Command Line Interface (CLI) use case, confirm the warning and create.

You will get two values:

• Access key ID: something like AKIAIOSFODNN7EXAMPLE. It is public-ish; it identifies the key.
• Secret access key: something like wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. It is shown only once. If you lose it, you have to create a new key.

Warning: never create access keys for the root user (lesson 01-02). And if you would rather not handle permanent keys at all, the modern alternative is aws configure sso with IAM Identity Center, which issues temporary credentials.

It will ask you four questions:

• Default region: eu-west-1, the one we settled on in lesson 01-03. It saves you writing --region on every command.
• Output format: json is a good default value because it is the one consumed by other tools.

aws configure performs no magic: it writes two text files in your home directory (~/.aws/ on Linux and macOS, %USERPROFILE%\.aws\ on Windows).

~/.aws/credentials — holds the secrets:

~/.aws/config — holds the non-secret settings:

You can edit them by hand perfectly well. Check as well that the permissions are restrictive:

This leaves the file readable and writable by your user only. On a shared machine it is essential.

You will almost never work with a single identity. MercadoFresco will soon have a development environment and a production one, and it is important that the two cannot be confused.

Create a new profile:

The files then look like this:

Mind the detail: in credentials the section is [mercadofresco-dev], but in config it is [profile mercadofresco-dev], with the word profile in front. It is a historical inconsistency that causes constant errors. If a profile "cannot be found", check this first.

To use a profile:

And to always know who you are acting as:

Get into the habit of running this command before any destructive operation. It is the difference between deleting the development database and deleting the production one.

When you run a command, the CLI looks for credentials in a fixed order and stops at the first source it finds. Knowing this order explains 90 % of all "but I did configure that" moments:

The last two are the conceptually important ones. When your code runs inside AWS —on an EC2 instance, in a Lambda function, in a container— it needs no keys at all: it is assigned an IAM role and the SDK automatically obtains temporary credentials that rotate themselves every few hours.

IAM roles are studied in depth in lesson 04-01. Hold on to the idea: permanent keys on your laptop only; inside AWS, always roles.

An access key for mercadofresco-admin is, quite literally, full control of the account. These rules are not optional.

This is the mistake that ruins accounts every single day. There are bots continuously scanning GitHub for strings starting with AKIA; a published key is exploited within minutes, typically by launching dozens of large instances to mine cryptocurrency. The resulting bill can run into thousands of euros.

Minimum protection in any repository:

And an automatic barrier, which is highly recommended:

If you ever publish a key by mistake: deactivate and delete it in IAM immediately, check CloudTrail (lesson 05-03) to see what was done with it, and create a new one. Deleting the commit achieves nothing: it has already been indexed.

The procedure for rotating without downtime is simple: create a second key, update wherever it is used, check that everything works, deactivate the old one, wait a few days and delete it. IAM allows two active keys per user precisely for this.

AWS commands return a great deal of information. Learning to trim it is what turns the CLI into a comfortable tool.

--query applies a JMESPath expression to the response, on the client. Let us build up step by step.

Level 1 — extract a list:

Level 2 — one field from each element:

Output:

Level 3 — rename fields into an object:

The braces {...} build a new object; to the left of the : goes the name you want to see, to the right the original field.

Level 4 — filter with a condition:

Note the backticks around eu-: in JMESPath, literals go inside backticks, not inside ordinary quotes. It is a frequent source of errors.

Level 5 — a real MercadoFresco case:

Breakdown:

• Reservations[].Instances[]: flattens EC2's nested structure (instances come grouped by reservation) into a single list.
• State.Name: navigates inside a nested object with the dot.
• Tags[?Key==\Name`]|[0].Value: filters the tags down to the one with key Name, and the |[0]` takes the first result of that list in order to extract its value.

The ideal is to combine them: --filters to bring back little, --query to present it well.

When there are many results, the CLI v2 paginates them automatically and opens a pager. Options to control it:

If you limit with --max-items and there are more results, the response includes a NextToken that you can pass with --starting-token to request the next page.

Many EC2 operations accept --dry-run: they check whether you would have permission to perform the operation, but they do not run it.

If you have permission, you will see:

That message, even though it says "error", is the confirmation of success. If you did not have permission, the error would be UnauthorizedOperation. It is a safe way of verifying permissions without creating anything and without spending anything.

These are the commands you will run hundreds of times. Keep them somewhere safe.

The output of the last one is especially useful because it indicates the origin of each value:

When something does not work the way you expect, this command tells you exactly which credential and which region the CLI is using, and why.

An SDK (Software Development Kit) is a library that lets your application call the AWS APIs from the language it is written in, without building signed HTTP requests by hand.

The difference from the CLI:

Languages with an official SDK:

They all share the same logic: a credential provider chain (the same order of precedence we saw), automatic retries with exponential backoff, and operation names equivalent to those of the API.

Luis is going to use Python for MercadoFresco's scripts, so let us look at boto3 in detail.

boto3 offers two different interfaces to the same thing:

The same goal with both:

boto3 uses exactly the same precedence order as the CLI, so if aws s3 ls works, your Python code will work with nothing to configure.

To choose a profile or a region explicitly you use a session:

Every error returned by an AWS service arrives as botocore.exceptions.ClientError. Ignoring it is the most common mistake in a beginner's scripts.

Key points in the snippet:

• head_bucket is the cheap way to check existence: it downloads no content.
• error.response["Error"]["Code"] is the standard path for reading the error code. Memorise it.
• Telling 404 apart from 403 matters: "it does not exist" and "it exists but you cannot see it" demand different actions.
• The final raise avoids the antipattern of swallowing unknown errors, which hides real problems.

APIs return at most a few hundred items per call. If you iterate only the first response, you will lose data silently. Paginators solve it:

So that you can see the concepts carry over, here is the same bucket listing with the AWS SDK for JavaScript v3:

Differences from boto3: v3 is modular (less weight in Lambda) and uses the command pattern: you create an XxxCommand object and send it with send(). The credential, retry and error logic is equivalent.

As we saw in lesson 01-04, AWS CloudShell gives you a browser terminal with the CLI v2, Python and boto3 already installed, and automatically authenticated as the console user.

CloudShell is especially useful for the "I am on a computer that is not mine and I need to diagnose something right now" scenario. It does not replace the local CLI for day-to-day work, but it is an excellent safety net.

• Unable to locate credentials. You have not configured the profile, or you have used a profile name that does not exist. Diagnose it with aws configure list.
• ProfileNotFound with the profile spelt correctly. It is almost always the ~/.aws/config section without the profile prefix: it must be [profile mercadofresco-dev], not [mercadofresco-dev].
• You must specify a region. Neither the profile nor the environment variable nor the command states a region. Add --region or set it in the profile.
• An unexpected AccessDenied. Check which identity you are acting as first, with aws sts get-caller-identity; very often it is the wrong profile, not a permissions problem.
• Pushing keys to Git. The most expensive and most frequent disaster. .gitignore, git secrets and, better still, temporary credentials.
• Using permanent keys inside an EC2 instance. Never. You use an instance role (lesson 04-01), which rotates the credentials for you as a bonus.
• Forgetting pagination. Your script says "there are 1,000 objects" when there are 50,000. Use paginators in boto3 and --max-items/NextToken in the CLI.
• Confusing aws s3 with aws s3api. The first is high level (ls, cp, sync); the second exposes the full API (get-bucket-tagging, put-bucket-policy). If an option does not exist in s3, look for it in s3api.
• Ignoring ClientError. A script that fails silently is worse than one that breaks.
• Tip: define aliases in your terminal for whatever you repeat a lot, for example alias whoami-aws='aws sts get-caller-identity'.
• Tip: when a command does not work and you cannot see why, add --debug. You will see the full signed request and the service's response.

1. Install AWS CLI v2 on your system and check the version.
2. Configure a profile called mercadofresco-admin with region eu-west-1 and json output.
3. Run aws sts get-caller-identity and note down your full ARN.
4. Write a single command that shows, in table format, every region whose name starts with eu-, with a single column called Region.

Write a script inventario-mercadofresco.sh that:

1. Checks which identity and which account it is running under, and displays it.
2. Lists every S3 bucket in the account.
3. Lists the Availability Zones of eu-west-1 with their physical ID.
4. Lists the EC2 instances with their ID, type, state and zone (if there are none, it must say so without failing).
5. All of it with table output and without opening the interactive pager.

Write a Python script crear_bucket_mercadofresco.py that:

1. Takes the bucket name as a command line argument.
2. Checks whether it already exists (telling "does not exist" apart from "no permissions").
3. If it does not exist, creates it in eu-west-1 with public access block enabled.
4. Applies the tags Proyecto=mercadofresco, Entorno=pruebas, Componente=formacion.
5. Displays the resulting tags.
6. Handles errors with ClientError and returns a non-zero exit code if it fails.

Cost: an empty bucket generates no appreciable cost and the Free Tier covers 5 GB. Even so, delete it when you are done with aws s3 rb s3://<bucket-name>.

Solution 1

You can use either starts_with() or contains(); starts_with is more precise because contains would also accept a name carrying eu- in the middle.

Solution 2

Important details:

• set -euo pipefail stops the script from merrily carrying on after a failure.
• export AWS_PAGER="" is essential in scripts: without it, the CLI would open a pager and the script would hang waiting for somebody to press q.
• Storing the output in a variable lets you check whether it is empty and give a clear message instead of printing an empty table.

You run it with:

Solution 3

Keys to the solution:

• LocationConstraint: if you create a bucket in any region other than us-east-1 and omit this parameter, S3 creates it in us-east-1. It is one of the historical oddities of the API.
• put_public_access_block with all four options set to True is the protection that prevents the most common data leak on AWS.
• put_bucket_tagging overwrites the complete tag set: if you wanted to add one without deleting the others, you would have to read them first with get_bucket_tagging and merge them.
• Exit codes: 0 if all goes well, 1 if it fails, so that the script can be chained into a pipeline.

Running it and cleaning up:

With this lesson you close the introductory module and, above all, make the leap from "AWS user" to "AWS operator". You have installed the AWS CLI v2, you have understood the anatomy of a command (aws <service> <operation> --parameters) and the naming conventions that let you guess commands you do not know yet. You have configured credentials with aws configure, you have seen what is really inside ~/.aws/credentials and ~/.aws/config, and you know how to work with named profiles so as never to confuse development with production. You know credential precedence, which explains almost every strange behaviour, and the non-negotiable security rules: never keys in Git, periodic rotation and, inside AWS, roles instead of keys.

You have learned to tame command output with --output and with --query and JMESPath, to avoid the silent trap of pagination, to verify permissions without spending anything using --dry-run, and you have in hand the handful of exploration commands you will use every day. And you have taken your first steps with the SDKs: boto3 in Python, with the distinction between client and resource, correct error handling through ClientError and paginators, plus the JavaScript equivalent to show that the concepts are universal.

MercadoFresco now has everything it needs to start in earnest: a secure, monitored account, a region chosen with judgement, a console you know how to drive, and command line and programming tools that make the work repeatable.

In module 2, "Core AWS services", starting with lesson 02-01 "Amazon EC2", we will stand up MercadoFresco's first real server in the cloud and begin the migration that will solve, one by one, the four problems we uncovered in the very first lesson.