Lesson 05-01 ended with an uncomfortable success. Marta found the eight-second order: 7,402 ms inside PostgreSQL, an N+1 pattern with 39 queries. But getting there took four Logs Insights queries, a timeline assembled by hand and, above all, somebody having had the foresight to propagate the X-Peticion-Id header across every component. If tomorrow the problem is in the database connection, or in the payment gateway, or in a Lambda import that takes two seconds to start, you have to begin from scratch and guess all over again where to look.

The underlying problem is not one of effort: it is one of data model. Metrics aggregate and lose the individual case. Logs keep the individual case but lose the relationship between what happened in the shop and what happened in the Lambda. Neither of them stores the structure of the request: who called whom, in what order, how long each leg took and which ones overlap.

AWS X-Ray stores exactly that. Its unit of work is not a number or a line of text: it is a complete request, with its call tree and its timeline. From "something is slow" to "this specific call takes 6.2 seconds, and 5.9 of them are in a query against pedidos".

This lesson instruments the MercadoFresco shop, enables tracing on the mercadofresco-estado-pedido Lambda and goes back to the 8.14-second order to see it, this time, in a single graph.

Cost warning. X-Ray charges per trace recorded and per trace retrieved or scanned. It is cheap if sampling is well configured and ruinous if you trace 100 % of high traffic. There is a whole section on economical sampling strategy.

1. Why metrics and logs are not enough
2. The data model: trace, segment and subsegment
3. The trace ID and its propagation
4. Sampling: why not every request is traced
5. MercadoFresco's sampling rules
6. Annotations versus metadata
7. A complete trace of an order, in a diagram
8. The permissions X-Ray needs
9. Instrumenting the shop with aws_xray_sdk
10. Manual subsegments and useful annotations
11. The X-Ray daemon on EC2 and in containers
12. Enabling tracing on mercadofresco-estado-pedido
13. Enabling tracing on the ALB, on CloudFront and on API Gateway
14. The service map: how to read it
15. Trace filters: the search language
16. Latency analysis: histograms and percentiles
17. The guided case: the 8.14-second order
18. X-Ray Insights
19. Relationship with CloudWatch ServiceLens
20. OpenTelemetry and ADOT: the open alternative
21. Cost and economical sampling strategy
22. Cleanup

The column that changes everything is "keeps the causal relationship". One shop log line says llamada a lambda estado-pedido and another log line, in a different group, says REPORT Duration: 480 ms. That those two events belong to the same request is something you have to reconstruct. X-Ray knows it from birth, because the identifier travels with the request.

And there are three questions that only a trace can answer:

• How much time did the request spend waiting versus working? A log line saying "total duration 8,140 ms" does not distinguish between computing and waiting for somebody else.
• Which calls were made in parallel and which in series? It is the difference between 39 queries of 190 ms in sequence (7.4 s) and 39 queries in parallel (0.2 s).
• What happened before the code? A Lambda's cold-start time, the TLS connection setup, DNS resolution. None of that appears in your application log, because your application was not running yet.

Three concepts and one hierarchy:

• Trace: everything that happens as a result of one request. It is identified by a trace ID and groups segments from every service involved.
• Segment: the work of one service or resource within the trace. The shop generates one segment; the Lambda generates its own. It contains a name, start and end times, and status.
• Subsegment: a leg inside a segment. A SQL query, an HTTP call to another service, a block of computation you want to measure.

Note the detail that makes this model useful: the Lambda's segment (145-685 ms) is contained in the shop's Invoke subsegment (140-690 ms), and the 10 ms difference is the network latency plus that of the Lambda API itself. That figure appears in no log of either service.

A segment, in its real —simplified— JSON representation, looks like this:

The error, fault and throttle fields mark the outcome, and the distinction matters because the service map paints them in different colours:

A trace ID looks like this:

That the timestamp is inside the identifier is not decorative: it lets X-Ray locate the trace without a global index, and it is the reason traces older than 30 days cannot be queried.

Propagation is done with the X-Amzn-Trace-Id HTTP header:

Three rules to internalise:

1. The first instrumented component generates the Root. If the ALB has tracing enabled, it generates it. If not, your application does.
2. The sampling decision is taken once, at the origin, and everybody respects it. If the Root arrives with Sampled=0, the Lambda will not send its segment even with tracing enabled. This avoids incomplete traces and confuses many people when debugging: "I have enabled X-Ray on the Lambda and I see nothing" usually means the sampling decision was taken upstream.
3. The SDKs propagate the header automatically on the outgoing calls they intercept. On a call they do not intercept —an exotic HTTP client, a home-made queue— you have to propagate it yourself.

MercadoFresco receives, on a normal Friday, in the order of 2 million requests a day. Tracing them all would cost, at 5 USD per million traces recorded, around 300 USD a month in recording alone, without counting retrievals. And 99.9 % of those traces would be identical and boring: 80 ms requests that work.

Sampling decides what fraction is traced. A sampling rule has two parts:

• reservoir: a fixed number of requests per second that are always traced. It is the floor: it guarantees you always have examples, even with very low traffic.
• fixed_rate: the percentage of the rest that is traced. It is the proportional ceiling: it guarantees representativeness when traffic goes up.

With reservoir: 1 and fixed_rate: 0.05:

That is the design: at low volume you trace almost everything (and it costs you nothing); at high volume you trace a stable percentage (and the cost grows in a controlled way).

Rules are evaluated by priority, from the lowest number to the highest, and the first one that matches wins. The matching criteria are: service_name, service_type, host, http_method, url_path, and request attributes.

Marta's strategy: trace little of what is cheap and frequent, and a lot of what is expensive and important.

That /salud deserves a calculation. The ALB checks /salud every 15 seconds against each instance (03-03). With 4 instances at peak: 4 × 4 = 16 requests per minute, 691,200 a month. At 5 % that would be 34,560 monthly traces of a 3 ms response saying OK. They add absolutely nothing and they dirty the latency histograms with thousands of points at 3 ms that shift the percentiles. Excluding them is the first optimisation to make, always.

Rules are managed centrally: the SDK downloads them every 10 seconds from the X-Ray API. You change a rule in the console and every instance in the ASG applies it within seconds, with no code deployment. This enables something very useful: raising sampling to 100 % during an incident and lowering it when it is over.

It is the most practical distinction in X-Ray and the one that decides whether you find something or not:

The rule is simple: if you are going to search by it, it is an annotation; if you only want to see it when you open the trace, it is metadata.

And here is the advantage over CloudWatch that is worth underlining. In 05-01 we saw that putting PedidoId as a metric dimension would cost 190,000 USD a month. As an X-Ray annotation it is free and searchable on top: annotation.pedido_id = "48213". It is exactly the gap that metrics cannot cover.

MercadoFresco's annotations, chosen to answer real questions:

Privacy warning. Annotations and metadata are stored in X-Ray and are visible to anyone with read permission. Never put emails, names, addresses, card numbers or direct personal identifiers there. That is why MercadoFresco uses cliente_hash, a hash of the internal identifier, and not the email address. If you handle personal data under the GDPR, this decision must be reviewed by whoever handles compliance in your organisation.

A graph like this —which in the X-Ray console is called the waterfall view and generates itself— tells the whole story in two seconds of reading: there is a red bar of 7.4 seconds and everything else is irrelevant in comparison. Compare it with the spreadsheet Marta assembled by hand in 05-01.

The rol-mercadofresco-tienda role needs to be able to send segments and download the sampling rules. AWS has a managed policy for this:

If you prefer explicit least privilege, which is what we teach in 04-01, the effective content is this:

None of those actions accepts a resource ARN —X-Ray has no per-trace resources—, so "Resource": "*" is correct here; it is the same case as cloudwatch:PutMetricData in 05-01.

For the Lambda roles, rol-lambda-miniaturas and that of mercadofresco-estado-pedido, the managed policy AWSXRayDaemonWriteAccess is enough, or the even more specific AWSXRayWriteOnlyAccess.

And to read traces, the group of people on call needs:

Installation:

And the minimal instrumentation of a Flask application, which is what runs on the ASG instances:

Four decisions to understand, because each one avoids a real problem:

• context_missing="LOG_ERROR" is probably the most important parameter in the file. With the default value (RUNTIME_ERROR), any instrumented code that runs outside an HTTP request —a scheduled task, a maintenance script, a background thread, the batched metric publisher from 05-01— raises an exception and brings the task down. With LOG_ERROR, it writes an error to the log and carries on. Observability must never bring the application down.
• plugins=("EC2Plugin",) makes every segment include the instance_id and the availability zone. When a single ASG instance is degraded, that annotation is what reveals it. There are ECSPlugin and ElasticBeanstalkPlugin equivalents.
• patch_all() includes psycopg2, the PostgreSQL driver. That means every query against mercadofresco-pedidos generates a subsegment automatically, with the query text sanitised. It is exactly what is going to give the N+1 away.
• daemon_address: the SDK does not talk to the X-Ray API. It writes UDP packets to a local daemon, which takes care of batching and sending them. That is why an instrumented call adds microseconds, not milliseconds.

If you prefer finer instrumentation, patch() accepts a list of modules: patch(("boto3", "psycopg2", "requests")). Avoid instrumenting sqlite3 in production if you use it for high-frequency local caches: you would generate thousands of useless subsegments.

Automatic instrumentation covers the calls to external services. The legs of your code you have to mark yourself:

And the decorator, for functions that are called from several places:

The most common instrumentation failure in production. The SDK stores the current segment in a per-thread context variable. If you spawn a new thread, that thread has no context and everything instrumented that it runs will fail or be lost:

If this strikes you as fragile, you are right: it is. It is one of the serious arguments in favour of OpenTelemetry, which solves context propagation more cleanly. We look at it at the end of the lesson.

The X-Ray daemon is a lightweight process that listens on port 2000/UDP, batches the segments the SDKs send it and forwards them to the X-Ray API in batches.

Why this architecture? Because it decouples the application from the network. If the X-Ray API is slow or down, the application never notices: it keeps writing UDP packets that, in the worst case, are lost. A lost trace is not a problem; a shop blocked waiting for the tracing API is.

Installation on the ASG instances, added to the user data of lt-mercadofresco-tienda:

Field notes:

• UDPAddress: 127.0.0.1:2000, not 0.0.0.0. It should only listen on the local interface; nobody from outside has any business sending it segments.
• TotalBufferSizeMB: if the buffer fills up, the daemon drops segments and notes it in its log. Seeing SegmentsRejectedCount growing means you have to raise it or lower the sampling.
• Nothing needs opening in sg-mercadofresco-tienda: the traffic is local. The outbound path to the X-Ray API goes through the NAT (03-01), or better still, through a VPC interface endpoint for com.amazonaws.eu-west-1.xray if you want it never to touch the internet at all.

In containers (module 10), the daemon is deployed as a sidecar container in the same task definition, and the application reaches it over localhost in ECS with awsvpc networking. Detailed in 10-01.

In Lambda there is no daemon to install: AWS runs it for you inside the execution environment. All you need is a checkbox and a permission.

The two modes:

For mercadofresco-estado-pedido, which the shop calls, PassThrough would be enough and cheaper. For mercadofresco-generar-miniaturas, which an S3 event triggers and which has no instrumented origin, you need Active or nothing will ever be traced.

With just that you already get the segment with the duration, the cold start and the errors. To see inside the function, you have to instrument it just as on EC2:

The detail that costs everybody time the first time round: in Lambda you cannot annotate the root segment. The execution environment creates and controls it. xray_recorder.current_segment() exists, but put_annotation on it is silently ignored. Annotations always go in a subsegment of your own.

Another valuable detail: the Initialization subsegment you see in Lambda traces is the cold start. If your Duration p99 is bad but the p50 is fine, look at that subsegment: it is almost always a heavy import that can be moved inside the handler or trimmed. It is the diagnosis that in 02-05 we could only guess at.

The ALB generates the header automatically and there is nothing to enable: it is the reason the MercadoFresco shop receives a Root without anyone having programmed it. What the ALB does not do is generate a segment of its own, so its processing time does not appear as a node on the map. That latency is still visible in CloudWatch's TargetResponseTime metric (05-01), and that is a good example of the two tools complementing each other.

If one day MercadoFresco puts an API behind API Gateway:

CloudFront is the missing piece, and it pays to be honest: CloudFront does not take part in X-Ray traces. Its own identifier is x-amz-cf-id, which appears in its access logs. To correlate the CDN's time with the trace, MercadoFresco records that value as an annotation:

With that, given a trace you can look up the corresponding line in the CloudFront logs in S3, and the other way round. It is not as convenient as a node on the map, but it closes the circle.

The service map is the graph X-Ray builds by aggregating every trace in the selected time window. It is not an architecture diagram drawn by anybody: it is what actually happens, deduced from the traffic.

How to read it, element by element:

The node types matter too: X-Ray distinguishes service nodes (something you instrument, with its own segment) from downstream resource nodes (a database, an S3 bucket, which are not instrumented but whose time is measured from the caller). A red RDS node means calls to RDS are failing or slow, not that the PostgreSQL server is reporting anything.

Three patterns you recognise at a glance on a real map:

1. One isolated red node with everything else green: the culprit is obvious.
2. Everything red downstream of a node: there is a cascading failure; the deepest one is to blame.
3. A node that appears out of nowhere and should not be there: a dependency somebody introduced without telling anyone. It is one of the reasons this map is useful even with no incident at all.

Groups let you have a filtered map, for example of the purchase flow only:

A group with InsightsEnabled also generates its own metrics in CloudWatch, on which you can create alarms.

Here is X-Ray's real power. The filter expression language:

From the CLI:

A script Marta uses when a specific complaint comes in, and which replaces the four Logs Insights queries from 05-01:

The X-Ray console offers, for every node on the map and for every search, a latency histogram on a logarithmic scale. It is the tool that reveals what a percentile summarises away.

A unimodal histogram —a single hump— means every request behaves alike: if it is slow, it is slow for everybody, and the problem is structural.

A bimodal histogram —two humps— means there are two distinct populations of requests, and that is the most valuable finding X-Ray offers. The one for mercadofresco-tienda at Friday's peak:

A p95 of 1.8 s would not have shown anything odd. The p99 of 8.2 s did. But what really convinces is seeing that there are two separate humps: it is not a general degradation, it is a specific subset of requests behaving differently. And in X-Ray you can select the second hump on the histogram with the mouse and see only those traces. That leads straight to the guided case.

A useful comparison that is worth learning to make, between two versions of the application:

And the filter that isolates exactly the effect of a deployment, which is the question asked most often after releasing a version:

We go back to order 48213, this time with X-Ray. Compare the effort with the four queries of 05-01.

Step 1. Find the trace. A single search:

One trace appears, 8.14 s. In 05-01 this took two Insights queries and knowing the customer's hash.

Step 2. Open the waterfall view. You see immediately:

Step 3. See the nature of the problem. Expanding the database subsegment reveals 39 consecutive subsegments, each of about 190 ms, with the same sanitised SQL:

This is what a log cannot show: the shape. Thirty-nine identical bars, one after another, never overlapping. The diagnosis is read off the graph before anybody reads the SQL. It is the N+1 pattern: the code walks the order lines and queries each product's price separately.

Step 4. Confirm it is systematic and not an isolated case. The filter:

Returns 187 traces in 24 hours. And the complement:

Returns 2. The slowness grows with the number of order lines: confirmed. This is exactly the question the annotations were there to answer, and it is why num_lineas was an annotation and not metadata.

Step 5. Quantify the impact on the business. With the previous filter over 30 days: 5,610 orders affected, with more than 4 seconds of waiting at confirmation. They are the largest orders, that is, the highest-value ones. That figure is what turns a technical task into a priority.

Step 6. Fix it. A single query instead of 39:

Step 7. Verify with data, not with impressions. It is deployed as version 2.14.4 and compared:

Note the last row, which is a side effect nobody had anticipated: removing the N+1 halves the pressure on mercadofresco-pedidos. The alarm mercadofresco-rds-conexiones-altas from 05-01 stops sitting on the edge every Friday. A latency problem was also a capacity problem.

And step 8, which closes the loop with 05-01: now that we know what the problem is, an alarm is created so it does not return unannounced. The grupo-mercadofresco-pedidos group with Insights enabled publishes metrics in CloudWatch, and the TiempoConfirmacionPedido metric we created in 05-01 already serves to detect the regression. X-Ray diagnoses; CloudWatch watches. Neither one replaces the other.

Insights continuously analyses the traces of a group, builds a baseline of normal behaviour and detects anomalies without you defining any thresholds. When it finds one, it opens an "insight" with:

• The start time and the state (active or resolved).
• The probable root cause: the node and the error type that contribute most.
• The impact: number of requests and users affected.
• An anomaly graph against the baseline.

With NotificationsEnabled=true, each insight generates an EventBridge event (07-03), which can be routed to alertas-mercadofresco. That is the link that turns X-Ray from a diagnostic tool into a detection tool.

Two honest limitations:

• Insights detects anomalies relative to the usual, not problems. If your service has always taken 8 seconds, it will never flag it.
• It needs volume. With little sampled traffic, the baseline is noisy and produces false positives. One more reason not to sample at 1 %.

ServiceLens is the CloudWatch view that brings the three signals together on a single screen. It is literally the X-Ray service map, but with the CloudWatch metrics and a direct link to the correlated logs.

Requirement: X-Ray enabled and, for log correlation, that the groups are associated. The workflow ServiceLens enables, and the one Marta ends up using every day:

Metric → map → trace → subsegment → log. That is the complete path, and MercadoFresco now has all of it.

It would be dishonest to teach X-Ray without saying that today the industry standard is OpenTelemetry (OTel), a CNCF project that defines an API, an SDK and a protocol (OTLP) that are neutral with respect to the vendor.

AWS Distro for OpenTelemetry (ADOT) is AWS's officially supported distribution of OpenTelemetry, which can send traces to X-Ray and to any other destination at the same time.

What to recommend, with judgement:

• A new project, or one that intends not to be tied to AWS: OpenTelemetry with ADOT. The extra initial cost —configuring the collector— pays for itself the first time you want to send the same traces somewhere else, or to migrate.
• An existing project already instrumented with the X-Ray SDK, all on AWS: there is no urgency. It works, it is supported and migrating has a cost.
• MercadoFresco's case: Marta has instrumented with the X-Ray SDK because it is the quickest to get going and her whole system is on AWS. She has noted in the technical decision record that migrating to ADOT is the natural evolution when the application grows or if multi-cloud ever comes up. It is not hidden technical debt: it is a conscious decision with a review date.

A compatibility note that saves a bad time: X-Ray propagates X-Amzn-Trace-Id and OTel propagates traceparent (W3C). If you mix the two in one system, you have to configure the xray propagator in OTel so that the traces do not break at the boundary.

A minimal ADOT example in Python, so you can see the difference in shape:

Note that there is not a single line of application code: opentelemetry-instrument wraps the process and instruments the known libraries. For a large, existing application that is an enormous advantage.

"Trace retrieved" is every time somebody opens a trace or runs a filter. Querying is cheap; recording is the expensive part.

Calculation for MercadoFresco. Traffic: about 2 million requests a day, 60 million a month.

Scenario A: no rules, default sampling (1/s + 5 %).

Scenario B: with MercadoFresco's rules.

Less cost than scenario A and with 100 % of the order confirmations traced, which is the only thing that really matters. That is what a good sampling strategy does: it does not trace less, it traces better.

Scenario C: the mistake, 100 % sampling of everything.

60 million traces: 300 USD/month, plus the impact on the daemon's performance and on the console's usability, which fills up with /salud traces. Never do it outside a bounded debugging window.

The five rules of the economical strategy:

1. Always exclude /salud and the static files. It is 57 % of MercadoFresco's traffic and its diagnostic value is zero.
2. Trace 100 % of the business-critical routes. They are few requests and they are what matters.
3. A reservoir of at least 1/s in the default rule, so you have examples even in the small hours.
4. Raise sampling temporarily during an incident and lower it when it is over. The rules are centralised and apply within 10 seconds, with no deployment.
5. Watch the cost with a budget filtered by service = X-Ray (module 11).

And a performance warning, not a cost one: every subsegment has a price in size. A segment cannot exceed 64 KB, and a patch_all() over an application that makes thousands of small queries per request generates enormous segments that the daemon ends up dropping. If you see SegmentsRejectedCount in the daemon log, review what you are instrumenting.

Important: traces already recorded cannot be deleted and expire on their own after 30 days. There is no storage that keeps costing money. What you do have to do is remove the instrumentation from the code if you are not going to keep using it, or at least leave context_missing="LOG_ERROR" so an SDK with no daemon does not fill the log with errors.

1. "I have enabled X-Ray and I see nothing". In order of likelihood: (a) the daemon is not running or is not listening on 2000/UDP; (b) xray:PutTraceSegments is missing from the role; (c) the request arrived with Sampled=0 and the decision is respected downstream; (d) wrong region.

2. Putting annotations on a Lambda's root segment. They are silently ignored: the root segment is controlled by the execution environment. Annotations go in a subsegment you create yourself.

3. Leaving context_missing at its default value. Any instrumented code outside an HTTP request —scheduled tasks, threads, scripts— will raise exceptions and bring the process down. Always set LOG_ERROR.

4. Confusing an annotation with metadata. If you cannot filter by something, you put it in as metadata. Only annotations are indexed, and there is a maximum of 50 per trace.

5. Putting personal data in annotations. Email, name, address, card: never. Use a hash. Traces are seen by anyone with read permission, and they live for 30 days.

6. Tracing the health check. /salud every 15 seconds generates tens of thousands of useless traces a month, dirties the histograms and shifts the percentiles. It is the first exclusion to create.

7. Sampling at 100 % "so as not to miss anything". 300 USD a month in MercadoFresco's case, an unusable console and a daemon dropping segments. Tracing better is not tracing more.

8. Losing the context in threads. The segment lives in a per-thread variable. If you spawn a thread, you have to pass it the segment explicitly. It is the most frequent failure in applications with parallelism.

9. Looking for traces from two months ago. They do not exist: X-Ray retains 30 days, full stop. If you need historical analysis, export what interests you to S3 while it is still available.

10. Expecting CloudFront to appear on the map. It does not. Correlate with the cf_id annotation and the CDN logs in S3.

11. Instrumenting without propagating on your own calls. If your code calls another service with an HTTP client the SDK does not patch, the X-Amzn-Trace-Id header does not travel and the trace is cut off there. Propagate it by hand.

12. Using X-Ray as a replacement for logs. It is not. Traces are sampled: the request you are looking for may not be there. Logs have everything. They complement each other: the trace tells you where to look, the log tells you what it said.

Final tip: the investment that pays best in X-Ray is not the instrumentation, but choosing five or six annotations well. pedido_id, cliente_hash and num_lineas are the ones that solved this case. Think about what questions you are going to be asked —"it always happens to this customer", "it started with Tuesday's deployment", "only with large orders"— and annotate exactly what is needed to answer them with a filter.

MercadoFresco is launching a public API so that the affiliated local shops can check stock and create wholesale orders. Expected traffic:

Requirements: every order and every invoice must be individually investigable; you must be able to diagnose /stock latency without going bankrupt; the X-Ray budget is 25 USD a month.

Design the complete set of sampling rules with priorities, reservoirs and rates. Calculate the number of monthly traces and the cost. Justify each rule and say what annotations you would put in the API.

This is the waterfall of a 6.45-second trace of POST /api/pedidos/confirmar:

Identify at least three distinct problems, order them by impact in milliseconds, propose a concrete solution for each one and estimate the resulting latency. State as well what annotation or subsegment you would add so as to be able to detect each of these problems automatically in the future.

One Monday, Sara reports that 3 % of last week's orders have not produced an invoice. Customers have their order confirmed and their delivery under way, but there is no invoice. There are no errors in CloudWatch, no alarm has fired, mercadofresco-estado-pedido records no Errors, and in X-Ray the traces of those orders appear complete and green.

Architecture involved: the shop confirms the order, publishes to the alertas-mercadofresco topic, and an invoicing process subscribed to that topic generates the PDF and uploads it to mercadofresco-informes-analitica. That invoicing process is not instrumented with X-Ray.

Explain: why X-Ray has not detected it, what MercadoFresco's observability is missing, what combination of the tools from 05-01 and 05-02 you would use to diagnose it, and what instrumentation and what alarm you would leave in place so that next time it is detected in minutes. Be specific with the commands.

Solution 1

Preliminary analysis. Total traffic is 4,124,960 requests/day ≈ 124 million/month. At the default sampling (5 %) that would be 6.2 million traces: 30.50 USD. We go over budget, and on top of that we would spend almost all of it on identical /stock traces.

Proposed rules:

Detailed calculation for /stock, which holds 97 % of the traffic and where everything is decided:

• 4,000,000/day ÷ 86,400 s = 46.3 requests per second.
• Reservoir: 1/s × 2,592,000 s/month = 2,592,000… which is already too many. Careful with the reservoir when volume is high: the reservoir is a floor, not a ceiling, and with 46 requests per second there is always one to trace.

We redo it: with a 1/s reservoir the monthly floor is 2.59 million traces from /stock alone (12.95 USD). That is too much for what it contributes. Reservoir 0 and a fixed rate of 0.05 %:

• 124,000,000 × 0.0005 = 62,000 traces/month. Enough for a representative latency histogram of a route that responds in 25 ms.

Corrected table:

Cost: (586,000 − 100,000) × 5 / 1,000,000 = 2.43 USD/month in recording, plus retrievals within the free tier. Well below the 25 USD, and with 100 % of invoices and wholesale orders traced.

With the margin left over you can raise /catalogo to 25 % and enable Insights on the API group.

The lesson of the exercise: the reservoir is dangerous on very-high-volume routes, because it a floor of one trace per second, which over a month is 2.6 million. For massive traffic, reservoir 0 and a very low rate; for scarce and valuable traffic, a high reservoir and a 100 % rate.

Proposed annotations for the API:

Solution 2

Problem 1 — The Lambda's cold start: 2,980 ms (46 % of the total).

The Initialization subsegment of nearly 3 seconds is a pathological cold start. Normal in Python is 200-600 ms; 2,980 ms points to very heavy imports —typically pandas, numpy, a full SDK— or a connection established at module scope.

Solutions, in order of cost-benefit:

• Review the imports and move into the handler those used on only some paths.
• Use boto3.client() at module level (that is fine: it is reused between invocations) but do not open database connections there.
• Reduce the size of the deployment package; use layers for the large dependencies.
• If after that it is still high and latency matters: provisioned concurrency (02-05), which removes the cold start in exchange for paying for reserved capacity.

Estimated saving: from 2,980 to ~400 ms. −2,580 ms.

Problem 2 — The N+1, again: 1,305 ms (20 %).

Fourteen SELECT ... FROM productos WHERE id=? in series, of about 93 ms each. The same pattern as the guided case, on a different route.

Solution: WHERE id = ANY(...) in a single query. Saving: from 1,305 to ~100 ms. −1,205 ms.

Note: 93 ms for a SELECT by primary key is also high. It is worth checking whether there is a suitable index, whether the connection is being established on every query (no connection pooling), or whether the RDS instance is saturated. A separate conectar_bd subsegment would clear it up.

Problem 3 — Configuration loaded on every request: 890 ms (14 %).

cargar_configuracion taking 890 ms at the start of every request is configuration read from disk, from the network or from Parameter Store while hot. It should be loaded once when the process starts and kept in memory.

Solution: load it at process start and refresh it every N minutes in a background thread. Saving: from 890 to ~1 ms. −889 ms.

Problem 4 — Secrets Manager on the critical path: 310 ms (5 %).

GetSecretValue on every request. As we saw in 04-03, the secret rotates every 30 days: there is no reason whatsoever to read it on every purchase. Cache it with a TTL of 5-15 minutes and retry on receiving an authentication error.

Saving: from 310 to ~0 ms on 99.9 % of requests. −310 ms.

Problem 5 — The call to the Lambda is synchronous and blocking.

The 3,780 ms of the Invoke are entirely on the customer's critical path. But do you really need to wait for the order status to be computed before answering "confirmed"? Almost certainly not: it is a good candidate for asynchronous invocation or for a queue (07-01).

Potential saving: the whole 3,780 ms disappear from what the customer perceives.

Summary ordered by impact:

Resulting latency applying 1-4: 6,450 − 4,984 = ~1,470 ms. Applying 5 as well: ~430 ms. From 6.45 s to less than half a second.

Instrumentation to detect it automatically in the future:

The num_consultas_sql annotation deserves a comment: it is a derived annotation, computed by the application itself by counting the queries of the request. With it, a filter annotation.num_consultas_sql > 10 finds every N+1 in the system, present and future, without knowing in advance where they are. It is the kind of annotation that separates thought-through instrumentation from copied instrumentation.

Solution 3

Why X-Ray did not detect it, and it is important to understand this properly.

X-Ray traces what is instrumented. The invoicing process is not, so as far as X-Ray is concerned it does not exist. The shop's traces end at SNS Publish with status 200 —the message was published correctly— and everything looks green. The failure is after the last instrumented point, and that is the structural blind spot of any tracing system.

And there is a second, subtler reason: even if you instrumented the invoicer, the pattern is asynchronous. The shop publishes and forgets. The shop's trace ends at publication; the invoicer's would be a different trace, triggered by the message. X-Ray relates them if the SDK propagates the header through SNS (it does, in the message attributes), but the absence of a trace generates no signal at all. Nobody is counting how many messages should have been processed.

What MercadoFresco's observability is missing: it has no check that the two halves of an asynchronous process add up. It is the classic gap of event-driven architectures, and it is solved with reconciliation, not with traces.

Diagnosis, step by step:

1. Quantify it and bound it in time. A direct query against the database to find out how many and when:

If the failures cluster in specific hours, there is a one-off cause; if they are spread out uniformly at 3 %, there is a probabilistic failure —an exhausted timeout, a race condition, a concurrency limit—.

2. Check the SNS link. The SNS metrics in CloudWatch say whether the message went out and whether delivery failed:

If NumberOfNotificationsFailed is zero, the message reached the invoicer and the problem is inside it. If it is not zero, the problem is delivery and you have to look at the SNS retry policy (07-02) and whether there is a dead-letter queue.

3. Look at the invoicer's logs with Logs Insights. This is where 05-01 does the work:

That is: orders that entered the invoicer but never got as far as writing the completion line. And the definitive cross-check, looking for one specific order with no invoice across all the groups at once:

If the order appears in the shop and never appears in invoicing, the message was lost. If it appears and then cuts off halfway, the process died: memory, an exhausted timeout, an uncaught exception.

Probable diagnosis —and the most frequent in this scenario—: the invoicer takes longer than its timeout allows when generating the PDF for large orders, dies, and SNS does not retry indefinitely. With no dead-letter queue, the message disappears silently. It is a stable 3 %: the largest orders.

What to leave in place, in four layers:

a) Instrument the invoicer with X-Ray. It is the first and most obvious thing:

With pedido_id and factura_id annotations. From then on, an order's trace includes its invoicing and the filter service("mercadofresco-facturacion") { fault } finds the failures.

b) A dead-letter queue. Without one there is no way to know what was lost. MercadoFresco already has the pattern in place with mercadofresco-miniaturas-fallidas (02-05); here the equivalent is needed, and with an alarm on ApproximateNumberOfMessagesVisible > 0. The full pattern —queues, retries, idempotency— is lesson 07-05.

c) The reconciliation metric, which is the underlying solution. A process that every 15 minutes compares confirmed orders with issued invoices and publishes the difference:

And its alarm:

With --treat-missing-data breaching: if the reconciliation process itself stops running, that is an incident too.

d) The general rule to extract. In an asynchronous system, traces do not detect what did not happen. They only detect what happened badly. For what did not happen you need a reconciliation metric comparing the two halves of the process: orders against invoices, messages sent against messages processed, files uploaded against thumbnails generated.

It is exactly the same reasoning that in 05-01 led to putting --treat-missing-data breaching on mercadofresco-sin-pedidos: the absence of a signal is a signal, but only if somebody is counting it.

MercadoFresco has gone from "something is slow" to "this specific call takes 6.2 seconds, and 5.9 of them are in one query". You know why metrics and logs are not enough: metrics aggregate and lose the individual case, logs keep the case but not the causal relationship between what happened in the shop and what happened in the Lambda. Only a trace stores the complete structure of a request.

You have mastered the model: trace, segment per service and subsegment per leg, with error (yellow, 4xx), fault (red, 5xx) and throttle (purple). You know the trace ID with its embedded timestamp —and why that implies the 30 days of retention— and the X-Amzn-Trace-Id header with its Root, Parent and Sampled fields, along with the rule that misleads most: the sampling decision is taken once at the origin and everybody respects it, so "I have enabled X-Ray on the Lambda and I see nothing" almost always means it arrived with Sampled=0.

You know how to configure sampling with its reservoir and its fixed rate, and why that combination traces almost everything when there is little traffic and a stable percentage when there is a lot. You have set up MercadoFresco's rules: 100 % of /api/pedidos/*, 0 % of /salud and of the static files, 5 % by default. And you have learnt that the reservoir is treacherous on very-high-volume routes, where 1/s is 2.6 million traces a month.

You are clear on the distinction that decides whether you will find anything: indexed, filterable annotations (50 maximum, free cardinality) versus metadata that is only visible when you open the trace. And you know that pedido_id as an X-Ray annotation is free and searchable, whereas as a metric dimension in CloudWatch it would cost 190,000 USD a month: it is exactly the gap X-Ray covers. With the corresponding privacy warning: cliente_hash, never the email address.

You have instrumented the shop with aws_xray_sdk —xray_recorder.configure with context_missing="LOG_ERROR" so that observability never brings the application down, patch_all() which instruments boto3 and psycopg2 so that every query generates its subsegment, and the EC2Plugin that adds the instance and the AZ—, you have created manual subsegments with in_subsegment, and you know that in threads the context has to be passed by hand. You have deployed the daemon on the ASG instances, understanding why it exists: the SDK writes local UDP and never waits for the network. And you have enabled tracing on mercadofresco-estado-pedido with a policy and Mode=Active, knowing that in Lambda annotations go in a subsegment because the root segment is read-only, and that the Initialization subsegment is the cold start we could only guess at in 02-05.

You know how to read the service map —size, colours, client and resource nodes, and the three patterns you recognise at a glance—, and how to search with the filter language: service(), fault, responsetime > 5, edge(), annotation.pedido_id = "48213". And you know how to read a bimodal histogram, which is where you really see the two populations of requests that a percentile summarises into a single number.

And you have closed the case we had been dragging along since module 4. Order 48213: a single search by annotation, the waterfall, and there it was —39 consecutive subsegments of 190 ms, 91 % of the total time, an N+1—. Confirmed as systematic with annotation.num_lineas > 25 AND responsetime > 4, quantified at 5,610 affected orders in 30 days, corrected with WHERE id = ANY(...), and verified: p95 from 6.8 s to 0.44 s, and as a bonus DatabaseConnections from 185 to 96 at Friday's peak. Compare that with the four queries and the spreadsheet of 05-01.

You know X-Ray Insights with its notifications via EventBridge, ServiceLens as the view that joins metric → map → trace → subsegment → log, and OpenTelemetry with ADOT as the open standard AWS recommends today for new projects, with MercadoFresco's decision written down and given a review date instead of hidden away. And you know that all of this costs about 10.60 USD a month thanks to a sampling strategy that traces better, not more, against the 300 USD of tracing everything.

One question from module 4 remains unanswered, and it is the one neither metrics nor traces can answer. CloudWatch knows what the application says about itself. X-Ray knows where a request went. Neither of the two knows who called the AWS API. Nobody yet knows who decrypted the last database backup with alias/mercadofresco-datos, or who read the mercadofresco/produccion/rds/mfadmin secret, or from which IP address, or whether any of those calls failed with AccessDenied because somebody was trying doors.

That log exists, it is called AWS CloudTrail and it has been recording everything since day one without anybody looking at it. In lesson 05-03, "AWS CloudTrail", we will see the essential difference between logging API calls and logging what your application says, the free 90-day history versus a persistent trail, how to create trail-mercadofresco towards an encrypted bucket with integrity validation —and why that bucket must be impossible to delete—, the annotated anatomy of a real kms:Decrypt event, the cost difference between management events and data events, and how to investigate with Athena and SQL who assumed a role, who read a secret and which calls failed with AccessDenied.