The previous lesson left MercadoFresco half decoupled. cola-mercadofresco-pedidos works, the confirmation is down to 400 ms and the ERP can fall over without taking sales with it, but the queue is a point-to-point channel: each message is processed by one consumer and then disappears. With the warehouse, email, delivery and analytics all interested in the same fact, the shop has ended up sending four messages to four different queues. It knows again who its consumers are, which is exactly what we wanted to avoid. And when marketing asks to hear about orders for its loyalty programme, somebody will have to touch the shop's code, deploy it and test it.

Amazon SNS (Simple Notification Service) is AWS's publish/subscribe service. The shop publishes once to a topic —"order PED-084417 has been confirmed"— and SNS delivers a copy to every subscriber. Who is subscribed is a configuration detail, not a code detail. Adding marketing goes from being a deployment to being a one-line command.

In this lesson Luis creates mercadofresco-pedido-confirmado, builds the SNS→SQS fan-out pattern with the warehouse, email and analytics queues, filters by attribute so that the 24-hour delivery queue only receives what concerns it, and finally understands what the alertas-mercadofresco topic we have been using since module 5 really was.

Cost warning. SNS charges per publication and per delivery: the first million API requests are free, and deliveries to SQS and Lambda are free too. What really costs money is SMS (around 0.06 USD per message in Spain) and, to a lesser extent, email. A test loop sending SMS can generate a surprising bill in minutes. At the end you have the cleanup. All data is fictitious.

1. Publish/subscribe versus the point-to-point queue
2. Topics, publishers and subscribers
3. Subscription types and their guarantees
4. Subscription confirmation and its traps
5. The SNS→SQS fan-out pattern
6. Why a queue in the middle and not a Lambda directly
7. Building MercadoFresco's fan-out with the CLI and boto3
8. Filtering by attribute and by message body
9. Retries, delivery policies and SNS's own DLQ
10. Messages with a structure per protocol
11. FIFO topics and how they fit with FIFO queues
12. Security: topic policy, encryption and cross-account access
13. alertas-mercadofresco revisited, SMS and transactional email
14. Metrics and debugging deliveries
15. Cost and cleanup
16. Common mistakes and tips
17. Exercises
18. Conclusion

The difference between SQS and SNS is not one of technology, it is one of who knows whom.

The row to memorise is the storage one: SNS stores nothing. A topic is a router, not a mailbox. If you publish to a topic with no subscribers, the message evaporates with no error and no warning. And if the only subscriber is a dead HTTP endpoint, SNS retries according to its delivery policy and, once exhausted, discards the message. This is reason number one why the correct pattern for important work is SNS plus SQS, not SNS on its own.

The conceptual change is that the shop stops giving orders ("put this in the warehouse queue") and starts communicating facts ("an order has been confirmed"). A fact has no addressee: whoever is interested subscribes. That inversion —from command to event— is the basis of the event-driven architecture we will go deeper into in 07-03.

A topic is a named logical channel with an ARN. Creating one is instant and free:

There are two classes of topic:

A publisher calls Publish with the topic ARN; it only needs sns:Publish. A subscriber is an endpoint registered on the topic through Subscribe, with a protocol and an address. The limit per topic is 12.5 million subscriptions, so in practice it does not exist.

The guarantees are not the same. Deliveries to internal AWS destinations (SQS, Lambda, Firehose) are the most reliable: long retries, no dependency on the public network and a DLQ available. HTTP/S deliveries depend on your endpoint being alive, answering a 2xx in under 15 seconds and coping with bursts. Those for email and SMS are, in practical terms, "at most once": if the provider rejects them, SNS can do little about it, and they suit nothing that has to happen no matter what.

The protocols that point outside AWS —HTTP/S and email— require confirmation. SNS sends the endpoint a SubscriptionConfirmation message with a SubscribeURL, and until somebody visits it the subscription stays in PendingConfirmation and receives nothing.

The traps, all of them seen in production:

• The HTTP endpoint receives SubscriptionConfirmation and does not know what to do with it. Your handler expects Type: "Notification" and returns 400. The subscription is never confirmed and nobody notices until an event goes missing. The handler must tell the three types apart: SubscriptionConfirmation, Notification and UnsubscribeConfirmation.
• Confirming without verifying the signature. Anybody who knows your URL can send you a JSON with a SubscribeURL pointing at their own topic and, if you visit it blindly, you end up subscribed to somebody else's topic. Always verify Signature with the certificate from SigningCertURL —and check that that domain is sns.<region>.amazonaws.com—.
• The confirmation email ends up in spam. The SNS notice arrives from no-reply@sns.amazonaws.com; corporate filters block it frequently.
• The token expires after 3 days. Once that deadline passes you have to subscribe again.
• AuthenticateOnUnsubscribe. Without this attribute, anybody with the unsubscribe link can unsubscribe the endpoint. Set it to true on production HTTP subscriptions.

Subscriptions to SQS, Lambda and Firehose do not require confirmation when created by a principal in the same account with sufficient permissions: they confirm themselves. Another reason to prefer them.

This is the most used integration pattern in AWS and the one MercadoFresco adopts as its standard: one topic that several queues subscribe to, and one consumer per queue.

Each consumer moves at its own pace. The ERP one can run slowly and pile up 4,000 messages without the email one noticing. If the analytics consumer has a bug and has to be stopped for two hours, its messages wait in its queue and are processed afterwards: nobody else is affected.

SNS can invoke Lambda functions directly, and it is tempting to save yourself the queue. It is a mistake as soon as the work matters. Three reasons:

1. Retries and durability. If SNS invokes a Lambda and the function fails, SNS retries according to its delivery policy and then discards the message (or sends it to its DLQ, if you configured one). With a queue in between, the message is stored for up to 14 days, the consumer retries it as many times as you say and it ends up in a DLQ you can inspect and reprocess with start-message-move-task. The queue turns an ephemeral event into persistent pending work.

2. Buffering and backpressure. On Friday at 18:00, 900 orders/hour arrive in bursts. With SNS→Lambda, Lambda scales as fast as the invocations arrive: if behind it there is Aurora with 200 connections or SMTP with its sending limit, that peak propagates intact. With SNS→SQS→Lambda, the queue absorbs the burst and MaximumConcurrency (07-01) sets the rate at which it drains.

3. Failure isolation. With four Lambdas subscribed to the topic and one of them throttled by the account's concurrency limit, its invocations fail and are lost. With four queues, the problem stays contained in the affected queue.

When SNS→Lambda directly is a good idea: trivial, idempotent reactions with no consequences if one is lost —refreshing a cache, writing a metric— and when minimum latency is a requirement. For anything that is a sale, a charge or a notice to a partner, put a queue in between.

The step that is forgotten most often is the queue policy: SNS is not an IAM principal in your account, so it needs explicit permission on every destination queue.

Without the aws:SourceArn condition, any SNS topic in the world could write to your queue. It is the same protection pattern as in 07-01 with S3.

With that in place, the full setup:

RawMessageDelivery=true deserves a paragraph of its own. By default, SNS wraps your message in a JSON envelope with metadata, and the original body ends up as an escaped string inside the Message field:

This forces the consumer to do a double json.loads and to read the attributes from a different place than it would if the message arrived straight at the queue. With RawMessageDelivery=true, the queue receives the body as it is and the SNS attributes become native SQS attributes: the same consumer works whether the message comes from the topic or was sent directly to the queue. Always enable it on SQS and Firehose subscriptions.

The publisher then looks like this:

Notice that the fields used for filtering are duplicated: in the body, because the consumer needs them, and in the attributes, because by default SNS only looks there. It is a deliberate and cheap duplication.

For high volumes, publish_batch accepts 10 messages per call with the same trap as send_message_batch: it returns 200 with a Failed list that has to be inspected.

A filter policy is a JSON attached to the subscription. If the message does not match, SNS does not deliver it to that subscriber —and does not charge for it—. It avoids the antipattern of "everyone receives everything and each consumer discards the rest", which wastes invocations and fills the queues.

Marta wants cola-mercadofresco-reparto-24h to receive only orders in the 24-hour slot:

The filter language supports far more than equality:

The rule that confuses people most: within one attribute the list is an OR; between different attributes it is an AND. The last row of the table demands the 24h slot and also an amount ≥ 150. To express an OR between different attributes you need $or:

Filtering by message body. Since 2022 SNS can filter by looking inside the body's JSON, which removes the duplication of fields. You enable it by setting FilterPolicyScope to MessageBody:

The policy can navigate nested objects and arrays. Two limits that matter: the body must be valid JSON —if it is not, the subscription receives nothing and the failure is silent— and a subscription has a single filter scope, either attributes or body, never both.

MercadoFresco uses attributes for basic routing (event type, slot, province) and the body for rich cases, such as alerting the cold chain team when one of the lines starts with PESC-.

When SNS cannot deliver, it retries according to the protocol's delivery policy. For SQS, Lambda and Firehose it is predefined and generous: immediate, pre-backoff, exponential backoff and post-backoff phases, up to about 23 hours in total. For HTTP/S it is configurable:

maxReceivesPerSecond is the backpressure towards the partner's ERP: even if MercadoFresco publishes 900 messages in a minute, SNS will not send it more than 20 per second. It is the only way to protect an HTTP endpoint you cannot scale.

The SNS DLQ is configured per subscription, not per topic, with the RedrivePolicy attribute. It collects the messages that exhausted the retries towards that particular subscriber:

Two DLQs coexist in the same architecture and must be told apart, because they do not mean the same:

Both need an alarm on ApproximateNumberOfMessagesVisible towards alertas-mercadofresco.

The same event does not read the same way in a 160-character SMS as it does in a queue. With MessageStructure="json", a single Publish carries a different payload per protocol:

The default key is mandatory and is used for any protocol without an entry of its own. If it is missing, Publish returns InvalidParameter. Only email and HTTP use the Subject; it is ignored on SQS and Lambda, and it is limited to 100 ASCII characters.

About attributes: a message accepts up to 10, and their size counts towards the message's 256 KiB limit. With MessageStructure="json", each per-protocol payload also counts against that total limit.

A FIFO topic guarantees end-to-end ordering and deduplication, but it can only deliver to SQS FIFO queues (and, since 2023, to Lambda functions). It does not support HTTP, email or SMS. It fits the stock queue we created in 07-01:

When publishing you have to supply MessageGroupId and, if there is no content-based deduplication, MessageDeduplicationId:

The MessageGroupId is propagated to the queue, so the ordering is preserved along the whole chain. And attribute filtering works on FIFO topics too, which lets one queue receive only the reservations and another one all the movements.

The topic policy controls who publishes and who subscribes. By default, only the account owner. This one tightens it: the shop publishes, and only SQS subscriptions can be created.

The sns:Protocol condition is a practical safeguard: it stops somebody subscribing their personal email address to a topic that carries customer addresses. There is also sns:Endpoint to restrict to specific endpoints.

Encryption. KmsMasterKeyId=alias/mercadofresco-datos encrypts the messages at rest inside SNS. Two warnings: the publisher needs kms:GenerateDataKey* and kms:Decrypt on the key, and if an AWS service publishes to the topic —CloudWatch when it fires an alarm, S3 when an object is uploaded— the key policy must allow it explicitly. It is the usual cause of alarms that stop warning anybody right after encryption is switched on.

Cross-account access. The delivery partner has its own AWS account and wants to consume the events. They are not given credentials: their account is added to the topic policy with sns:Subscribe, and they create in their account a queue whose policy accepts our topic. Neither of the two parties gets access to the other's infrastructure. It is the same least-privilege principle from 04-01 applied to integration.

Since module 5 we have been writing --alarm-actions arn:aws:sns:...:alertas-mercadofresco without fully explaining what was going on. Now it is clear: a CloudWatch alarm is an SNS publisher. When it moves to ALARM, it publishes a JSON with AlarmName, NewStateValue, NewStateReason and the metric's dimensions. The topic has Marta's email subscribed, and it could have subscribed —without touching a single alarm— a queue that records the history, a Lambda that opens a ticket or an HTTPS endpoint towards the on-call tool.

That is exactly the value of the model: CloudWatch does not know who hears about its alarms. Adding a destination does not require modifying 40 alarms, only creating a subscription.

An immediate improvement now that we know about filtering: subscribe the on-call SMS with a policy that only lets the critical things through.

With FilterPolicyScope=MessageBody, Marta receives an SMS only for order alarms that enter the ALARM state, while the email keeps receiving everything, including the returns to OK.

About SMS and email. SNS is for operational notices to known people: an SMS to the on-call engineer, an email to Marta. It is not for transactional email or for campaigns: there are no templates, no open tracking, no bounce handling, no sender reputation control, and the sender is no-reply@sns.amazonaws.com. MercadoFresco's order confirmation email is sent with Amazon SES, which does offer your own domain, DKIM, templates and deliverability metrics; the Lambda subscribed to cola-mercadofresco-correo calls SES, not SNS. For bulk SMS, SNS also requires leaving the sandbox and registering the sender with the carrier.

The three FilteredOut metrics are the ones that save the most time. The classic symptom —"the queue receives nothing and there is no error"— is almost always a filter: either the publisher does not send the attributes, or it sends them with the wrong type. A Number sent as a String does not match a numeric filter, and SNS discards it without saying a word.

To debug failed deliveries you need to enable the delivery status logs, which are not on by default: you configure a role with write permission on CloudWatch Logs and a sampling percentage (SQSSuccessFeedbackSampleRate, HTTPSuccessFeedbackSampleRate…). With them, the endpoint's real response codes appear in Logs, which is the only thing that lets you tell a 403 from permissions apart from a 500 from the partner. And since every Publish is recorded in trail-mercadofresco (05-03), you can always check whether the event was published at all.

What has to be watched is SMS: a test loop sending 5,000 messages costs 300 USD in a few minutes. Always set MonthlySpendLimit in the account's SMS preferences and an alarm on SMSMonthToDateSpentUSD. Note as well that fan-out multiplies deliveries: publishing 240,000 times with four subscribers is 960,000 deliveries, free towards SQS but chargeable towards HTTP.

Deleting the topic deletes neither the subscribed queues nor their messages: clean those separately with the 07-01 commands. And beware: deleting a topic does not remove unconfirmed email subscriptions.

Publishing to a topic with no subscribers. Publish returns 200 and a MessageId, and the message evaporates. No error, no failure metric. Check NumberOfNotificationsDelivered after any deployment that touches subscriptions.

Forgetting the queue policy. The subscription is created without complaint, but SNS cannot write. Messages are lost and NumberOfNotificationsFailed goes up. It is the number one fan-out failure.

A queue policy without aws:SourceArn. It works, but it leaves your queue open to any SNS topic.

Not enabling RawMessageDelivery. The consumer receives the SNS envelope, needs a double json.loads and the attributes turn up where it is not looking. Always enable it on SQS subscriptions.

Filters that discard everything silently. The publisher does not send the attributes, or sends a number as a String when the filter uses numeric. Symptom: zero messages and zero errors. Look at NumberOfNotificationsFilteredOut and its two variants. And do not confuse OR and AND: within one attribute, list = OR; between different attributes, AND; for an OR between attributes you need $or.

Subscribing Lambdas directly for important work, or assuming that SNS stores the messages. It does not: a consumer that is down during a deployment permanently loses everything published in that window if there is no queue in between.

Encrypting the topic and forgetting the key policy. CloudWatch alarms stop publishing and nobody finds out —because the very mechanism that warns you is the one that has failed—.

Tip: name topics after the fact, not after the destination. mercadofresco-pedido-confirmado, not mercadofresco-avisar-almacen. The name is part of the contract and should not age when the consumers change.

Tip: put version in the body from the very first message. When a field has to be added a year from now, older consumers will be able to ignore it safely.

Tip: filter at the topic, not at the consumer. A consumer that discards 90 % of what it receives pays for invocations, queue requests and complexity for nothing.

Marketing wants to process confirmed orders for its loyalty programme, but it is only interested in those of 60 € or more or those from customers in Catalonia, and its system is an HTTPS endpoint that copes with 5 requests per second and suffers outages of up to 40 minutes.

Design the integration: (a) what you subscribe to the topic and why; (b) the exact filter policy in JSON; (c) what you configure to protect its endpoint from the Friday peak; (d) which DLQs are involved and what each one means; (e) what changes in the shop's code.

Luis deploys the fan-out on a Thursday. On Friday morning, Marta sees: NumberOfMessagesPublished 8,400; NumberOfNotificationsDelivered 16,800; NumberOfNotificationsFailed 8,400; NumberOfNotificationsFilteredOut 0. The warehouse and email queues are receiving normally; the analytics one is empty. cola-mercadofresco-analitica exists, its consumer works if messages are sent to it by hand, and the subscription shows as Confirmed.

Explain: (a) what exactly is failing and how you deduce it from the numbers; (b) why the subscription is confirmed even though it does not work; (c) what command you would run to confirm the diagnosis; (d) the fix; (e) what alarm would have warned on Thursday afternoon.

For each case, decide whether to use SNS, SQS or SNS→SQS, and justify it in two sentences: (a) the shop tells the warehouse to prepare a box; (b) three teams want to hear about order cancellations and a fourth is expected; (c) when a photo is uploaded the thumbnail has to be generated; (d) Marta has to be warned by SMS when the DLQ has messages; (e) stock movements have to be sent to two different systems keeping the ordering per SKU; (f) the shipping cost service has to be called to show the price on the checkout screen.

Solution 1

(a) What gets subscribed is an SQS queue (cola-mercadofresco-fidelizacion), not the HTTPS endpoint. The reason is the 40-minute outage: if SNS delivered directly over HTTP, we would depend on the delivery policy holding out, and anything that failed once the retries were exhausted would be lost. With a queue, the events wait up to 14 days and are processed when the system comes back. The consuming is done by a Lambda subscribed to the queue that calls the marketing endpoint.

(b) Since the requirement is an OR between two different attributes, $or is needed:

If the two attributes were written in the same object without $or, SNS would demand both conditions at once and marketing would lose most of the events. Requirement: the shop must publish importe_eur as a Number and provincia as a String.

(c) The protection is in the consumer, not in SNS: MaximumConcurrency=5 on the Lambda's event source mapping, which limits the pace to what the endpoint can take; a small batch-size; and a queue VisibilityTimeout ≥ 6 × the function's timeout. The queue absorbs the Friday peak and drains it at 5 requests per second. (If the endpoint had been subscribed directly instead of a queue, the equivalent tool would be throttlePolicy.maxReceivesPerSecond in the delivery policy.)

(d) Two different DLQs. The SNS subscription DLQ would collect delivery failures from the topic to the queue —badly set permissions, a deleted queue—; in practice it almost never fires because delivery to SQS is very reliable, but it is worth having. The queue DLQ (mercadofresco-fidelizacion-fallidas, with maxReceiveCount=4) collects what the Lambda could not process: if after 40 minutes of outage the endpoint is still rejecting, those events are isolated there and reprocessed with redrive.

(e) Nothing. That is the result we were after: the shop already publishes the fact with all the attributes needed, and adding marketing means creating a queue, a queue policy, a subscription with its filter and a Lambda. Zero deployments of the shop's code.

Solution 2

(a) The numbers match exactly with one of the three subscriptions failing every single time. With 8,400 publications and three subscribers there should be 25,200 deliveries; there are 16,800 successful (two subscribers × 8,400) and 8,400 failed (the third × 8,400). Since FilteredOut is 0, it is not a filter problem: delivery is attempted and it fails. Combined with the fact that the analytics queue exists and its consumer works, the diagnosis is clear: the policy on cola-mercadofresco-analitica does not allow SNS to write to it. Almost certainly, Luis's deployment loop failed when applying the policy on that queue, or applied it with the wrong aws:SourceArn.

(b) Confirmed and functional are different things. SQS subscriptions within the same account self-confirm on creation: SNS checks that the ARN is valid, not that it has write permission. The permission is evaluated on every delivery, not on subscribing. That is why the state is Confirmed and every publication fails with AccessDenied.

(c) aws sqs get-queue-attributes --queue-url ...cola-mercadofresco-analitica --attribute-names Policy to see whether there is a policy and whether the aws:SourceArn matches the topic ARN. As a complement: enable the topic's delivery status logs and look in CloudWatch Logs for the real error code of the failed deliveries, which will say AccessDenied unambiguously.

(d) Apply to the queue the same policy as to the other two, with Principal sns.amazonaws.com, action sqs:SendMessage, Resource the analytics queue ARN and an ArnEquals condition on aws:SourceArn with the topic ARN. Nothing else: the messages from the last few hours are already lost —SNS does not store anything— and that is where you see why EventBridge's event archive and replay (07-03) are so valuable.

(e) An alarm on the topic's NumberOfNotificationsFailed > 0, with a five-minute period, would have warned within minutes of Thursday's deployment instead of the problem being discovered on Friday morning. It is the minimum mandatory alarm for any SNS topic. As a complement, the analytics queue's ApproximateNumberOfMessagesVisible sitting at 0 during working hours would also have been a clear signal.

Solution 3

(a) SQS. It is an order with a single known addressee and it needs durability and retries. A topic adds nothing if there is only one interested party, and it would add one more piece to maintain.

(b) SNS→SQS. It is the canonical fan-out case: several parties interested in the same fact and one more expected. Each team with its own queue to isolate itself from the others' failures and pace.

(c) SQS. A single consumer and idempotent work. S3 can publish directly to the thumbnails queue; putting a topic in the middle would only make sense if another system needed to hear about the new photos.

(d) SNS. The destination is a person over a channel SNS supports natively. No queue is needed here: the CloudWatch alarm publishes and SNS delivers. Adding SQS would improve nothing, because an SMS is not reprocessed.

(e) FIFO topic → FIFO queues. Two destinations demand fan-out and the ordering per SKU demands end-to-end FIFO. mercadofresco-stock-movimientos.fifo with MessageGroupId equal to the sku and two FIFO queues subscribed, each with its own filter where appropriate.

(f) Neither: a synchronous call. The customer needs the shipping price now in order to decide. Neither queue nor topic: an HTTP call with a short timeout, bounded retries and a default value if the service does not answer. It is the reminder that asynchronous is not always better —it only is when the result is not part of the answer to the user—.

MercadoFresco has gone from a shop that sent four messages to four queues to a shop that publishes one fact to mercadofresco-pedido-confirmado and walks away. The warehouse, email and analytics each have their own queue subscribed with RawMessageDelivery=true, their own retries and their own DLQ; cola-mercadofresco-reparto-24h receives only the orders in the urgent slot thanks to a filter policy; the topic is encrypted with alias/mercadofresco-datos and its policy restricts who publishes and with which protocols you can subscribe. Adding marketing is now creating a queue and a subscription, without touching or deploying the shop's code.

Along the way the ideas that govern the pattern have become clear: that SNS stores nothing, which makes the intermediate queue mandatory for any work that matters; that the queue provides durability, buffering and failure isolation that a direct Lambda subscription cannot give; that filtering by attribute or by body avoids the waste of everyone receiving everything, with the rule of OR within an attribute and AND between attributes; that the subscription DLQ and the queue DLQ signal different problems —delivery versus processing—; and that the alertas-mercadofresco topic from module 5 was this very mechanism all along, with CloudWatch as a publisher that does not know who is listening to it.

But fan-out has a ceiling. SNS delivers the same message to every subscriber and only knows how to filter with simple rules over whatever the publisher remembered to include. When MercadoFresco wants to route by content for real —PedidoCancelado going to one place and StockBajo to another, all over the same channel— it would have to create one topic per event type and go back to a map of subscriptions that is hard to govern. There are also facts the shop does not publish that are equally interesting: an EC2 instance changing state, Trusted Advisor spotting an underused resource, a deployment failing. And something is missing that hurts today: when the analytics consumer was misconfigured, the events from those hours were lost for ever, because a topic stores nothing.

In 07-03, "Amazon EventBridge", we take the leap from the topic to the event bus: a channel where both applications and AWS services themselves publish, with rules that route according to the full content of the event, input transformation so the destination is not coupled to the source format, targets ranging from Lambda to an ERP API, scheduled tasks with cron, a direct connection to the mercadofresco-carritos Streams through Pipes, and —the thing that would have saved the analytics queue incident— archive and replay of everything published.