The previous lesson ended with an unanswered question: bicycle 417 has its plate, its model and its status in PostgreSQL, and also copied inside thousands of MongoDB documents. Station 12 has eleven columns in one engine and a thirty-field profile in the other. Nobody has said which one wins.

This lesson answers that question, and adds two more pieces to the system: Redis for real-time availability and application sessions, and Elasticsearch for searching stations by name and address. Four engines for a single municipal bike service.

It is important to say from the start what kind of lesson this is. It redesigns nothing: the relational schema from 08-01 and the collections from 08-02 are taken as given and are not touched. What is dealt with here is what neither of the two previous lessons could deal with on its own — the split and the coherence: which piece of data lives in which engine, who is the source of the truth, with what delay changes propagate, what keeps working when one piece goes down, and how you operate a system with four stores instead of one.

And it also deals with the question that almost never shows up in architecture presentations: when to dismantle all of this. Because most of the polyglot architectures that exist should not exist, and knowing how to recognize that is part of the craft.

Contents

  1. Polyglot persistence, put into practice
  2. The real price, and why the default answer is still a single database
  3. The VallBici architecture
  4. The master decision table: who rules over each piece of data
  5. The four synchronization patterns
  6. Coherence in practice: 4 bikes in the app, 3 on arrival
  7. Failure modes and graceful degradation
  8. Operations: coordinated backups, monitoring and team cost
  9. When to dismantle the polyglot architecture
  10. Common Mistakes and Tips
  11. Exercises
  12. Conclusion

  1. Polyglot persistence, put into practice

The concept was introduced in 03-04 with Martin Fowler's definition:

Polyglot persistence: deliberately using several storage engines within a single system, choosing for each type of data the one that manages it best, instead of forcing everything into a single engine.

There it was an idea. Here it is a production system, and the difference between the two is enormous. In the idea, every piece of data "lives where it fits best" and it all sounds reasonable. In the real system four questions appear that the definition does not mention and that are 90% of the work:

  1. Who is the source of the truth for each piece of data? When two engines disagree, one of them is right by definition, and that has to have been decided before it happens.
  2. With what latency does a change propagate? It is not "instantaneous". It is 50 ms, or 3 seconds, or 15 minutes, and that figure has to be a design decision with a requirement behind it.
  3. What happens if a copy is lost? A derived store that can be rebuilt in 20 minutes is one thing; one that holds the only instance of a piece of data is something completely different.
  4. What keeps working when one piece goes down? Four engines mean four independent ways for the system to fail, and a system where any of the four brings it all down is worse than one with a single database.

This lesson is, at bottom, those four questions answered for VallBici.

  1. The real price, and why the default answer is still a single database

Before the pretty diagram, the bill. In 03-04 we listed it; now we quantify it with what it really costs.

Cost With one engine With four engines
Systems to update and patch 1 4, on different schedules
Backups to design and test 1 4, and coordinating them with each other
Permission and audit systems 1 4, with incompatible models
Independent failure modes 1 4, plus the synchronization failures
Skill sets on the team 1 deep 4, or 2 deep and 2 shallow
Debugging "this value is wrong" Look at one table Work out in which of the four it went wrong and when
Restoring to a coherent instant One PITR Open problem (section 8)
Onboarding a new person Days Weeks

That second-to-last point deserves emphasis. With a single engine, "restore the system to 11:39" is a solved operation. With four, the 11:39 PostgreSQL backup, the 11:41 MongoDB one and the 11:20 Elasticsearch index do not describe the same instant of the world, and you have to decide what "coherent" means in that context.

The correct default answer is still a single database. Not out of conservatism: because the cost of the second piece is permanent and the benefit is usually one-off.

The signals that do justify adding an engine

A new engine is justified when you can fill in this sentence with data: "we have [number] of [data] that requires [property], our current engine gives [measurement] and we need [measurement]". If you cannot fill it in, there is no case. The concrete signals:

Signal Example in VallBici Engine that solves it
Write volume the main engine cannot absorb without degrading 272 M GPS points/year MongoDB
Required latency an order of magnitude below what is achievable Availability in < 5 ms, 60,000 times/day Redis
A functional capability the main engine does not have Search with typos and synonyms Elasticsearch
Genuinely variable structure that produces constant migrations Incident detail by type MongoDB
Ephemeral data whose loss is irrelevant App sessions Redis

And the ones that do not justify it, however often you hear them: "it is what everybody uses", "this way we learn a new technology", "relational does not scale" (said without a measurement), "we want to be modern". Each of those sentences costs, in a system like VallBici, between 20,000 and 60,000 euros a year in operations and team time.

  1. The VallBici architecture

flowchart TB
    subgraph clients["Clients"]
        APP["Mobile app<br/>24,000 people"]
        OPS["Operations and<br/>workshop dashboard"]
        CNC["City council<br/>reports"]
    end

    API["VallBici API<br/>(the one that decides who to ask)"]
    APP --> API
    OPS --> API
    CNC --> API

    API --> RD["REDIS<br/>availability per station<br/>sessions (EXPIRE)<br/>10-minute reservations"]
    API --> PG["POSTGRESQL · source of the truth<br/>subscriptions · trips · charges<br/>bicycles · docks · workshop"]
    API --> MG["MONGODB<br/>telemetry · station profiles<br/>incidents"]
    API --> ES["ELASTICSEARCH<br/>station search<br/>by name and address"]

    PG -->|outbox + publisher<br/>~2 s| RD
    PG -->|outbox + publisher<br/>~2 s| MG
    MG -->|incremental reindexing<br/>~30 s| ES
    PG -->|incremental reindexing<br/>~30 s| ES

    IDM["MUNICIPAL IDENTITY<br/>(shared with BiblioRed)"]
    IDM -.->|OIDC| API
    BIB["BiblioRed<br/>(neighboring system)"]
    IDM -.->|OIDC| BIB

    style PG stroke-width:3px

The thick arrows all point the same way, and that is no accident: PostgreSQL writes toward the others and nobody writes toward PostgreSQL. A system where the synchronization arrows form a cycle is a system where data oscillates, and debugging it is a nightmare. Rule: the synchronization graph must be acyclic.

What each piece contributes

PostgreSQL — the transactional core. What we had in 08-01, unchanged. It is the only engine where money is billed, the only one with referential integrity and the only one with declarative constraints. It is the source of the truth for everything with legal or financial consequences.

MongoDB — telemetry, profiles and incidents. What we had in 08-02, unchanged.

Redis — availability, sessions and reservations. The new piece, and the one that best illustrates the criterion. The app asks "how many bikes are there at each of the 60 stations?" 60,000 times a day. In PostgreSQL that is a SELECT over stations returning 60 rows: it would cost about 2 ms and it would work perfectly. So why Redis?

For two reasons that really are numbers. First, that query fires every time somebody opens the map and every 10 seconds while they keep it open: at peak hours that is 400 queries per second against the same database that is processing unlocks, and competing for connections with the billing transaction is exactly what you do not want. Second, and more important: the 10-minute reservation from exercise 1 of 08-01 and the app session are data that expire on their own, and EXPIRE does in one line what in PostgreSQL requires a cleanup process.

# Availability: one hash per station
redis> HSET stn:12 free 4 free_docks 20 total 24 ts 1749888000
(integer) 4
redis> HGETALL stn:12
1) "free"        2) "4"
3) "free_docks"  4) "20"
5) "total"       6) "24"
7) "ts"          8) "1749888000"

# All the stations at once, in a single round trip
redis> MGET stn:1:free stn:2:free stn:3:free
1) "7"  2) "0"  3) "12"

# App session: expires on its own after 30 days
redis> SET sess:9f3a2b... '{"subscriber_id":8801,"subscription_id":10233}' EX 2592000
OK
redis> TTL sess:9f3a2b...
(integer) 2591994

# Bicycle reservation: expires on its own after 10 minutes
redis> SET resv:bike:417 10233 EX 600 NX
OK
redis> SET resv:bike:417 10999 EX 600 NX     # another person tries to reserve the same one
(nil)                                         # ← NX rejects it: it is already reserved

That last block is the operational gem. SET ... NX is an atomic check-and-set operation: the second person gets nil and knows they arrived too late, with no transactions and no locks. And the reservation expires on its own: there is no cleanup process that can fail. Compare it with the solution to exercise 1 of 08-01 —partial unique index plus implicit expiry in the query— and you will see the same rule solved with two different tools, each idiomatic in its own engine.

Elasticsearch — the search. The real requirement: somebody types "north wharf", "warf promenade" (with the typo) or "staton 12" and expects to find station 12. In PostgreSQL that is ILIKE '%...%' —which does not use an index, as we saw in 06-03— or pg_trgm, which works pretty well. With 60 stations, pg_trgm would be enough and that has to be said. Elasticsearch comes in for a concrete, forward-looking reason: the municipal plan foresees 200 stations and unifying the search with BiblioRed's and other council services' into a single citizen search engine. It is the "functional capability the main engine does not have" signal from section 2 — typo tolerance, synonyms, highlighting, tunable relevance.

The municipal identity, shared with BiblioRed. Vallmar has a single identity provider: the same person logs into BiblioRed to reserve a book and into VallBici to unlock a bike with the same credentials. It is a piece of data shared between two systems of the municipal platform, and here you have to be very strict about one thing: VallBici stores no passwords and no credentials. It stores a subject_id from the identity provider in subscribers and nothing else. If the council changes provider, VallBici changes one column; if VallBici stored credentials, a failure of its own would also compromise BiblioRed.

  1. The master decision table: who rules over each piece of data

This is the most important document in the system. If a team with a polyglot architecture does not have it written down, it does not have an architecture: it has four databases.

Dataset Source of the truth Copies in Sync latency If the copy is lost
Subscribers and subscriptions PostgreSQL Not applicable
Trips (opening and closing) PostgreSQL Mongo (only the _id) ~2 s Irrelevant
Charges and amounts PostgreSQL Not applicable
Bicycle inventory PostgreSQL Mongo (plate, model, type) ~2 s Rebuildable from PG
Dock occupancy PostgreSQL Redis (aggregate counter) < 1 s Rebuildable in 200 ms
App session Redis It is lost: you have to log in again
10-minute reservation Redis It is lost: the reservation is voided
GPS telemetry MongoDB Genuinely lost
Enriched station profile MongoDB ES (name, address, district) ~30 s Rebuildable from Mongo
Basic station data (code, docks, district) PostgreSQL Mongo, Redis, ES ~2 s / ~30 s Rebuildable from PG
Incident detail MongoDB Genuinely lost
Workshop order (accounting fact) PostgreSQL Mongo (the workshop_order_id) ~2 s Irrelevant
Search index Elasticsearch (derived) ~30 s Rebuildable in ~2 min

Three readings you have to take from this table:

First: the left-hand column has a single value per row, always. There is no "both". If two engines could modify the same piece of data, conflict resolution rules would be needed, and that is a far harder problem than anyone wants to have in a municipal system.

Second: there are three rows with "genuinely lost". Telemetry, incident detail, and —partially— sessions and reservations. Those are the ones that need a backup of their own. Everything else is derived and gets rebuilt. A derived value does not need a backup; it needs a tested rebuild procedure. Confusing the two leads to expensive backups of what does not matter and none of what does.

Third: dock occupancy has its truth in PostgreSQL, not in Redis. It is counterintuitive —Redis is the one serving it— and it is the decision that holds up the whole of section 6. Redis has a fast, approximate copy; the truth is in the docks table, protected by its primary key and its UNIQUE. When the two disagree, PostgreSQL wins, no exceptions.

One special case deserves a note: the station appears in four engines. Its basic data is born in PostgreSQL (code, docks, district), its enriched profile is born in MongoDB (photos, accessibility, opening hours), its availability lives in Redis and its searchable text in Elasticsearch. Four engines, a single object in the world. And it works because each field has a single owner: nobody edits the number of docks from MongoDB, and nobody edits the photos from PostgreSQL. The rule is not "each entity in one engine"; it is "each field, one owner".

  1. The four synchronization patterns

How a change gets from PostgreSQL to the others. There are four ways to do it and only one is bad.

Pattern 1 — Dual write (and why it is fragile)

The application writes in both places, one after the other.

await pg.query('UPDATE docks SET bicycle_id = NULL WHERE ...');
await redis.hincrby('stn:12', 'free', -1);        // ← and what if it fails here?
sequenceDiagram
    participant API
    participant PG as PostgreSQL
    participant RD as Redis
    API->>PG: UPDATE ... COMMIT
    PG-->>API: OK
    API->>RD: HINCRBY stn:12 free -1
    Note over API,RD: 💥 the API crashes here
    Note over RD: Redis says 4 bikes · PostgreSQL says 3 · forever

The problem is not that it fails: it is that it fails silently and does not recover. There is no transaction spanning the two engines, so the incoherence stays. And reversing the order does not help: if you write to Redis first and PostgreSQL fails, Redis announces a bike that has not been unlocked.

It is used anyway, in one very specific case: when the copy is rebuildable and the drift is tolerable. Availability in Redis meets both conditions, so VallBici does do a dual write toward Redis… accompanied by a periodic rebuilder that corrects the drift. Dual write on its own, with no safety net, is what is never done.

Pattern 2 — Outbox

The idea: if you cannot have a transaction across two engines, you have a transaction inside one that includes the intention to write to the other.

CREATE TABLE outbox (
    event_id     BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    aggregate    VARCHAR(20)  NOT NULL,    -- 'trip', 'bicycle', 'station'
    aggregate_id BIGINT       NOT NULL,
    type         VARCHAR(30)  NOT NULL,    -- 'trip_started', 'bike_to_workshop'
    payload      JSONB        NOT NULL,
    created_at   TIMESTAMPTZ  NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);
CREATE INDEX idx_outbox_pending ON outbox (event_id) WHERE published_at IS NULL;

The unlock transaction from 08-01 gains one more INSERT, inside the same BEGIN:

BEGIN;
  UPDATE docks    SET bicycle_id = NULL WHERE station_id = 12 AND dock_number = 3;
  UPDATE bicycles SET status = 'in_use' WHERE bicycle_id = 417;
  INSERT INTO trips (...) VALUES (...) RETURNING trip_id;          -- 884213

  INSERT INTO outbox (aggregate, aggregate_id, type, payload) VALUES
    ('trip', 884213, 'trip_started',
     '{"trip_id":884213,"bicycle_id":417,"plate":"VB-0417",
       "model":"Ciclmar E-Vall","type":"electric","origin_station":12}'),
    ('station', 12, 'availability_changed', '{"station_id":12,"delta":-1}');
COMMIT;

And an independent publisher:

-- Runs every 500 ms. SKIP LOCKED allows several publishers in parallel (06-02).
BEGIN;
SELECT event_id, type, payload FROM outbox
 WHERE published_at IS NULL ORDER BY event_id
 FOR UPDATE SKIP LOCKED LIMIT 100;
-- ... writes to MongoDB and Redis ...
UPDATE outbox SET published_at = now() WHERE event_id = ANY($1);
COMMIT;

The exact guarantee the outbox gives, stated precisely: if the business transaction committed, the event is in the outbox table, because it was written in the same transaction. Therefore it will be published: if the publisher crashes, on restart it finds it pending. What it does not guarantee is that it will be published only once: if it writes to MongoDB and dies before the UPDATE, on retry it will publish it again. It is at-least-once delivery, and that is why all the destination writes have to be idempotent — which is exactly why in 08-02 we used the trip_id as MongoDB's _id. The pieces fit together.

Pattern 3 — Change data capture (CDC)

Instead of the application declaring what has changed, the write-ahead log of the engine itself is read. It is the same WAL from 06-01: the file where PostgreSQL notes down every change before applying it, and which there served durability and PITR. Here it is given a second use.

-- Logical publication: PostgreSQL emits the changes of these tables
CREATE PUBLICATION vallbici_cdc FOR TABLE docks, bicycles, stations;
SELECT pg_create_logical_replication_slot('vallbici_slot', 'pgoutput');

A connector (Debezium and the like) subscribes to the slot and produces, for each change, an event with the before and the after:

{ "op": "u", "source": { "lsn": 47118293, "ts_ms": 1749888000123 },
  "before": { "station_id": 12, "dock_number": 3, "bicycle_id": 417 },
  "after":  { "station_id": 12, "dock_number": 3, "bicycle_id": null } }

The advantage over the outbox is that the application does not take part. Nobody can forget to emit an event, not even an UPDATE done by hand by an administrator at 3 in the morning. The drawback is that the events are row changes, not business facts: the event above says "dock 3 went to null", not "person 8801 unlocked an electric bike". Reconstructing the meaning out of the rows is work, and that work is coupled to the schema: changing a column breaks the consumer.

Operational warning that has taken down more than one system: a logical replication slot with a dead consumer makes PostgreSQL retain WAL indefinitely so it can serve it when the consumer comes back. If the consumer has been stopped for three days, the disk fills up and the engine halts. A forgotten slot is a time bomb, and monitoring it (pg_replication_slots) is mandatory.

Pattern 4 — Periodic rebuild

The simplest and the most underrated: every so often, the derived store is recomputed from scratch or compared against the truth.

-- Rebuild the availability of the 60 stations. Runs every 5 minutes.
SELECT s.station_id,
       COUNT(*) FILTER (WHERE d.bicycle_id IS NOT NULL AND d.status = 'operational') AS free,
       COUNT(*) FILTER (WHERE d.bicycle_id IS NULL     AND d.status = 'operational') AS empty_docks
  FROM stations s JOIN docks d USING (station_id)
 GROUP BY s.station_id;
 station_id | free | empty_docks
------------+------+-------------
         12 |    4 |          20
         41 |    0 |          20
        ...
(60 rows · 8 ms)

Eight milliseconds to rebuild the complete state of Redis. With that figure on the table, the conclusion is a strong one: any drift in the counter corrects itself in less than five minutes and nobody gets to notice. When the full rebuild is cheap, incremental synchronization can afford to be imperfect — and that simplifies everything else.

For Elasticsearch, the full rebuild (60 documents, or 200 in the future) takes about two minutes and is done with a parallel index and an alias swap, so the search is never interrupted:

$ curl -XPOST localhost:9200/_aliases -d '{"actions":[
    {"remove":{"index":"stations_v3","alias":"stations"}},
    {"add":   {"index":"stations_v4","alias":"stations"}}]}'
{"acknowledged":true}

The table to remember

Pattern Consistency Complexity Typical latency When to use it
Dual write No guarantee; silent drift Very low Immediate Only if the copy is rebuildable and there is a rebuilder
Outbox At least once, no losses Medium (table + publisher) 0.5-3 s The general case. The default option
CDC At least once, no losses and nothing forgotten High (connector, slot, operations) 0.1-1 s Many consumers or writes outside the application
Rebuild Guaranteed convergence, with delay Low Minutes or hours Safety net for the other three, always

VallBici uses outbox as the main mechanism toward MongoDB and Redis, dual write as a fast shortcut toward Redis for availability, and periodic rebuild of Redis (5 min) and of Elasticsearch (nightly) as a safety net. CDC is discarded today: two consumers do not justify operating a connector and watching a slot. If tomorrow the council wants an analytical store and an alerting system reading the same thing, the decision changes — and that is also correct.

  1. Coherence in practice: 4 bikes in the app, 3 on arrival

Now the concrete problem, which is the one people suffer.

The app says there are 4 bicycles at station 12. The person walks for seven minutes, arrives and there are 3. Is that a system error?

No. And understanding it properly is the difference between designing a distributed system and suffering one. That figure could never have been exact, not even with a single engine: between the instant PostgreSQL read it and the instant the person looked at the screen, 200 ms went by, and in those seven minutes of walking other people have unlocked bikes. Redis's lag (< 1 s) is negligible next to the intrinsic lag of seven minutes.

That is the useful formulation:

The question is not whether the value is stale. It is always stale. The question is whether the lag the architecture introduces is small compared with the lag that already existed.

With that yardstick, the table becomes clear:

Data Intrinsic lag Lag added by the architecture Acceptable?
Free bikes on the map Minutes (the time it takes to get there) < 1 s Yes, comfortably
Station profile Days ~2 s Yes
Search results Days ~30 s Yes
Free bikes at the moment of unlocking Zero No. No lag fits here
Amount billed Zero No

The last two rows are the ones that govern the design.

How you design so that the lag is acceptable

1. The short-lived reservation. It turns an uncertain promise into a firm commitment. The person sees 4 bikes, presses "reserve", and from that moment on has a guaranteed bike for 10 minutes. The reservation is taken in Redis with SET ... NX EX 600, which is atomic — but only after confirming it against PostgreSQL:

-- It is confirmed against the source of the truth, not against Redis's counter
BEGIN;
SELECT d.station_id, d.dock_number, d.bicycle_id
  FROM docks d JOIN bicycles b USING (bicycle_id)
 WHERE d.station_id = 12 AND d.status = 'operational' AND b.status = 'docked'
 FOR UPDATE OF d SKIP LOCKED LIMIT 1;
COMMIT;

If PostgreSQL returns 0 rows, the app says "sorry, somebody just took the last one" — and updates Redis on the way. The fast counter is there to draw the map; the truth is consulted when committing.

2. Confirmation on unlocking. The same principle at the critical moment. The app never unlocks based on Redis's counter. Unlocking is the 08-01 transaction, in full, against PostgreSQL, with its FOR UPDATE SKIP LOCKED. If two people ask for the last bike, one wins and the other gets a correct error. Redis takes no part in that decision.

3. Visible timestamp. The app shows "updated 3 s ago". It is a product decision that changes perception: a value with a declared age reads as an estimate, not as a promise.

sequenceDiagram
    participant P as Person
    participant API
    participant RD as Redis
    participant PG as PostgreSQL
    P->>API: open map
    API->>RD: MGET stn:*:free
    RD-->>API: fast, approximate (< 5 ms)
    API-->>P: "station 12 · 4 bikes · 3 s ago"
    P->>API: reserve
    API->>PG: SELECT ... FOR UPDATE SKIP LOCKED
    PG-->>API: bike 417 — truth confirmed
    API->>RD: SET resv:bike:417 ... NX EX 600
    API-->>P: "bike VB-0417 reserved · 10 min"
    Note over API,PG: Billing NEVER leaves PostgreSQL

Why the billing transaction never leaves PostgreSQL

It deserves a section of its own because it is the rule that admits no exceptions.

Closing a trip touches four things —the trip, the dock, the bicycle and the charge— and all four must change all or none. A failure halfway that left the trip closed with no charge is lost money; one that left the charge without closing the trip is a duplicate charge on the next closing. The atomicity from 06-01 is not a luxury here: it is the requirement.

And there is no transaction that spans four engines. Two-phase commit protocols exist, and in practice they are avoided: they lock resources in every participant until the coordinator decides, and if the coordinator goes down, the participants stay locked. Nobody wants that in the billing path of a municipal service.

The design consequence is direct and has to be respected without exceptions:

Everything taking part in the same business transaction lives in the same engine. If two pieces of data have to change atomically, they are not split up. And if somebody proposes splitting them, the correct answer is to redesign the split, not to invent a distributed transaction.

That is why the outbox is in PostgreSQL and not in an external queue: it is the only way for "the trip was closed" and "the other engines have to be told" to be atomic with respect to each other.

  1. Failure modes and graceful degradation

Four engines are four ways to fail. The design question is not "how do we stop them going down?" —they are going to go down— but "what keeps working when each one goes down?".

A well-designed polyglot system degrades in layers. A badly designed one falls over entirely when the least important piece fails, and then it is objectively worse than a monolith with a single database.

Piece that goes down What stops working What keeps working Contingency plan Severity
Redis Fast map, sessions, reservations Everything else: unlocking, docking, billing, workshop, reports The API falls back to querying availability in PostgreSQL (8 ms, 60 rows); sessions expire and people have to log in again; reservations are disabled Low
Elasticsearch Free-text search Everything else; the station list by district and the map keep going The app falls back to filtering by district and to ILIKE over the 60 stations in PostgreSQL Very low
MongoDB Enriched profile, telemetry, incident detail Unlocking, docking and billing: untouched The profile shows PostgreSQL's basic data; telemetry accumulates in the gateway and is flushed on return; incidents are opened only as a workshop order Medium
PostgreSQL Unlocking, docking, billing, registering subscriptions — the service Map lookup (from Redis, frozen), search, profiles Read-only mode declared; the app warns; no trips are opened Critical

The Redis row is the one that validates the architecture. Redis is the piece that serves the most requests and the one that matters least: if it goes down, the system gets a bit slower and keeps billing. That asymmetry —the most requested piece is the most expendable— is the sign of a good split. The opposite sign, an auxiliary engine that takes down billing, indicates there is data in the wrong place.

What you must not do when MongoDB goes down: write the telemetry into PostgreSQL "provisionally". It sounds helpful and it is a disaster: it creates a second, untested write path, it puts 272 million rows a year into the database that holds up billing, and it leaves data in two formats that somebody will have to reconcile. The correct answer is to accumulate in the gateway with a limit and discard the oldest if it fills up. Telemetry is valuable; it is not worth a billing outage.

And what you must do when PostgreSQL goes down: declare it. A system that keeps accepting unlocks "blind" with a promise to record them later will give away bicycles and bill incorrectly. In read-only mode, the app shows the map and clearly says that trips cannot be started. Failing visibly and honestly is a design decision, not a surrender.

  1. Operations: coordinated backups, monitoring and team cost

The problem of restoring to a coherent instant

Each engine has its own backup strategy:

Engine Strategy Frequency Restore granularity
PostgreSQL Base backup + archived WAL (PITR) Continuous Any instant
MongoDB Filesystem snapshot + oplog Every 6 h + continuous Any instant within the oplog window
Redis No backup of availability; daily RDB of the sessions Daily Approximate, and it does not matter
Elasticsearch None: it is rebuilt Not applicable

And here is the problem that has no perfect solution: restoring to 11:39 does not mean the same thing in the four engines. PostgreSQL can go back to exactly 11:39; MongoDB can go back to 11:39 if the oplog reaches that far; Redis will go back to a state from yesterday that is no longer any good.

VallBici's strategy, and the reasoning:

  1. PostgreSQL is restored to the exact instant. It is the truth. Everything else is defined relative to it.
  2. MongoDB is restored to an equal or later instant. If MongoDB has telemetry for trips that, after the restore, PostgreSQL no longer knows about, those are orphan documents — annoying but harmless. The other way round (Mongo earlier than PG) telemetry for existing trips would be missing, which is detected but not recovered. When in doubt, have too much information in the derived stores, not too little.
  3. Redis is flushed and rebuilt. The 8 ms from section 5 make this decision trivial. Sessions are lost: 24,000 people log in again. It is annoying and it is acceptable.
  4. Elasticsearch is reindexed from scratch. Two minutes.
  5. A full reconciliation is run before reopening the service, and its result is logged.
# Coordinated restore script, abridged
$ pg_ctl stop && pg_restore_pitr --target-time "2026-06-14 11:39:00+02"
$ mongorestore --oplogReplay --oplogLimit 1749893999
$ redis-cli FLUSHALL
$ ./rebuild_availability.sh                # 60 stations from PostgreSQL
$ ./reindex_elasticsearch.sh --alias-swap
$ ./reconcile.sh --report /var/log/vallbici/reconciliation-20260614.txt
[reconcile] orphan bicycles in MongoDB ............... 3
[reconcile] trips with no telemetry .................. 128
[reconcile] divergent Redis counters ................. 0
[reconcile] missing ES documents ..................... 0
[reconcile] RESULT: tolerable divergences · service ready

Those 128 missing telemetry traces are the real cost of the restore, and it is a good thing they show up in a report instead of being discovered by chance six months later.

Minimum monitoring

With four engines, monitoring each one separately is necessary but not sufficient. What is specific to the polyglot architecture is watching what lies between them:

Metric Alert threshold Why
Age of the oldest unpublished event in outbox > 60 s The publisher is down or stuck
Pending rows in outbox > 5,000 They pile up faster than they are published
Divergences in the Redis rebuild > 2 stations The dual write is failing
Elasticsearch reindexing lag > 5 min The search returns stale data
WAL retention by replication slots > 5 GB The time bomb from section 5.3
Orphans detected in the nightly reconciliation > 10 Something is breaking slowly

The first three are given by no engine: they belong to the architecture and have to be instrumented by hand. A team that monitors four engines impeccably and does not watch the outbox queue has a blind spot exactly where this design's characteristic failures happen.

The team cost

The part that does not show up in the diagrams. For VallBici, with four engines, the team needs real competence in:

  • PostgreSQL: modeling, execution plans, transactions, PITR. Deep, non-negotiable.
  • MongoDB: document modeling, aggregation pipelines, indexes, replica sets. Deep.
  • Redis: data structures, expiry, persistence and its limits. Shallow is enough.
  • Elasticsearch: analyzers, mappings, reindexing without downtime. Medium.
  • The synchronization: outbox, idempotence, reconciliation. It is the skill nobody has on their résumé and the one that is needed most.

In a small team this means, in practice, that two people become indispensable and that the August holidays are an operational risk. It is a real cost and it has to be put on the table when the architecture is decided, not discovered afterwards.

  1. When to dismantle the polyglot architecture

This part is almost never written down, and it is the one that saves the most money.

The signals that an engine is surplus

Signal What it indicates
The auxiliary engine holds less data than it cost to deploy It was added out of enthusiasm, not out of need
Half the queries to the auxiliary engine end up querying the main one too The split is wrong: that data belongs together
Nobody has looked at that engine's dashboard in three months It is not solving any visible problem
Every incident starts with "which of the four is the problem in?" The debugging cost exceeds the benefit
The synchronization process has more code than the functionality it supports A classic. That is the moment to stop
The volume that justified the engine has plateaued far below the forecast The premise was false
Only one person on the team knows how to operate it An operational risk greater than the benefit

Applied to VallBici, honestly: Elasticsearch is the candidate. It was justified by a plan for 200 stations and by a unified municipal search engine. If two years from now there are still 60 stations and the unified search engine has not been built, Elasticsearch will be indexing 60 documents that pg_trgm would serve just as well, and it will have cost two years of operations, upgrades and monitoring. The right decision then is to remove it, not to defend it because it is already there.

How you go back, without drama

The order matters, and it is easier than it looks if the engine was derived:

  1. Check that it is derived. If its source of the truth is another engine, it can be switched off without losing anything. If it holds original data, those have to be migrated first and that is another project.
  2. Implement the alternative path in the main engine. For the search: a GIN index with pg_trgm over name and address.
  3. Run both in parallel and compare results for two or three weeks. Log the queries where they disagree and decide whether the difference matters.
  4. Switch the traffic, leaving the old engine on and synchronized.
  5. Wait. Two weeks with no incidents.
  6. Switch it off, and delete the synchronization code. This step is the one that gets forgotten: leaving the publisher writing to a switched-off engine produces log errors that somebody will chase for months.
-- The alternative path for the search, in PostgreSQL
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX idx_stations_search ON stations
       USING gin ((name || ' ' || address) gin_trgm_ops);

SELECT code, name, address,
       similarity(name || ' ' || address, 'warf promenade') AS score
  FROM stations
 WHERE (name || ' ' || address) %  'warf promenade'
 ORDER BY score DESC LIMIT 5;
  code  |           name           |      address       |  score
--------+--------------------------+--------------------+---------
 VB-012 | Station 12 · North Wharf | 14 Wharf Promenade |    0.41
 VB-019 | Station 19 · South Wharf | 88 Wharf Promenade |    0.39

Typo included. For 60 stations, this is enough — and that is exactly the argument. Removing a surplus piece is not admitting a mistake: it is the same analytical capability that put it there, applied with new data.

Common Mistakes and Tips

Mistake 1: not having the table from section 4 written down. Without it, every person on the team has their own idea of which engine rules, and the day they disagree whichever one is looked at first gets "fixed". The table of sources of the truth is the founding document of a polyglot architecture.

Mistake 2: bidirectional synchronization. Two engines writing to each other produce cycles, oscillations and conflicts that have to be resolved with ad hoc rules. Acyclic graph, always.

Mistake 3: splitting a transaction between two engines. If two pieces of data have to change atomically, they live in the same engine. There is no acceptable exception in a system that bills money.

Mistake 4: dual write with no rebuilder. It works 99.9% of the time, and the remaining 0.1% leaves a permanent incoherence that nobody detects. Every dual write needs its safety net.

Mistake 5: believing that "eventual" means "soon". Eventual consistency (03-04) guarantees convergence if there are no more writes and if the mechanism works. A dead publisher means "never". That is why the outbox age is an alert.

Mistake 6: backing up the derived data and not the original data. We saw that Redis and Elasticsearch need no backup; MongoDB, for the telemetry and the incidents, does. Confusing that gets expensive on exactly the day it matters.

Mistake 7: leaving a logical replication slot with no consumer. The disk fills up and the engine halts. It is the silliest and most frequent failure of CDC architectures.

Tip 1: start with one engine and add the second when you have the number. The number: how much data, what latency is needed, what your current engine gives. Without a number, there is no case.

Tip 2: write the rebuild procedure before the synchronization one. If you know how to rebuild the derived store in minutes, incremental synchronization can fail without serious consequences. It is what makes the whole architecture tolerable.

Tip 3: make the derived writes idempotent from day one. A meaningful _id, upsert, $max for the timestamps. "At least once" delivery is the only thing you can get; idempotence is what makes it harmless.

Tip 4: test the outages. Switch Redis off in a test environment and check that a bicycle can be unlocked. The table in section 7 is a hypothesis until it is executed.

Tip 5: review the architecture once a year with the table from section 9 in front of you. Architectures do not simplify themselves.

Exercises

Exercise 1 — The price history and the public dashboard

The city council wants a real-time public dashboard —a website open to the citizens— with: free bikes per station (updated every 10 s), the current day's trips, and last month's heat map. About 3,000 people a day will consult it, with peaks when it is covered in the press.

  1. Decide which engine serves each of the three pieces of data and justify it with the table from section 4.
  2. Does any new piece have to be added? Argue with the criterion from section 2.
  3. Write out what happens to the dashboard when each of the four engines goes down.

Exercise 2 — A new event in the outbox

The workshop wants that, when a bicycle is pulled into the workshop in PostgreSQL, that bike's profile in MongoDB and the Redis counter are updated automatically, and that the app stops offering it.

  1. Write the PostgreSQL transaction that pulls the bike out and emits the outbox event.
  2. Write what the publisher does in MongoDB and in Redis, guaranteeing idempotence.
  3. Explain what happens if the publisher processes the same event twice, and what happens if it processes it 40 minutes late.

Exercise 3 — The identity shared with BiblioRed

The council wants that, when somebody registers with VallBici, the system detects whether the person is already a BiblioRed member in order to offer them a combined fare. BiblioRed has its own PostgreSQL, separate from VallBici's.

  1. List three ways of solving it and choose one, justifying with the lesson's criteria.
  2. Explain why VallBici must not query BiblioRed's database directly.
  3. State who is the source of the truth for "this person is a BiblioRed member" and what happens if the copy goes stale.

Solutions

Solution 1

1. The split:

Dashboard data Engine Justification
Free bikes every 10 s Redis It is exactly the case it exists for: massive reads, approximate value, lag < 1 s against an intrinsic lag of minutes
Current day's trips PostgreSQL, with a 60 s cache It is an aggregate over the source of the truth. 3,000 queries a day with a one-minute cache are ~1,440 real queries: irrelevant
Last month's heat map MongoDB, precomputed The pipeline with $unwind takes seconds and cannot run per request. It is computed once a month with $merge and the dashboard reads the result

2. No new piece. Applying the criterion: is there a volume the current engines cannot absorb? 3,000 daily visits are ~0.03 requests/second on average and maybe 20/s at peak. Redis serves tens of thousands per second. An unreachable latency? No. A missing functional capability? No. With no number to justify it, nothing gets added.

What does have to be added is a 10-second HTTP response cache in front of the dashboard: it turns the press peak into one query every 10 seconds, whatever the number of visits. It is the intervention with the best benefit/cost ratio and it is not a database.

3. Dashboard degradation:

Goes down Dashboard
Redis Free bikes are served from PostgreSQL (8 ms); the dashboard stays whole
Elasticsearch No effect: the dashboard does not search text
MongoDB The heat map disappears; the other two blocks keep going. If the precomputed result is copied to PostgreSQL when it is generated, not even that
PostgreSQL The day's trips freeze; the free-bike map keeps going from Redis, with a "data not up to date" warning

Note that the last row describes a public dashboard that keeps working visually during a core outage. For a dashboard that is the council's public face, that is worth a lot.

Solution 2

1. The transaction:

BEGIN;
  UPDATE bicycles SET status = 'workshop'
   WHERE bicycle_id = 417 AND status = 'docked';    -- if it is in use, 0 rows: abort

  UPDATE docks SET bicycle_id = NULL
   WHERE bicycle_id = 417
  RETURNING station_id;                              -- 12

  INSERT INTO workshop_orders (bicycle_id, type, reason)
  VALUES (417, 'breakdown', 'Rear brake with no travel')
  RETURNING order_id;                                -- 30412

  INSERT INTO outbox (aggregate, aggregate_id, type, payload) VALUES
   ('bicycle', 417, 'bike_to_workshop',
    '{"bicycle_id":417,"order_id":30412,"station_id":12,
      "reason":"Rear brake with no travel","ts":"2026-06-14T07:02:11Z"}');
COMMIT;

The AND status = 'docked' is the guard: if the bike is on an open trip, the UPDATE affects 0 rows and the application must abort instead of pulling out a bike somebody is using. The 08-01 trigger takes care of available_bikes; the outbox event does not repeat it.

2. The publisher:

// MongoDB — idempotent thanks to the filter on workshop_order_id, which is unique
db.incidents.updateOne(
  { workshop_order_id: NumberLong(30412) },
  { $setOnInsert: {
      bicycle: { id: 417, plate: "VB-0417", type: "mechanical", model: "Norvent Urban2" },
      type: "brakes", status: "in_workshop", severity: 3,
      reported_by: { channel: "operator" },
      opened_at: ISODate("2026-06-14T07:02:11Z"),
      station: 12, detail: {}, schema_v: 1 } },
  { upsert: true }
);
# Redis — idempotent because it sets a value, it does not increment it
redis> SREM stn:12:bikes 417
(integer) 1
redis> HSET stn:12 free 3 ts 1749884531
(integer) 0

The key is the kind of operation. updateOne with upsert and $setOnInsert produces the same result whether it runs once or a hundred times. SREM over a set is idempotent by nature. And the counter is set with HSET to an absolute value, not with HINCRBY -1: an HINCRBY applied twice would subtract two. This is the general rule: in a system with "at least once" delivery, set absolute values, not increments.

3. The two scenarios:

  • Processed twice: nothing changes. The upsert finds the existing incident and $setOnInsert touches nothing; the SREM returns 0 and the HSET sets the same value. It is exactly the behavior idempotence must give.
  • Processed 40 minutes late: for 40 minutes the app went on offering bike 417 at station 12. And nothing serious happens, because unlocking confirms against PostgreSQL: whoever tries to take it gets a correct error ("this bicycle is not available"), because bicycles.status = 'workshop' has been set since minute zero. The lag produces a bad experience, not an incoherence. That is the design working: the truth was consulted at the moment of commitment. What should indeed happen is that the outbox age alert (> 60 s) fired forty minutes ago.

Solution 3

1. Three ways and one chosen:

Option How Assessment
A — Direct query to BiblioRed's DB VallBici opens a connection to BiblioRed's PostgreSQL Discarded (see point 2)
B — BiblioRed API queried at registration time VallBici calls GET /members/{subject_id} during registration Correct, but it couples registration to BiblioRed's availability
C — Attribute in the municipal identity provider BiblioRed publishes the attribute bibliored_member: true in the OIDC profile; VallBici receives it in the token Chosen

C wins for three reasons aligned with the lesson: the value arrives at login time with no extra call; the identity provider is already the shared piece between the two systems, so no new dependency is added; and if BiblioRed goes down, VallBici keeps working with the last value from the token. B is still needed as a fallback for people who register in person without going through the identity provider.

2. Why not the direct query. Four reasons, in order of weight:

  1. It couples the schemas. The day BiblioRed renames a column, VallBici stops working, and nobody on the BiblioRed team will know that could happen. It is the worst form of dependency there is: invisible to whoever breaks it.
  2. It breaks the permission model. VallBici would need read credentials on the library's member database. A security incident in VallBici would become an incident in BiblioRed.
  3. It breaks the table of sources of the truth. A system that reads another system's database does not have a contract, it has a habit.
  4. Personal data. VallBici being able to read what books a person reads is a processing of data nobody has authorized. An API contract exposes bibliored_member: true and nothing else.

3. The source of the truth is BiblioRed, and the identity provider is an intermediary that distributes a copy. VallBici stores that copy in subscribers.bibliored_member along with the date on which it received it.

If the copy goes stale —the person cancels their BiblioRed membership and VallBici does not find out— the effect is that they keep enjoying the combined fare until the next renewal. It is a small, bounded financial cost, and the appropriate mitigation is not to synchronize faster, but to expire the copy: the attribute is valid for 30 days and is refreshed at every login. If it has gone more than 30 days without a refresh, the combined fare is not applied at the next renewal. It is the same idea as Redis's EXPIRE, applied to a business value: when you depend on somebody else's copy, give it an expiry date.

Conclusion

You have seen a complete polyglot architecture and, above all, you have seen what it costs. Four engines for VallBici: PostgreSQL with the transactional core from 08-01, MongoDB with the telemetry, the profiles and the incidents from 08-02, Redis with availability, sessions and reservations, and Elasticsearch with the search. And sharing identity with BiblioRed, because in a municipal platform the systems are neighbors, not islands.

What you should take away is not the diagram. It is five ideas, and they work the same with four engines as with two.

The first: each field has an owner. Not each entity, each field. Station 12 lives in four engines and there is no conflict because nobody edits the number of docks from MongoDB nor the photos from PostgreSQL. The table in section 4 is the system's founding document, and a team that does not have it written down does not have an architecture: it has four databases.

The second: what changes together, lives together. The trip, the dock, the bicycle and the charge change in the same transaction, so they are in the same engine. There is no transaction spanning four systems, and the ones that try cost more than they solve. That is why the outbox table is inside PostgreSQL: it is the only thing that makes "the trip was closed" and "the others have to be told" atomic.

The third: lag is not the enemy; unknown lag is. The app saying 4 bikes when there are 3 is not a fault: it is a value with a declared age, and the one second of lag Redis adds is negligible next to the seven minutes it takes a person to get there. What is non-negotiable is that the moment of commitment —reserving, unlocking, billing— is resolved against the source of the truth. Fast counters draw screens; decisions are made against PostgreSQL.

The fourth: you design for the outages, not against them. That Redis can go down without preventing a single charge is the proof that the split is well made. And that contingency table is a hypothesis until you switch Redis off in testing and check that a bicycle can be unlocked.

And the fifth, which is the most uncomfortable: the correct default answer is still a single database. Every engine added costs operations, team skills, failure modes and weeks of learning for whoever joins. VallBici has four because there are numbers behind them: 272 million GPS points a year, 60,000 daily availability queries, a search with typos that ILIKE does not do. If those numbers did not exist, the correct architecture would be PostgreSQL with jsonb, pg_trgm and a cache — and saying that out loud in the design meeting is worth more than any four-box diagram. That is why section 9 exists: knowing how to dismantle a piece when its numbers stop supporting it is the same competence that put it there, exercised a year later with new data.

This lesson closes module 8 and closes the practical journey of the course. You started in 01-01 asking yourself what a database is and why a spreadsheet is not enough; you finish splitting the data of a municipal service across four engines and knowing how to justify every split, every duplication and every synchronization. Along the way you have normalized up to BCNF and denormalized on purpose, you have seen a lost update with your own eyes in two terminals, you have read execution plans and you have designed documents by thinking about the queries first. That is, with reasonable precision, the job. What lies ahead is not more syllabus: it is judgment, and judgment is made with projects and with your own mistakes. Module 9 gathers the material to carry on by yourself —09-01 the books worth keeping at hand, 09-02 the courses and tutorials for going deeper into each engine, and 09-03 the tools people actually work with—. Pick one of the three VallBici cases, set it up on your machine and break it: it is the only part of the course we cannot hand you finished.

© Copyright 2026. All rights reserved