The previous lesson gave Reservalia strategies for making the version handover smooth, but they all rest on the same optimistic assumption: that the failure is noticed while the deployment is under way. Many are not. They show up three hours later, once the canary has been promoted, or only for businesses in one particular country, or only on the first day of the month. On top of that we are still carrying an earlier problem: at Reservalia today, deploying and releasing are the same action, so a half-finished feature cannot reach main. This lesson separates those two things with feature flags, builds Reservalia's go-back button and turns the 68 minutes of time to restore into a target of ten.

1. Separating deployment from release
2. Types of feature flag, expected lifetime and owner
3. A minimal implementation in apps/api
4. Merging incomplete code into main without long-lived branches
5. Flag debt: why they expire and how they are retired
6. Artifact rollback versus roll-forward
7. The hard limit: the database
8. The go-back button: rollback.yml
9. Automatic rollback triggered by metrics
10. Incident recovery: mitigate first, diagnose afterwards
11. Blameless post-mortems
12. From 68 minutes to under 10
13. Common Mistakes and Tips
14. Exercises
15. Conclusion

They are two distinct acts that habit has fused into one:

• Deploying is putting code into production: a technical, reversible act that can happen twenty times a day.
• Releasing is making that code visible or effective for users: a product act, with a date and marketing behind it, that does not have to coincide with the previous one.

When the two coincide, the pathologies Reservalia knows well appear: long-lived branches waiting for the feature to be "complete", enormous deployments loaded with changes, and the impossibility of switching off something that is going wrong without deploying again. A feature flag (or toggle) is simply a condition in the code whose value is decided at run time, outside the artifact:

The artifact reservalia/api:a3f9c21 contains both implementations. Which one runs no longer depends on which image is deployed, but on a value that can be changed in seconds and with no pipeline. That turns "switch off the broken feature" into a ten-second operation instead of a deployment.

Not all flags are the same, and treating them as if they were is the number-one cause of a flag system rotting. Reservalia adopts this classification:

The two most ignored columns are the important ones. Expected lifetime distinguishes what has to be deleted from what stays: a release flag that has been in the code for eight months is debt, an eight-month-old kill switch is a tool. The owner prevents the orphan flag, the one nobody dares touch because nobody knows what happens if it is switched off.

You do not need a commercial platform to get started. Reservalia stores its flags in its own database:

The evaluation module lives in apps/api/src/flags/index.ts:

1. An unknown flag returns false. The default value is always the old behaviour: if the table cannot be read or somebody mistyped the key, the system behaves as it did before the change, not after it.
2. The allowlist lets you enable the feature for specific businesses — the three customers who asked for the beta, or the internal test business — without touching the global percentage.
3. The percentage split is computed with a hash, not with a random number. That is the difference between a progressive rollout and chaos: Math.random() < 0.1 would give a different result on every request and the same business would see the feature appear and disappear.
4. By including the key in the hash, two flags at 10% do not affect the same businesses; if the hash were only of the businessId, the same "chosen ones" would always come up.
5. The 30-second cache avoids one query per request. The price is that a change takes up to half a minute to propagate to all the tasks: acceptable for a kill switch, and something you need to know before an incident.
6. Evaluation is per business, not per user, and that is a domain decision: at Reservalia every employee of the same salon must see the same schedule. Enabling per user would mean two receptionists seeing different slots, which is exactly the kind of irreproducible incident nobody wants.

Lesson 02-07 established that long-lived branches are the enemy of continuous integration: the longer they live, the more they diverge and the more painful the merge. But the team had a legitimate objection: "I cannot merge a half-finished feature".

Flags dissolve that objection. Diego can merge calculateSlotsV2 into main on day one, with the schedule_new_calculation flag off: the code travels to production on every deployment, it compiles, it passes tsc --noEmit and its unit tests run in CI, but no user executes it.

And a practical rule: the tests must cover both branches of the flag; in CI the critical set is run with the flag forced to on and to off, because a flag whose active branch is never tested is dead code that will wake up at the worst possible moment.

Every release flag adds a fork to the code. With five flags coexisting there are up to 32 possible behaviour combinations, and nobody tests 32 combinations. Flags that are not retired produce concrete symptoms: unreadable code, tests that depend on an implicit configuration, and the terror of touching something that "we do not know whether it is still in use".

Reservalia establishes three rules: every release flag is born with a mandatory expires_on, typically 30 days out; a weekly job lists the expired flags and opens an issue assigned to their owner — it does not delete them, because asking for a human action is different from breaking production on a Tuesday; and retiring the flag is part of the task, not a future task: on the board, a feature is not done until its flag has disappeared from the code.

Retirement has an order that avoids nasty surprises: first the flag is set to 100% and left there for a few days, then the condition is removed from the code leaving only the new branch, and only at the end is the row deleted from the table. The other way round — deleting the row first — the flag starts evaluating as false and the already-released feature disappears in an instant.

When something goes wrong in production there are two paths, and choosing by reflex is a mistake.

A mature team does not always choose the same thing. The rule Reservalia adopts: if the impact is serious — users who cannot book, 5xx errors, data loss — always roll back; the fix is thought through afterwards, calmly and without production on fire. If the impact is minor — a badly translated string, a crooked icon — roll forward, because a rollback is also a change and it can also fail. There is one condition that makes a fast rollback possible and it has been paid for since 02-06: immutable artifacts identified by digest. Going back to b7e2d10 does not mean rebuilding anything, it means pointing the service at the digest still sitting in ECR; rebuilding would give a different version from the same sources, and that is not a rollback, it is a lottery.

An artifact rollback is reversible. The database migration that came with it is not. If version c1d4a55 included a migration that renamed the column duration to duration_min, going back to b7e2d10 leaves code running that queries a column that no longer exists: the container rollback takes three minutes, the schema one may not exist at all.

From that comes a rule that governs the whole design: a migration must be compatible with the previous version of the code and with the next one. You achieve that with the expand and contract pattern we saw in 03-04, applied to data: add the new column, write to both for a while, migrate the reads, and only much later remove the old one. As long as that is respected, any deployment is reversible. The full detail — migrations in the pipeline, their order relative to the deployment, long migrations and locks — is lesson 04-06, Databases in the Pipeline: Safe Migrations.

Nuria writes the workflow that was missing. Her design requirement is explicit: anyone on call, with no knowledge of AWS, must be able to revert in under five minutes from their phone.

The decisions that make this work under pressure are these:

1. workflow_dispatch with typed inputs. The type: choice removes any possibility of typing production instead of prod at three in the morning, and the mandatory reason guarantees the record is useful for the post-mortem.
2. The same concurrency group as cd.yml. A rollback and a normal deployment cannot overlap; the second one waits instead of treading on the first.
3. environment: ${{ inputs.environment }} reuses the protection rules. Careful: if prod requires a reviewer's approval, the rollback will require it too. Reservalia solves this with a broad list of approvers so somebody is always available; blocking the rollback behind an approval that is hard to get is worse than the original failure.
4. Verify the artifact exists before touching anything. Failing fast with a clear message is better than leaving the service half-updated. 5. The same deploy.sh as the normal deployment is reused, which starts from the current task definition and only swaps the image for the given digest: this way the rollback does not revert legitimate configuration changes made since then, and the emergency path uses code that is already exercised daily. 6. The smoke test against /version turns "I think it has gone back" into "it is serving b7e2d10"; without it, the rollback is a hope.

Marta adds a practice that looks trivial and is not: rehearsing the rollback once a month in staging, timed. A button nobody has ever pressed is not a button, it is an ornament.

During a canary there is no need to wait for somebody to look at a dashboard; the pipeline itself can decide:

The key pattern is if: failure(): the abort step runs precisely when something has gone wrong, and it returns the canary weight to zero without waiting for a human. At Reservalia, check-metrics.sh compares the canary group with the stable one on 5xx rate and p95 latency during the ten minutes of watching. Two cautions: the automation needs thresholds with headroom, or a spike of noise will revert healthy deployments and the team will end up disabling it; and the manual path must always exist, because if the automation fails, rollback.yml is still there.

A healthy continuous deployment does not boast about never failing: it boasts about recovering fast. The order matters and it is counter-intuitive for anyone coming from a "you must understand the problem before touching anything" culture.

Mitigate first, diagnose afterwards. While the cause is being investigated, users still cannot book. Switching off the flag or reverting the artifact stops the damage and buys back the time you need to think: the root cause will still be there in two hours; the angry customers, not necessarily. Reservalia fixes three roles during an incident: whoever coordinates (decides and does not type), whoever operates (executes the actions) and whoever communicates (informs support and the affected businesses). In a team of three the roles can overlap, but coordination cannot be missing: without it, two people run contradictory mitigations at the same time. And communication has a simple rule: little and early beats late and complete; a message after five minutes saying "we are investigating problems creating appointments" is worth more than a perfect report an hour later.

A blameless post-mortem starts from one premise: if a person could break production with a reasonable action, the problem is the system that allowed it. Looking for culprits produces concealment, and concealment produces incidents that repeat. Reservalia uses this template, in docs/postmortems/:

Two sections are usually missing and they are the most useful: "what went well", which stops you dismantling defences that did work, and actions with an owner and a date, because a post-mortem with no assigned actions is a literary exercise. And one rule: the actions go on the same board as the rest of the work, or they will not get done.

Time to restore is not a magic number: it is the sum of three segments, each attacked with a different tool.

It is worth pointing out which segment gets the biggest improvement: detection. You can have the best rollback button in the world and still take half an hour if nobody realises there is a problem; that segment is precisely the one the next lesson takes care of closing. And there is an even better case: when the failure sits behind a flag, execution drops to seconds and the total comes in at around five minutes.

Mistake 1: using flags as permanent configuration. If flags accumulates 60 keys of which 50 are release flags and nobody retires them, the code becomes unauditable. Mistake 2: flags with no owner and no expiry, which nobody dares switch off years later. Mistake 3: evaluating at random instead of by hash, so the same business sees the feature appear and disappear between requests. Mistake 4: not testing the active branch of the flag; CI goes green and the feature blows up the day it is switched on.

Mistake 5: a rollback that rebuilds the image from the previous commit. That is not going back to a known version, it is building a new one under pressure: always deploy the digest. Mistake 6: never rehearsing the rollback, and discovering that the IAM role expired right in the middle of the incident.

Tip 1: make the default value of every flag the old behaviour, so a read failure is harmless. Tip 2: record every flag change with author and date, because during an incident the question "what changed?" includes flags, not just deployments. Tip 3: document the three emergency commands in the README — switch off a flag, launch rollback.yml, set the canary to 0 — where whoever is on call will find them in twenty seconds.

Classify these four Reservalia flags by type, propose an expected lifetime and an owner, and say which one must never be retired: payments_stripe_v2 (a new payment gateway replacing the old one), sms_reminders (sending SMS messages with a per-message cost), metrics_dashboard_beta (a dashboard available only to customers on the advanced plan) and experiment_suggested_times (two ways of ordering the proposed slots).

It is 22:40. The deployment of c1d4a55 twenty minutes ago has pushed the error rate of the appointment creation endpoint to 30%. The previous version was b7e2d10. The change included a migration that added the column reminder_sent_at. Describe the exact steps in order and justify whether you choose rollback or roll-forward.

Same scenario, but the migration renamed duration to duration_min. Is your plan still valid? Describe what you would do and what should have been done differently weeks earlier so that this scenario never existed.

Solution 1. payments_stripe_v2 is a release flag: its purpose is to replace the old gateway, so it has an expiry date (a few weeks, perhaps longer given how delicate payments are), its owner is whoever develops it, and it is retired once 100% has been stable for a while. experiment_suggested_times is an experiment flag: owned by product, it lives as long as the measurement lasts and ends with a variant being chosen. metrics_dashboard_beta is a permissions flag: its owner is product/business, it is permanent and it is not really a temporary flag but an access-rights rule tied to the contracted plan. sms_reminders is operational (a kill switch) and it is the one that is never retired: since SMS messages cost money and depend on an external provider, Nuria needs to be able to switch them off on the fly if the provider degrades or if the cost spikes. A useful nuance: payments_stripe_v2, although it is a release flag, is worth keeping for a sensible period after 100% because it effectively acts as a kill switch for the new gateway.

Solution 2. Rollback, without hesitation: the impact is serious (a third of appointment creations fail), the previous version was healthy and the migration is compatible — adding a column does not bother b7e2d10, which simply ignores it. Steps: (1) declare the incident and name a coordinator; (2) if the change sits behind a flag, switch it off, which is ten seconds against the five minutes of a rollback; (3) if it does not, launch rollback.yml with environment=prod, sha=b7e2d10 and the reason; (4) confirm with the smoke test that /version returns b7e2d10 and check that the error rate drops; (5) inform support and the affected businesses; (6) the next day, diagnose calmly, fix, add the missing test and deploy forwards; (7) post-mortem with assigned actions. The new column is left orphaned for a while, which is harmless.

Solution 3. No, the plan is no longer valid: b7e2d10 queries duration, which no longer exists, so the rollback would trade a 30% error rate for a 100% one. The real options are worse: an urgent roll-forward with a fix written under pressure, or reverting the schema as well — renaming the column back — at the risk of losing writes made in between. The sensible thing is to mitigate by another route (switch off the flag if there is one, degrade the affected feature) while the forward fix is prepared. What should have been done weeks earlier is to apply expand and contract: deployment 1, add duration_min and write to both columns; deployment 2, read from duration_min; deployment 3, weeks later and with everything stable, remove duration. With that sequence, each individual deployment is reversible and the scenario in the question never arises. That is exactly the territory of lesson 04-06.

Separating deployment from release changes the nature of the risk. Feature flags let Reservalia merge incomplete code into main without long-lived branches, enable per business with a stable hash-based split, and switch off on the fly whatever breaks — as long as every flag is born with a type, an owner and an expiry date, because flag debt is real. The artifact rollback by digest, made concrete in rollback.yml with workflow_dispatch, returns production to a known state in minutes, and the automatic metric-driven rollback does so without waiting for a human during a canary. All of it with one hard limit worth keeping in mind at all times: if the database migration is not compatible, no rollback will save you.

With this, two of the three segments of Reservalia's time to restore are solved: the decision, by a written rule, and the execution, by a rehearsed button. The biggest one remains, detection: today Reservalia finds out production is unwell because a customer calls. And there is a twin gap: with no production signal, neither the automatic canary nor the deployment budget has anything to base a decision on. The next lesson, Monitoring and Feedback, closes the loop with the three pillars of observability, the four golden signals, SLOs with an error budget and the automatic feeding of the DORA metrics from the pipeline itself.