Technology & product
The hidden cost of undo
Restoring code does not necessarily restore data, expectations or trust. A useful experiment needs an exit path designed before it begins.
A switch can return; an experience cannot
Imagine a notes app enabling automatic archiving for a small group. When the feature misclassifies notes, the team turns it off. The old interface returns and servers remain healthy. The deployment has been rolled back. Yet some files have moved, collaborators have received notifications, and a person who could not find their work has reorganized it manually. Yesterday’s code is back. The user’s day is not.
This hypothetical case separates two jobs: stopping new effects and undoing effects already produced. A switch usually guarantees only the first.
Our testable proposition is that, when an experiment changes existing user state, teams can overestimate reversibility by measuring system recovery while omitting the work required to restore the user’s working situation. Include that labor, and some apparently cheap experiments become expensive.
Four different recoveries
Execution is the first layer: has the new behavior stopped? State is the second: can data, permissions, ordering and subscriptions be restored? Coordination is the third: can actions taken by other people in response also be reversed? Expectations are the fourth: will the next click still mean what a user thinks it means?
Not every experiment touches every layer, and exact restoration is not always necessary. A color change may leave almost nothing behind. An automated email cannot be unread by deleting the sending log. Calling both “rollback supported” conceals the difference that matters.
A reversibility claim therefore needs an object: reversible for whom, to which state, within what time, and at whose expense? “We restore service; customers redo their settings” describes a recovery path. It does not describe a costless undo.
An old version is not necessarily an exit
AWS describes a concrete deployment hazard: new software may write data that old software cannot read. Redeploying the old code can then fail. Its discussion of a staged compatibility transition also explains why rollback still has limits.[1]
That engineering account supports a narrow claim about state compatibility. It is not an experiment about trust and does not establish that every product rollout strategy works.
A useful product analogy follows: an exit is a capability to maintain, rather than an old artifact to preserve. Keep original folder membership when reorganizing files. Preserve source records before merging contacts. Offer an automatic rule as a suggestion before letting it modify existing work. These examples put the return path into the design before the next action makes it harder to recover.
A smaller rollout can hide a larger mistake
Google’s SRE workbook describes canarying as a partial, time-limited deployment followed by evaluation. It discusses comparing versions to detect defects and notes that its simple model does not cover effects such as data leaks.[2]
“Only one percent of users” therefore cannot by itself justify an experiment. Irrecoverable loss for one person is not less severe because ninety-nine others were unaffected. This distinguishes the reach of an error from its severity; it does not dismiss gradual rollouts.
Delayed discovery creates another blind spot. Archiving may produce no immediate error, while a missing document becomes important next week. A same-day success metric would confuse absent evidence with evidence of safety. The observation window should reflect when harm can become visible, rather than only how quickly the team wants to ship.
Counterexample: preserving every exit can stop progress
Reversibility is not an unlimited obligation. Keeping every historical version consumes resources and can retain information that should be deleted. Asking for confirmation on every automated action can erase automation’s benefit. During an active security incident, waiting for a perfect return path may prolong exposure.
Some changes are deliberately final, such as a clearly explained final submission or a permanent deletion the user has knowingly confirmed. Clear boundaries and an auditable outcome can matter more than an undo button that cannot honor its promise.
The standard should be proportionality, not a perfectly untouched past. A disposable preview may simply be regenerated. A user’s only copy of important material warrants a different investment. An urgent fix may first contain damage and then provide repair and explanation. The relevant distinction is the consequence of being wrong.
How to test what undo is worth
Return to the archiving example. Before touching originals, rehearse on an isolated copy of permitted test material. Specify which locations, sharing relationships and visibility settings must survive. Run the rule, introduce manual edits and collaborator actions, then reverse it. Compare final state with the intended result, rather than merely checking that a switch is off.
Record time to stop new effects, existing changes restored automatically, repairs requiring human work, and effects that cannot be erased, such as notifications already sent. Count the affected person’s labor as well as the team’s. Closing an internal ticket is not necessarily the end of recovery. This is a proposed method, not a test we have performed.
To examine whether recovery design reduces total cost, consenting testers could handle equivalent failure scenarios in an isolated environment, comparing a provenance-preserving design with a switch-only design. Measure completion, omissions and total repair time. Assign groups or rotate order to reduce task familiarity effects. A small rehearsal can expose mechanisms and obvious failures; it cannot establish a production incident-rate reduction.
An experiment should turn failure into knowledge
An experiment earns its cost by producing information for the next decision. If customers silently repair the damage while the team records a successful rollback, both cost and learning are hidden. The next decision will price the same exit too cheaply.
A more honest conclusion has two parts: what did we learn, and who lost what while learning it—and how far did recovery actually get? One describes information gained. The other describes its price.
Fast action does not require believing that everything can be tried again. It requires knowing which consequences can be reversed, which can only be repaired, and which should be prevented before they occur.
Sources & scope
An original editorial analysis. The sources describe engineering practice, not controlled evidence for our product thesis. The product scenarios are hypothetical; the four-layer framework and evaluation method are proposed here, not claimed as validated in our products.
京公网安备11010802050788号