By KYN AI Advisory Team — AI implementation specialists, Singapore
Runbook decay is the gap between what a runbook says and what the live system now does. An operations agent that follows a stale runbook acts on assumptions that stopped being true, and nothing tells it. The fix is not a quarterly review. It is a check inside the execution path: before each step, the agent compares the runbook's assumption with the live system and stops when they disagree.
Why does a runbook go stale?
A runbook is written against a system as it existed on one day. It names an endpoint, a field, a file format, a chart-of-accounts code, an approval threshold. The system then changes on its own schedule. A vendor renames a field. A bank alters a statement layout. A policy owner lowers a limit and tells nobody in operations.
A person following the runbook often notices something is off. An agent does not, unless you build the noticing. It executes the written step and reports success.
Review cadence does not close this. Drift happens between reviews. A calendar reminder cannot catch a change that lands on a Tuesday.
How does drift show up in practice?
Drift rarely fails on the first bad record. It waits for volume, then arrives as one large problem.
| Domain | The stale assumption | How it surfaces | |---|---|---| | System integration | A master record (item, client, account) matches the external format | A batch is submitted, then rejected in bulk | | Bank reconciliation | A foreign-currency charge posts at the receipt amount | Exact-amount matching quietly misses charges settled a day later at another rate | | Content pipeline | A topic in, article out, publish step is enough | Output drifts into generic text or unverified claims within a week | | Approval step | A threshold from last year | Items route to the wrong person or skip review |
The common thread is silence. Each failure looks like normal operation until enough of it accumulates.
Why does testing not catch it?
A vendor sandbox answers the questions the vendor thought of. It accepts edge cases that the live system rejects. A runbook checked against its own document or a staging copy proves only that the document agrees with itself.
The reliable test is a read-only run against real historical data from the live system before anything writes. We use this on integrations: replay real records, compare what the system would accept with what the spec says, and record every difference. Build the new work in a separate full copy of the environment, so a mistake while probing cannot touch the system still running on the old assumptions.
What does a pre-flight validation layer check?
Put the check immediately before the action, not after it.
- Preconditions against live state. Confirm the field, endpoint, status code or threshold the step relies on still exists and still has the expected value.
- Read before write. Query the current state, compare it with the runbook, and proceed only when they agree.
- Tiered matching. Match on the primary field first, a related identifier second, a named fallback third. A single brittle rule either passes everything or fails everything.
- Tolerance gated by a second signal. Allow expected variance, such as an FX settlement gap, only when an independent signal confirms it, for example the same vendor and booking reference. Never loosen every check at once.
When any check fails, the agent stops and says which assumption broke. It does not proceed on the belief that the document is close enough.
Why should an agent flag uncertainty instead of guessing?
A confident wrong action costs more than a held one. In reconciliation, a false match erodes trust and takes longer to find than an item left in a review queue. A runbook step run on a drifted assumption is the same error: it looks like success, and the consequence appears downstream.
Two behaviours to build in:
- Ask only when something is genuinely ambiguous, such as a currency mismatch or a missing quantity. Never guess a figure that reaches a client document.
- Report what could not be resolved and why, rather than dumping every doubtful item on a person without explanation.
How do you keep a person in the loop without slowing everything down?
Put the escalation where the person already works. A draft that lands in an existing approval queue takes seconds to clear. A new inbox gets ignored.
Bound the loop. If a step fails its pre-flight check, allow a small fixed number of retries, then hand it to a named person with the failing assumption attached. An unbounded retry loop stalls silently. A bounded one fails visibly.
Outbound messages, money movement, filings and deletions always wait for a named person, whatever the agent concluded.
How do you measure runbook health?
The validation layer produces its own signal. Track three counts per runbook step:
- Drift catches. How often pre-flight found an assumption that no longer matched live state.
- Escalation rate. How often a step went to a person instead of completing. A rising rate is the earliest sign a step needs rewriting.
- Retry rate. Repeated retries on one step mean the playbook is wrong, not the execution.
Review these as a standing weekly report, not at the next audit. One manufacturer we work with receives a 06:30 report on WhatsApp for each department, at roughly S$0.05 per report. A drift summary fits the same channel.
What does a runbook step look like with validation built in?
The example below is illustrative. It shows one step from a month-end routine: importing a bank statement file into the ledger.
| Part | Content | |---|---| | Step | Import the bank statement CSV for the month | | Assumption 1 | The file has the columns date, description, debit, credit, balance in that order | | Assumption 2 | Dates use one format throughout | | Assumption 3 | The opening balance equals last month's closing balance | | Pre-flight check | Read the header row; parse ten sample dates; compare balances | | If a check fails | Stop, name the failed assumption, attach the first three offending rows | | Retry | None for layout changes; one for a file that failed to download | | Escalation | Named finance lead, with the failing assumption in the message | | Action | Import, then compare row count and total with the file | | Post-check | Row count and totals match, or the step is marked failed |
Each assumption in the runbook has a check beside it. A bank that adds a column or changes a date format makes the pre-flight fail on the first file, not after the ledger has absorbed a month of misaligned rows.
How do you turn an existing runbook into checks?
Take a runbook you already have and work through it in five passes.
- Underline every noun the step depends on. A field name, a file layout, a threshold, an account code, an endpoint, a person's role.
- Write the assumption as a sentence. "The vendor code is 8 characters." "Invoices above the limit go to the finance director."
- Ask how you would know if it were false. Each sentence needs a query, a comparison or a count that can say yes or no.
- Place the check before the action. If the check can only run after the action, make the action reversible or hold it.
- Decide what happens on failure. Retry, escalate or stop. Name the person.
A runbook of twenty steps usually produces thirty to forty checks. Most are small. A few are the ones that would have caught last quarter's incident.
Which assumptions are worth checking first?
Not every assumption deserves a check. Rank by the cost of being wrong and the chance of change.
| Assumption | Chance it changes | Cost if wrong | Check priority | |---|---|---|---| | File layout from an outside party | High | High, bulk errors | First | | Approval threshold or policy limit | Medium | High, wrong routing | First | | Third-party endpoint or field name | Medium | Medium to high | Second | | Internal naming convention | Low | Low | Later | | Contact for escalation | Medium | Medium, delays | Second |
Anything that comes from outside your company changes without telling you. Start with those.
How do you run a read-only trial?
A read-only trial replays the runbook against real historical data without writing anything. Use it whenever a runbook is new, changed or revalidated.
- Copy last month's real inputs into a separate environment.
- Run every step in a mode that reads and compares but does not write to the live system.
- For each step, record whether the pre-flight passed, what the agent would have done and what a person actually did that month.
- List every difference. Each one is either a runbook error, a data error or a rule the person applied that was never written down.
- Fix the runbook or add a check, then repeat until the differences are explained.
The third category matters most. People carry rules in their heads that no document holds. A read-only trial is where those rules come to light.
What edge cases cause false alarms?
A validation layer that fails too often gets switched off. Watch for these.
- Expected variance. Foreign-currency settlements differ by a small amount. Allow a tolerance only when an independent signal, such as the same reference, confirms the match.
- Month-end timing. Items posted on the last day appear next month. Check against a date window, not a single date.
- Planned changes. When a vendor announces a new layout, mark the change in advance so the check expects it.
- Partial outages. A source that returns an error should read as "unknown", not "changed". Retry, then escalate as an outage.
- Holiday calendars. An approver who is away may need a deputy. Check that the named person is reachable before routing.
Track false alarms alongside real catches. If more than a small share of alarms are false, tighten the check.
Who owns the runbook?
Every runbook needs a named owner and a named deputy. The owner answers three questions each month: which checks fired, which were wrong and which assumptions were added. The owner also decides when a check is retired. A check nobody owns becomes noise, and noise is ignored.
Store the runbook as a versioned document. Each change records who made it, when and why. When an agent acts, the log records which version it followed, so a reviewer can tell whether an error came from the runbook or from the execution.
What does a drift report look like?
A short weekly message works better than a dashboard nobody opens. A useful one has five lines:
- Steps run this week, and steps completed without escalation.
- Checks that failed, with the assumption and the step.
- Steps that went to a named person, and how long each waited.
- Steps with repeated retries.
- Runbook versions in use.
Send it to the owner and the deputy. If the same step appears three weeks in a row, schedule a rewrite for that step.
How should you phase the rollout?
Do not add checks to every runbook at once. Phase the work so that each stage teaches you something.
- Pick one runbook that touches an outside party and runs at least monthly. Month-end imports and supplier statement handling are common choices.
- Add checks and run in observe mode for one cycle. Checks log what they would have done, but the old process still runs.
- Compare. Count the failures the checks caught that the old process missed, and the false alarms.
- Switch to enforce mode for that runbook. A failed check now stops the step and notifies the owner.
- Copy the pattern to the next runbook, reusing the check templates and the escalation route.
After three runbooks, you will have a library of checks for file layouts, thresholds and references that new runbooks can reuse. The first runbook takes the longest. The third takes a fraction of the time. Record the time each runbook takes to convert, so you can plan the next one and judge whether the effort is paying back in fewer escalations and fewer corrections.
Where KYN fits
KYN builds operations agents on cloud infrastructure in the client's own account. Each agent has its own restricted account, every action is logged with the actor, and anything that sends, pays, files or deletes waits for a named person. For a manufacturer, eleven systems became three on one AWS account, and a ledger of 3,309 reconciled rows was loaded with zero duplicates.
The first 14 days are a review: an inventory of your stack and interviews with the team. A full build ships in under 90 days. See operations systems and security.
Frequently asked questions
Is runbook decay a documentation problem?
Partly. Better documentation helps, but the systems the runbook describes change between reviews. The durable fix is a validation step in the execution path that checks assumptions against live state before each action.
What should an agent do when a pre-flight check fails?
Stop that step, record which assumption failed and what it found instead, and send it to a named person. Allow a small fixed number of retries first if the failure could be transient. Do not proceed on the old assumption.
Can a sandbox replace a read-only run on real data?
No. A sandbox passes the cases its author tested and accepts edge cases the live system rejects. A read-only replay of real historical data shows what the live system actually enforces, before anything writes.
Checked 30 October 2026.
Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.
