By KYN AI Advisory Team — AI implementation specialists, Singapore
An ops agent that corrects infrastructure drift without recording why is a risk, even when the fix is right. The state converges, but nobody can say what differed, whether the fix was the correct one, or whether the drift will return. A diff records what changed. It does not record the decision. The remedy is an audit trail that stores the evidence, the action and the outcome for every divergence the agent touches.
What is state divergence?
Divergence is the gap between two versions of the truth: declared state against running state, a plan against what was applied, a permission policy against what was granted six deploys ago. Reconciling them is the job of an ops agent.
The danger is not failure. It is silent success: the agent clears the signal it was watching and leaves no trace of what was different.
Scope note: this article applies patterns from financial reconciliation and logging work to infrastructure as illustration. Verify any tool-specific behaviour against your own stack before you build.
How does drift compound across layers?
Infrastructure has several layers, and an agent at each one can clear its own signal while the cause sits elsewhere.
| Layer | What the agent sees | What it may do | What may be hidden | |---|---|---|---| | Orchestration | A service crash-looping | Restarts it | A bad configuration or a recent unreviewed change | | Infrastructure as code | Drifted resource settings | Re-applies the plan | A manual change made for a reason | | Identity | A permission denied | Widens the grant | A dependency declared wrongly | | Scaling | Memory pressure | Adds replicas | A limit set too low |
Each action resolves its own metric. The divergence does not go away. It moves, and no record links the symptom to the layer where it began.
Why do logs and diffs fall short?
A log says what happened. A diff says before and after. Neither says why the agent decided the change was warranted, what evidence it weighed, or what it rejected.
When a person makes a change, you can ask them. When an agent makes it, the decision record is the only explanation. Without it, "the state changed" cannot become "we understand why it changed and can trust it."
What should the audit trail capture?
Take the model from a ledger. A double-entry accounting system gives every transaction a full change history and reconciles each one. Apply that to remediation.
- Divergence detected. The two states, the difference, and the evidence behind the finding.
- Decision. The rule or signal that justified action, and the options considered.
- Action. What was changed, by which agent account, at what time.
- Outcome. The state after, and whether the original signal cleared.
- Flagged cases. Items sent to a person, with the resolution, so the same ambiguity is not re-argued next time.
- Query access. A history you can search after the fact, not only a current-state dashboard.
Action-level logging is not a finance-only demand. We build full interaction logging into agent systems as a baseline, not an add-on.
What does a symptom fix without a record look like?
Three illustrative cases.
- An agent adds replicas to relieve memory pressure. The misconfigured limit stays and is never logged as a discrepancy.
- An agent widens a permission to unblock a deploy. Nobody records whether the narrower permission was right and the request was the real fault.
- An agent keeps restarting a crashing service. It never reports that the crash pattern is new, or tied to a recent change.
In each, the watched metric returns to normal. That is the hazard: the system looks healthy while the cause persists.
How should the agent treat uncertainty?
Follow the reconciliation rule: a false match is worse than an open item. If the agent is not sure that a divergence has one cause, it should not remediate. It should log what it found, state what it could not resolve, and pass it to a named person.
A short list of actions an ops agent may take alone, such as restarting a stateless service, should be explicit. Everything that widens access, deletes data, moves money or changes production configuration waits for a named person.
How do you detect repeat drift?
Reuse the trail. Count divergences by resource, by rule and by week. A resource that diverges every week means the fix addresses the symptom. Raise those to the owner as a root-cause task, with the log as evidence.
Track:
- Divergences found, auto-fixed and escalated, per week.
- Repeats on the same resource within 30 days.
- Median time from detection to a person's decision on escalations.
What does one audit record look like?
The record below is illustrative. It shows the fields for a single divergence that an agent handled.
| Field | Value | |---|---| | Record ID | Unique, sortable | | Detected | Date and time | | Resource | The item that diverged | | Declared state | What the plan or policy says | | Running state | What was found | | Evidence | Link to the query result or snapshot | | Rule applied | The rule name and version | | Options considered | Fix, escalate, ignore | | Action | What the agent changed, or "none" | | Actor | The agent's own account | | Outcome | Signal cleared, or still present after the check | | Escalated to | A named person, if any |
If the agent restarts a service and the memory signal clears, the record shows the restart and the outcome. If the same service appears again within a week, the history shows the pattern, and the owner can see that restarting does not fix it.
How do you introduce the trail without slowing the agent?
- Start with writes. Log every change the agent makes before logging every check. Changes are the evidence a reviewer needs.
- Write in the same step as the action. A log written after the fact can be lost if the process fails between them. Write the intent first, then the outcome.
- Keep it append-only. Records are added, not edited. A correction is a new record that points to the old one.
- Give it a search interface. A person must be able to ask "what did the agent do to this resource last month?" and get an answer in seconds.
- Review weekly. Read the escalated items and the repeats. Adjust the rules from what you find.
What belongs on the list of safe actions?
Be explicit. A list is easier to defend than a general permission. The items below are examples for a stateless service. Set your own.
| Action | Agent alone | Needs a named person | |---|---|---| | Restart a stateless service | Yes, with a daily limit | | | Re-apply a declared setting that drifted | Yes, if the change was not manual | If the change was manual | | Add replicas within a set ceiling | Yes | Above the ceiling | | Widen a permission | | Yes | | Delete a resource or data | | Yes | | Change production configuration | | Yes |
Every row that says "named person" should record who approved and when.
Where KYN fits
KYN builds ops agents on cloud infrastructure in the client's own account. Agents have their own restricted accounts, and every action is logged with the actor. Deletions, outbound messages, filings and money movement wait for a named person. For a manufacturer, a ledger of 3,309 reconciled rows was loaded with zero duplicates, with each row traceable.
The first 14 days are a review: an inventory of your stack and interviews with the team. A full build ships in under 90 days. See operations and security.
Frequently asked questions
Is a version-controlled history enough?
It records what changed in code. It does not record changes made directly in a console, or why an agent acted. You need an agent-level trail as well.
Should an agent ever fix drift on its own?
For low-risk, reversible actions on a defined list, yes. Anything that widens access, deletes or alters production configuration should wait for a named person.
How long should audit records be kept?
Set it by your own obligations and risk. Keep them at least as long as the resources they describe, and check any sector rules that apply to you.
Checked 30 October 2026.
Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.
