Knowledge Retraction: Why a Self-Improving SOC Needs an Undo
Self-improving SOCs learn from analysts but rarely unlearn. What retracting a wrong lesson actually requires: scope, consistency, recomputation and lineage.
Self-improvement is now standard in AI SOC positioning. The analyst corrects the system, the system improves, the work shrinks. The mechanism is real and the benefit is real.
This piece is about the half that is usually missing: what happens to knowledge after it gets in. In AVA we call that knowledge governance, and it has four operations: Learning, Revision, Retraction and Expiration.
Illustrative record from the case below.
What self-improvement actually buys
Strip the marketing and a learning loop does one thing: it lets the system stop re-deriving a conclusion it has already reached.
Investigation is the expensive part of triage. A system with no memory pays the full cost of every alert, every time, including the fortieth firing of a rule that has been benign on every previous occasion. A system with memory pays it once. It does not get cleverer. The population of alerts requiring original reasoning shrinks, so the same capacity covers more ground. In a SOC where most alerts are false positives, that is the whole game.
We are not going to argue against self-improvement. We build it.
What it lacks
A learning loop has an entry path and no exit path. Most implementations, including our own early ones, can answer “how does knowledge get in”. Very few can answer “how does it get out”. The usual answer is that an override gets logged, an override rate gets reviewed, and eventually somebody revises a prompt. That is not a loop. It is a backlog, and it closes at the rate somebody has time.
The gap matters because the two failure directions present differently.
A wrong malicious conclusion announces itself. Analysts get escalations they consider noise. They complain. It gets fixed, because the cost lands on the people who can raise it.
A wrong benign conclusion is silent. Alerts stop arriving. Dwell time on that pattern goes to zero. Every metric a SOC reports improves. The failure is indistinguishable from the product working until an incident review works backwards into it.
So the direction that needs an undo most is the direction least likely to prompt one.
A concrete case
A patch management tool runs overnight and invokes PowerShell with -ExecutionPolicy Bypass -WindowStyle Hidden against servers in sequence. The EDR alerts on it. An analyst confirms the parent process and the change window, and closes it as a false positive. This happens three times across three weeks, and the system starts handling the pattern automatically.
Six weeks later the patch management server is compromised. Management tooling is a well-understood intrusion path precisely because it is trusted by design, runs privileged, and reaches the whole estate. The attacker does not need to evade detection. They need to resemble something detection already forgives.
The command differs. The shape does not. The alert is closed automatically, with a complete and accurate record citing three prior analyst confirmations.
The analyst was asked a narrow question: is this alert malicious. They answered it correctly, three times. The system recorded something broader, because a conclusion that does not generalise has no recall value. Nobody chose the boundary of that generalisation. It came from whichever indicators happened to be present.
And the failure is not silent in the usual sense. It is documented. It cites human confirmation. It does not read like an error, it reads like procedure. Auditability records what happened. It does not test whether the belief behind it still holds.
Four operations, not one
A loop that only adds knowledge has one response to this case, and it is the one that caused it. AVA Knowledge Governance has four, which differ in what triggers them, what they change and what they keep.
| Operation | Triggered by | Current belief | Kept on record | In the patch case |
|---|---|---|---|---|
| Learning | a settled analyst decision | gains a lesson | the lesson and its evidence | three closes teach that the patch agent’s PowerShell is benign |
| Revision | new evidence that changes a conclusion | the lesson is replaced by a new version | both versions, and when one replaced the other | after the override, the lesson carries the analyst’s conclusion |
| Retraction | an analyst overriding the system’s verdict | loses the lesson and everything it trained | the lesson, when it stopped applying, and why | the benign lesson stops closing the pattern |
| Expiration | time, with nobody disagreeing | stops returning the lesson | the lesson, still readable | had the tool been retired, the lesson would have aged out |
Revision and retraction are easy to confuse, because in this case one override triggers both. They act on different things. Retraction removes what the system’s own verdict taught, from every place it was written. Revision records what the analyst concluded instead. Expiration is the odd one out: nobody triggers it, which is why no feedback loop ever produces it.
What retraction requires
Retracting what the patch lesson taught is not a delete. It has a scope to get right, components to keep consistent, derived knowledge to recompute and a history to keep.
1. Scope follows the scaffolding
A lesson is one sentence. Its knowledge scaffolding is everything that holds it up: the evidence it was drawn from, the conditions it holds under, the derived representations that let it be used quickly, and the dependencies that can rebuild it.
Recall speed comes from redundancy, so a learned conclusion lives in more than one place. In AVA, one triage decision is written to two knowledge stores: once as a recalled conclusion, and once as derived knowledge that scores the pattern. Either can send the next matching alert down the fast path on its own. When the recalled store clears an alert, that decision also feeds the derived one.
A retraction’s scope follows from that scaffolding.
- Every representation. A retraction that reaches the recalled store but not the derived one leaves the pattern scoring the same way. One that reaches the derived store but not the recalled one is undone when the recalled store re-teaches it. Both, or neither.
- The scope it was written in. A retraction has to find knowledge the way triage wrote it: in AVA, the organisation’s own partition of each knowledge store. Any other scope corrects some other tenant’s knowledge, or finds nothing.
- One alert at a time. One alert has one AI verdict and one analyst close, so a retraction undoes what that alert taught. The lesson for the wider pattern is then revised to carry the analyst’s conclusion.
The decisions the lesson already made are out of a retraction’s reach. That is a lineage problem, covered below.
2. Consistency is a choice of failure state
The two knowledge stores are separate, and neither is transactional, so a retraction across both cannot be atomic. The design decision is which half-state a failure leaves behind.
AVA corrects the derived store first and aborts if that fails, leaving the system visibly unchanged. If the recalled step then fails, the lesson is still live and still cited, so the system’s behaviour is explained by something an analyst can see. That state is not safe to leave, since the recalled store can re-teach the derived one, so it raises an alarm, not a log line. The retraction is idempotent and reports which halves completed, so retrying is the intended response.
The reverse order fails into the worst reachable state: the recalled conclusion gone, the derived store still closing the pattern, and nothing left to explain why. Reaching it requires no mistake beyond stopping halfway. A correction that can leave the system undefined is not a correction.
3. Recomputing derived knowledge
Withdrawing a recalled conclusion is a removal. Correcting derived knowledge is not: it has to be recomputed from what remains. AVA’s derived knowledge can be corrected exactly, by removing the alert’s evidence and recomputing.
4. History: retraction is not deletion
Deleting the record is the obvious implementation and the wrong one. After an incident, the useful question is what the system believed on the day, on what basis, and when that changed. Delete the conclusion and that question has no answer.
So retraction is a state change. The recalled conclusion is tombstoned: it stops being returned to triage and stays readable, carrying the time it ceased to apply and the reason. Operational lookups see current belief. Investigative lookups can see all of it, through two kinds of lineage.
Data lineage tracks where knowledge came from: the evidence it rests on and each transformation since. Which three alerts, which analyst reasons, which version replaced which.
Reasoning lineage records how the system reached a conclusion and what that conclusion went on to influence: which investigations cited it, which alerts closed because of it.
Data lineage tells you whether a retraction is justified. Reasoning lineage tells you how far it has to reach. Stopping the lesson today does nothing about the six weeks of alerts it already closed, and those are found only by following it forward. That is the difference between an incident review and an informed guess.
Expiration: when nobody is wrong
The tool is decommissioned. The change window moves. An exception for a backup job outlives the backup job by two years. Nobody was wrong: the conclusion was correct when reached and then stopped being true. No feedback loop detects this, because detection requires somebody to disagree and nobody will.
This needs a lifetime rather than a correction, and the lifetimes are not symmetric. A benign conclusion should age out, because the cost of re-investigating something still benign is one investigation. A malicious conclusion should not. Something dangerous does not become safe by being ignored for ninety days. A benign lesson that analysts keep confirming is renewed each time. One that nobody confirms stops being returned, and stays on record.
Implementation note. Apply the age limit when knowledge is read, not by writing an expiry onto the record. In most stores that field means superseded rather than scheduled, so writing it removes the record immediately rather than in September. We got this wrong once.
Closing the loop
With retraction in place, the analyst’s close is itself the trigger. Each close is compared with the system’s verdict. Agreement reinforces the lesson. A disagreement retracts what the original agreement taught and revises the lesson to carry the analyst’s conclusion. An inconclusive verdict is excluded, because it claimed nothing. No ticket, no prompt edit, no engineer.
Keeping those parts apart is what makes the whole thing buildable. Governance is policy and should differ between environments; the mechanism has to be correct whatever the policy says. In AVA, a language model writes the wording of a lesson, but whose conclusion it is, how far it is trusted and how long it is believed are decided in code, because those are the parts that must be predictable.
Once retraction is cheap, bounded and reversible, the system can afford to learn more aggressively, because a wrong lesson is no longer permanent. Systems without an undo have to be conservative about what they commit to, which caps how much work the loop can remove. The safety mechanism and the throughput turn out to be the same mechanism.
What we do not claim
Not that this removes the need for analysts. It changes what they are asked. The loop depends on a human disagreeing, which means the value of an override goes up, not down.
Not that every wrong lesson should be corrected automatically. An authorised red team exercise produces real technique with no actual threat. Most verdict taxonomies cannot express that, so the honest behaviour is to hold the lesson for a person rather than confidently learn a wrong one in either direction. Knowing which corrections to automate is a separate problem from being able to automate them.
Not that we run this unattended today. Automatic retraction ships switched off. The comparison runs and disagreements are recorded, but nothing is retracted until it is deliberately enabled. We want to see how often analysts actually disagree in a given environment before an automatic action rewrites anything. A loop you have not observed is not one to enable.
The question worth asking
Of anything that claims to learn from your analysts, the useful question is not whether it learns. Ask what happens when it learns something wrong.
- What retracts it, a person or a mechanism, and how long does that take? — Retraction
- Does the retraction reach everything the lesson influenced, or only the visible part? — Scaffolding, reasoning lineage
- Can you still reconstruct what the system believed last March? — Data lineage
- When new evidence changes a conclusion, is the old version kept or overwritten? — Revision
- What happens to a lesson that was right once and quietly stopped being true? — Expiration
If the answer involves an engineer, a prompt, and a free afternoon, the system is not self-improving. It accumulates in both directions and only notices in one.
Written by Ashfaaq Farzaan at Priam Cyber AI.