We Deleted Every Vulnerability Detector We Had Written

We removed the last signal detector from RiftX earlier this year. Not disabled behind a flag, not left in place returning None. Deleted, along with the detector_classes field on the profile object that used to point at them. There is now a unit test whose entire job is to assert that the field is gone.

That code was the part of the system I was proudest of. Removing it made the verdicts better, and this post is about why, because the reasoning generalizes past our product.

What the detectors did

A detector was a small, careful piece of code that answered one question about one vulnerability class. Did a dialog fire. Did the response body contain a SQL error string. Did the Location header point somewhere off-origin. Each returned a signal with a score, the scores were combined, and the combination gated what verdict the system was allowed to ship.

This is the standard architecture. It is also, on paper, the responsible one. Deterministic, testable, reviewable. You can write a unit test for a detector. You cannot easily write a unit test for a judgment.

We had detectors per class, an observation layer that fed them, and an input-transformation layer above both. Three clean tiers, each independently testable. I could draw the diagram from memory.

The problem was not that detectors were wrong

They were mostly right. That was the trap.

A detector answers a narrower question than the one the product exists to answer. It answers "does this evidence match the pattern I wrote down," and the pattern was written by a person at a particular moment, with a particular set of examples in mind. It cannot answer "did the thing actually happen," because it has no access to that question. It only has the pattern.

For a scanner that is fine. A scanner is supposed to surface candidates, and a pattern that fires often is doing its job. But every RiftX job is a retest. Someone reported a finding, a team claims they fixed it, and the product has to decide whether the fix holds. That is a different question, and it breaks detectors in a specific way.

Consider a reflected XSS that was patched. The fix strips <script> tags. Our detector fires on a dialog appearing, so it goes quiet, and the run reports that the fix holds. Now consider that the payload lands inside an attribute rather than an element, so the tag stripper never sees a tag, and the injection still executes through an event handler. The detector goes quiet in both cases. It is silent when the fix worked and silent when the fix was irrelevant, and the score it emits looks identical.

You can fix that. You write another detector. Then the next fix is a Content Security Policy with an unsafe fallback, and you write another one. Each is correct, each ships, and the system as a whole keeps having exactly one blind spot: whatever nobody has thought of yet. The detectors were not a floor under the verdict. They were a ceiling on it.

That is a bad property in general. In a retest it is close to disqualifying, because the whole product is the attempt to beat a fix that someone else designed specifically to stop the last thing that worked.

What replaced them

Two halves, and the important part is that they are strictly separated.

The first half executes. An agent drives a real browser, follows the reported steps, tries things, and records what happened. It captures the HAR, the DOM, dialogs, console output, screenshots, and video. It is deliberately incurious about meaning. Its output is evidence, not a conclusion.

The second half judges. A separate read-only loop reads that evidence and derives the verdict. It has its own budget, its own prompt, and its own termination conditions. It runs a small set of allowlisted read-only commands over the materialized bundle, one per step, reading the output on the next turn until it has enough to decide. Then a single synthesis call emits the verdict, a confidence score, and a short list of evidence references.

That loop cannot reach the target. It has no browser and no network. If it wants to know what the application did, its only route is the record of what the application did.

The judge is not allowed to see what the agent concluded

This is the part I would push back on if someone else described it to me, so it is worth being precise.

The evidence bundle the judge receives is built to omit things. Not by accident, and not only by prompt instruction. The materializer leaves out the agent's reasoning, its notes, and its standing verdict. On top of that, the judge runs under a capability profile that narrows the command allowlist to raw-evidence views only, and explicitly denies the lenses that would surface the agent's conclusion or reasoning trail.

Two layers doing the same job, because a judge that can read the agent's answer is grading its own homework, and the failure would be silent. The verdict would still look independently derived. It would just agree with the agent every time, which is exactly what a broken independence check looks like from the outside.

The floor applies to good news too

Every judge reports a confidence score, and a verdict ships only if it clears 0.90. The number is declared in one file and imported everywhere, so there is no second copy to drift.

The part that took an argument internally: the floor is symmetric. It used to be 0.60, and it used to apply only to "this still reproduces." A verdict of "the fix holds" shipped on the judge's word at any confidence at all.

That was backwards. In a retest, the expensive error is not calling a fixed thing broken. That wastes an hour and someone catches it, because the client pushes back and a human re-checks. The expensive error is calling a broken thing fixed, because nobody pushes back on good news. The finding gets closed, the closure letter goes out, and the vulnerability stays in production with a document saying it does not.

So the direction that silently leaves a live vulnerability in production is now held to the same bar as the other one. Below the floor, in either direction, the job returns Needs Review.

Absence of data is never a number

One rule in that file matters more than the floor, and it is the kind of thing you only write down after it has hurt you.

None means a judge did not run, or ran without producing a score. 0.0 means a judge ran and was certain of nothing. Those are different facts, and every bug we have had in this area came from collapsing them.

A job that reported no confidence was once persisted as 1.0. Maximum confidence, synthesized out of nothing, clearing every downstream check. A degraded audit once emitted 0.0 as a "no data" sentinel, which then read as a real and maximally pessimistic judgment. Unscored jobs were averaged into dashboard statistics as zeroes, quietly dragging down a number people were making decisions with.

Coercion now returns None for absent or malformed input rather than falling back to a number, and callers have to skip it rather than default it. When several judges contribute to a verdict, the headline confidence is the minimum across them, not the average, so one uncertain judge cannot be smoothed over by two confident ones.

Citations get checked against the evidence

The judge cites its evidence: which step, which request. Those citations are presentation only. They never change the verdict.

They can still be wrong in a way that misleads whoever is reviewing the job, and the canonical version of that bug is specific. The judge anchors the effect on one step, then cites, as the request that carried it, a HAR row that a different step issued. Often the pre-injection baseline request, which is the least interesting row in the file. The verdict is fine. The thing a reviewer clicks on to check the verdict is not.

So citations are validated against the step the judge anchored the effect on. On a cross-step mismatch, the inconsistency goes back into the judge as a correction, and it can re-synthesize. It is allowed to keep its verdict or revise it. The correction never tells it which. After a bounded number of attempts, a still-mismatched citation is repaired structurally or dropped, and the interface falls back to the full trace. A citation problem never demotes a verdict that already cleared the floor.

An auditor that can only take things away

After a job is delivered, a separate always-on service picks it up and re-derives the verdict independently. It can downgrade to Needs Review. It cannot upgrade, and it cannot flip a verdict from one direction to the other.

The asymmetry is the point. An auditor that could promote verdicts would be a second chance to ship a confident answer, and the system already has enough of those. An auditor that can only withdraw them is a net.

What this costs

It is more expensive. Several bounded LLM loops run per job where previously there was one call, and the judging half is real spend on work that produces no new evidence.

It is slower, for the same reason.

It abstains more. A detector-gated system almost always produces a verdict, because a threshold always resolves. A judge with a 0.90 floor in both directions returns Needs Review more often, and every one of those costs a human the time we were trying to save.

I think that trade is correct, and not because abstaining is virtuous. It is correct because the alternative is a confident wrong answer inside a document a consultant signs their name to. But it is a real cost, it lands on the customer, and the honest version of our roadmap is that reducing the abstention rate is the main thing we are working on.

There is one limit this architecture does not solve, and it is the one worth being loudest about. When the judge concludes that a fix holds, it is reporting that the fix withstood the techniques we tried. That is not the same claim as "this fix is sound." The gap between them is our bypass catalogue, and a catalogue is a list of things somebody thought of, which is the same class of problem I just spent this post describing.

We cannot close that gap by being more confident. So the verdict ships with the list of what was actually attempted, and you get to judge whether the coverage was good enough for the finding in front of you. Conceding that limit is more useful to you than any accuracy number we could put next to it.

What it does and does not cover

The boundary is the target, not a list of vulnerability classes. If the finding lives on a web or API surface and the report says what to do, it gets retested and reaches a real verdict — including classes we have never specifically tuned for, because the judge reasons from raw evidence rather than from a detector somebody wrote for that class.

Three things come back as Needs Review instead, and none of them is a vulnerability class: a target that is not web or API at all, a report that asks us to go looking for issues rather than naming one, and notes that do not carry enough to reproduce anything.

If you want the operational version of this problem rather than the architectural one, why retesting eats consultant hours is the other side of it. If you are trying to work out which category of tool this even belongs to, where the 2026 tools actually fit lays out the layers.

And if you want a public testbed for the classes where this works, PortSwigger's Web Security Academy is still the best reference available, because the lab structure makes the exploit path explicit enough to check a tool against.