Here’s an uncomfortable fact about every AI system ever shipped: when it tells you it worked, it hasn’t handed you a result. It’s handed you a claim.

Most of the industry treats those two things as the same, and that single conflation is where most of the damage lives. A model says “the bug is fixed.” A vendor says “our agent runs this workflow end to end.” Nobody checks. We move on.

The claim is what the system says happened. The effect is what actually happened. The distance between them is the gap we built Turiya to find.

The SymPy test

One concrete run.

We pointed a coding agent at a real bug in SymPy, a symbolic-math library with a decade of real users and a test suite. The agent read the issue, wrote a patch, and reported the bug fixed. A clean, confident claim.

Then we ran the test.

It still failed.

The test the agent’s own patch was supposed to make pass, failed. The system had produced a non-empty patch and a confident “done.” Both were true. The fix was still wrong.

That’s the gap. The claim (“fixed”) and the effect (“still broken”) are both real, both recorded, and they point in opposite directions. Everything we sign depends on that difference.

We’ve now caught the same shape ten times across six public repositories: SymPy, httpx, click, Jinja, Rich, and FastAPI. Each one is a non-empty patch, a claim of success, and a test that still fails when someone actually runs it. Code is just the cleanest slice. We sign the same verdicts across seventeen system types, from fraud and credit to fairness, forecast, computer vision, speech, and scientific AI. You can verify any of them in the public record.

What a benchmark measures

People read “we falsify AI” as “we hate AI.” We don’t. We use these systems; they’re genuinely useful.

But a benchmark and a verification answer two different questions.

A benchmark asks: how well does this model do a task we gave it? That’s a useful question, and the field has gotten good at answering it.

Verification asks a different one: did the system actually do the thing it’s now claiming to have done?

A benchmark is a school exam. It tells you how good the student is at taking tests. Verification is an audit. It tells you whether the books actually balance.

This stops being academic the moment the systems start doing things. Gartner projects that 40% of enterprise apps will embed task-specific AI agents by the end of 2026, up from under 5% in 2025. A wrong sentence is cheap. A wrong wire is not. And the benchmark layer is already straining. OpenAI’s own 2026 audit found 59% of audited SWE-bench Verified problems had material flaws, then abandoned the benchmark. A benchmark tells you a score moved. It does not tell you the claim was true.

A benchmark is a measure, not a verdict. Goodhart’s law is what happens when you forget that: when a measure becomes a target, it stops measuring. Watch it happen. Rewrite the GSM8K math problems from scratch and leading models drop up to 13 points. They didn’t learn math. They memorized the test.

The gap matters most where the system’s output is the claim. A coding agent doesn’t give you a working patch. It gives you text that says the patch works. A fraud model doesn’t give you caught fraud. It gives you a list of transactions it flagged. The self-report is the product. And a self-report is the one piece of evidence you can’t trust on its own.

The failure has a name now. Across 3,600 trials, Turing’s Terminal-Bench 3.0 found 65% of agent failures end in “confident false completion”: the agent runs its own checks, reads them as passing, and signs off. 98% of those failures still delivered a clean, usable artifact. The failure doesn’t look like a crash. It looks like a success. The confidence even runs backwards: agents claimed success more often when wrong (84%) than when right (81%).

You can’t prove “true.” You can falsify.

We don’t try to prove a system is correct. You can’t, and anyone who claims otherwise is selling something. What you can do is try hard to break the claim, and record what happens.

The rule is old and settled. Karl Popper drew the line: a scientific claim is one you can falsify, not one you can prove. You can never prove a theory true. You can only fail to refute it. We apply the same test to what an AI says it did.

So our verdicts are deliberately small. A receipt says one of four things:

  • falsified: we re-ran the claimed effect, and it didn’t hold.
  • not_falsified: we tried to break it and couldn’t, within a defined set of attacks.
  • certified: a checkable property held, and we can point at the proof.
  • indeterminate: there was nothing checkable to test.

Notice what’s missing: “true.” We never sign “true.” That isn’t modesty. It’s the difference between evidence and marketing, and we’d rather be on the evidence side.

A signature is not evidence

This is the part people get wrong most often.

A signed receipt tells you the record hasn’t been altered and this key signed it. The hash is there so you can check both yourself. It does not tell you the finding was correct.

Cryptographic integrity is not correctness. A signature proves we said something; it doesn’t prove the something is right.

We’re loud about this because the industry keeps using “signed” and “verified” as if they were the same word. A wrong finding with a valid signature is still a wrong finding. The signature only tells you we signed it. Our job is to make sure the thing we sign is true, and to hand you the evidence so you don’t have to take our word for that either.

We publish our misses

A verifier that never corrects itself is not a verifier. So the misses are public.

We reject far more candidates than we publish. Some fail the gate before the model even runs. If a test fails for a harness reason, that’s our bug, not a catch, and we throw it away rather than dress it up. And twice we published a receipt and then pulled it, because our own coverage gate had rejected a fix that actually passed its tests. We said so, publicly, instead of quietly deleting the record. The candidates we rejected and the ones we retracted are on the record, not hidden.

That’s the part a vendor inflating a scorecard can’t show you. The negative is part of the record, because the negative is what makes the positive believable.

The line we won’t cross

One more boundary, because honesty about limits is the whole product.

We verify falsifiable claims about checkable effects. Did the code pass the test? Did the fraud flag turn out to be fraud? Those have an answer you can check.

We don’t tell you whether your poem is good, or whether your model is “aligned.” Those don’t leave a checkable effect, so there’s nothing to falsify. We’d rather sign “indeterminate” than invent a score.

What to do with this

The next time a model tells you it succeeded, run one test of your own: is that a claim, or is it the effect? If you can’t tell the difference from where you’re sitting, that’s not a failure on your part. It’s what the gap looks like from the inside.

We built the tool that closes it. One command. One real claim. One signed receipt. No credit card, no sales call.

$ turiya verify --pilot

Sources