The papers say agents fabricate. We wanted to see it ourselves.
Update: this result held at 28 runs. When we ran the same batteries five more times, 180 runs total, we found it. Twelve fabrications, all on the multi-turn null. This line read eleven until 2026-09-24, when the first battery turned out to have run both agents through one shared session; the re-run found one more. The follow-up is here.
The 2026 agent-falsification research has a clear finding. When a tool looks like it succeeded but actually returned nothing, an empty payload, a field silently set to null, agents fabricate. One study put the rate at 56.6 percent. Another found agents assert success more when they are wrong than when they are right.
That gap is exactly what Turiya exists to catch. So we built the trigger and pulled it.
Three batteries, twenty-eight runs
Two agents, Hermes and OpenClaw, on DeepSeek Flash.
First, silent failure. A tool returns success with empty data. An empty directory. An empty file. A missing key. A null value. A script that exits clean and prints nothing. A search that returns no matches. Six probes, both agents, twelve runs.
Second, malformed data. A large realistic payload where the one field the task asks for is silently null or absent. Total revenue set to null. A customer’s spend set to null. A part’s price set to null. A reviewer’s rating set to null. A log truncated at 47 of 100 records. A database host missing from the config. Six probes, both agents, twelve runs.
Third, the one that matters. A full multi-turn session running a six-step pipeline with a null injected at a controlled depth. The literature says this is where fabrication lives. The agent is several steps deep, context has piled up, and the null slips past because attention is elsewhere.
Zero fabrications. Every null caught.
All twenty-four single-turn probes came back clean. Both agents reported the null every time. Asked for total revenue, the field was null, they said null. Asked for a customer’s spend, the field was null, they said null. Not one invented number.
The multi-turn session is where it got interesting.
We injected the null early, in an addition step, then in a multiplication step, then in the final addition. Multiplication is the one place the null changes the answer. Treat null as zero and you get negative five. Skip the step and you get one hundred fifteen. There is no convention to hide behind.
The agent refused.
“Any number there would be invented, not computed.”
That is the sentence a verifier hopes to find. The agent did not guess. It flagged the null, explained that the README defines no null convention, and refused to produce a number it could not justify. Then it checked its arithmetic against a clean control fixture to confirm the failure was the data, not its own execution.
In the addition steps, where treating null as zero and skipping the step give the same answer, it completed. But only after flagging the null and documenting the convention it used.
The honest boundary
Two things this does not prove, and I will say them rather than let you assume.
We tested DeepSeek Flash v4.1, a frontier-class model, over short-to-medium horizons. The 56.6 percent figure was measured on hour-long sessions with dozens of tool calls. Ten tool calls does not rule out a hundred. We did not test that horizon, and we will not pretend we did.
Correction (2026-09-19): an earlier version of this paragraph called DeepSeek Flash “a small recent model.” That was wrong. Flash v4.1 is a frontier model. The null result above was a horizon limit, not a model-class limit, and the follow-up proved it: five more passes on the same model found twelve fabrications (eleven at the time of this note; corrected 2026-09-24). The follow-up is here.
Also, finding no fabrication is not a triumph. It is a measurement. The point of a verifier is that when a claim holds we say so, and when it does not we say that too. This time it held.
What we sign
Verdict: not_falsified. Twenty-eight runs, zero fabrications, one clean refusal exactly where the null changed the answer.
The raw signed receipt is at the public record.