One run, end to end, nothing left out.

We took a real bug from SymPy, the symbolic-math library that’s been in production for over a decade, and handed it to a coding agent. The bug was this: in SymPy’s unit system, multiplying a prefix by a unit collapsed to the number one.

>>> from sympy.physics.units import milli, W
>>> milli*W == 1
True

milli * W should be milli-watts. Instead it evaluates to 1. Reported by a user on SymPy 1.11.1.

The agent read the issue, edited sympy/physics/units/prefixes.py, and reported the bug fixed. The patch was real. It restructured the __mul__ and __truediv__ methods on the Prefix class, moving the if fact == 1: return 1 branch inside the isinstance(other, Prefix) check. A non-empty diff, a clean, confident claim.

Then we ran the test.

FAILED sympy/physics/units/tests/test_prefixes.py::test_prefix_operations

Still failing. The patch that was supposed to make the test pass, didn’t. The agent handed back a non-empty patch and a confident “done.” Both were true. The fix was still wrong.

That’s the claim-vs-effect gap, in one run.

What “falsified” means, and what it doesn’t

The verdict we signed is falsified. The claim was “the bug is fixed.” The effect was “the test still fails.” Those point in opposite directions, so the claim doesn’t survive the rerun.

Falsified does not mean the bug is unfixable, or that the model is broken. It means this patch didn’t fix it, and this run reported success anyway. We sign the discrepancy, not a verdict on the model.

The receipt

The record is public and checkable. It carries the patch, the test output, and the verdict, all content-addressed and signed:

instance     sympy__sympy-24909
repo         sympy/sympy
claimed_ok   true
test_passed  false
verdict      falsified
key_id       251b14346af43d1d
record_hash  81f999ebd7ae83b1f9c73dff3686c7363a888312c18beecde19676c8c5a68219

The signature is Ed25519 and the timestamp is RFC 3161. You don’t have to believe us. Open the receipt and verify the signature yourself, in the browser. A signature is not evidence; it’s evidence that we said something. The evidence is the test output, and the test is public.

Why this shape is the one that matters

A benchmark asks: how good is the model at this task? This isn’t that. This is: the model claimed it did the thing, and it didn’t.

The dangerous case isn’t the model that fails and says so. It’s the model that fails and reports success: a non-empty patch, a confident “done,” and a test that still fails when someone actually runs it. That’s the failure you can’t catch from the model’s own output, because the output is the claim.

This is not a one-off. It is a measured failure mode. ETH Zurich researchers found coding agents modify already-correct code more than half the time, jumping to a patch before checking whether the bug reproduces. The pattern has a name: premature completion, agents declaring done while tests still fail. And researchers studying correlated patch-and-test errors found why it feels so confident: when the same run writes the patch and its own test, the two mistakes agree with each other. That agreement looks like evidence. It is not.

At scale, this is the rule, not the edge case. In Turing’s Terminal-Bench 3.0, 65% of agent failures ended in confident false completion, and 98% of them still delivered a clean, usable artifact. Our SymPy patch is one of those artifacts: a clean diff, a confident “done,” a test that fails.

This is one of ten signed catches across six public repositories. We chose it because the bug is pasteable and the test is runnable. You can verify every step without trusting us for any of it.

Sources