The signed falsification record
10 verified claim-vs-effect gaps, caught on DeepSeek frontier models across 6 public repositories. Each is a non-empty patch (the system asserted a fix) whose test command still fails. Each is Ed25519-signed and content-addressed.
Each card links to its receipt; the effect is derived from the recorded test output, not asserted.
Methodology
Each catch follows the SWE-bench methodology: clone the real repository at the buggy commit,
apply the test_patch, run a real agent (LangGraph ReAct over a frontier model) to
produce a fix, then independently verify the FAIL_TO_PASS test. A claim is
a non-empty patch (the system asserted it fixed the bug); the effect is whether the test
command actually passes. A catch is a non-empty patch whose test command still fails: the
claim-vs-effect gap.
Every candidate also passes a baseline gate before the system runs: the test file must fail at the buggy commit with no import errors, so a catch can never be a harness artifact (a missing dependency, an unrunnable test, or a coverage-only failure mislabelled as a bug). Candidates that fail the gate are skipped rather than published.
Every receipt is signed (Ed25519) and content-addressed (SHA-256). Anyone can re-verify a receipt against the published public key. No trust required to verify the record.
The record grows as new models and systems are tested. Contact holynttechnologieshq@outlook.com to contribute a repo.
Verify your own systems against real ground truth.
View the signed record Read the methodology Start Free Pilot