The standard way to evaluate an AI system is to have another AI read its output and score it. Often three, for safety. GPT and Gemini and Claude, a rubric, a recorded fingerprint per judge, and a consensus number at the end.

That is not nothing. It beats one model grading its own homework, and it is far better than a vendor grading itself. But it answers a different question than the one most buyers think they are asking.

What a judge measures

A judge measures agreement between a rubric and a text. That is a real property, and for some purposes it is the right one. If you want to know whether an output reads as non-compliant, or whether a document is missing a required clause, a rubric plus several models is a reasonable instrument. It is cheap, it runs in parallel, and it works on outputs where no independent ground truth exists to check against.

What it cannot measure

It cannot tell you whether the thing described in the text actually happened.

An agent says: refund issued. Three judges read that sentence, and the transcript around it, and the tool call, and agree that it reads as a completed refund. The consensus is real. It is also about the text.

Whether the billing system’s subscription state changed is a different question, and it is not one any of the judges was asked.

Consensus is not ground truth. It is evidence that the judges agreed.

We say something close about our own work, and it applies here too:

A signature is not evidence. It is evidence that we said something.

Same shape. An evaluation is not a finding. It is a finding that the evaluator concluded.

The recursion

Here is the part that took us a while to articulate.

Assurance exists because a system should not be the final authority on its own output. That premise is now uncontroversial. But it usually gets stated one layer too shallowly:

the system
  → the system's account of itself
  → the evaluator
  → the evaluator's judges
  → the evaluator's score

Moving the judgement one layer out, from the system to the evaluator, does not end the recursion. It relocates it. If the evaluator’s judgement is itself model-mediated, then that layer is now part of the trust problem, and nothing has checked it.

We are not alleging that anyone does this improperly. Rubric-based multi-judge scoring is a defensible method for the question it answers. The difficulty is that the question is narrower than the word “assurance” suggests, and compressing it into one number hides the seam.

Two kinds of evidence, layered

They are not competing. They answer different questions.

A score answers how a system looks against a framework, across dimensions. It is useful for comparing systems and for a governance dashboard. It summarises.

A receipt answers what happened for one specific claim when someone checked it. It does not summarise. It records one event and stops.

The second is narrower: one event recorded, rather than a summary. A verdict a stranger can re-run stops being a judgement and becomes a result. We are not there yet. A stranger can check our record without asking us; re-running the finding needs the harness, and that is a gap we list on the independence page.

A score cannot substitute for it. A system can sit at 91 on an accuracy dimension and still have failed the specific thing it claimed to do this morning. Those are not in tension. They are different claims.

The limit on our side

We do not get to make this argument and then exempt ourselves from it.

A Turiya receipt is signed, content-addressed, and verifiable in a browser without our server. That proves the record was not altered. It does not prove our verdict was right. The substantive finding still rests on our method and our judgement, and the honest status is independently checkable evidence, not proof of reality.

That is why we publish our rejects, and why we want the harness re-runnable by someone who is not us. We do not ship it that way today: how it reaches a customer is settled with them rather than decided in advance. Until then, what a stranger can check is our record, not our finding. It is also why we deliberately do not produce a score of our own. We would rather return indeterminate on a claim we cannot falsify than manufacture a number so a dashboard looks complete.

The recursion applies to us too. We are just one layer further down, asking about the effect rather than the account of the effect.