ActiveThird round in preparation · no validation study run · checked 13 Nov 2026

EQA

How far can you trust an AI judge?

Testing whether AI judges of code quality measure anything real, with preregistered checks that are able to fail.

Why it matters
AI systems are now marked by other AI systems. If the judge is biased or inconsistent, every result built on it inherits the problem.
What it does, and how
A language model reads the code and records narrow observations, each with the evidence it relied on. Ordinary code, not the model, applies the scoring rules. The readings are then checked against something the judge did not write, such as whether a real security fix scores as safer than the code it replaced.
Where it is up to
Earlier rounds produced mixed results on an older version of the instrument, including low agreement between judges that retired one of my own earlier claims. A third round is in preparation and is not yet registered. No independent expert review has happened, and the validation study is still ahead.

Where you come in

If you rely on an AI judge and have never checked it against an outside criterion, that check is what the evaluation audit does. Researchers who would review the protocol are especially welcome.

Write to me about EQA