AI evaluations
A benchmark is a measuring instrument.
Test it like one.
A benchmark, an AI judge and an interpretability probe all claim to measure something about a model. Psychology has spent a century learning how to check claims like that. I bring those checks to AI evaluation.
The approach
Five checks a test has to pass before its score means anything.
- Define it before you test it. Say what capability or tendency the test is meant to measure, and what would count as evidence of it, before a single item is written.
- Separate can from does. What a model is able to do under ideal prompting is a different quantity from what it does by default. Most scores blur the two.
- Say reliable across what. Across judges, across occasions or across tasks. Each is a different claim and needs its own evidence.
- Anchor it to something outside. A judge agreeing with itself is not evidence. A score has to be checked against a criterion the instrument did not write.
- Publish what failed. A null result or a withdrawn claim tells the next person where not to stand. It goes in the record with everything else.
What I can do for you now
Two ways to start.
Evaluation audit
You already use a benchmark or an AI judge and decisions rest on it.
A measurement review of what it defines, how consistent it is and across what, what outside criterion it is anchored to, and which of its claims the evidence supports.
Evaluation plan
You are about to test an AI agent on your own work.
Your question turned into a plan: test cases drawn from your real tasks, a named way of judging the answers, and a statement of what a pass would and would not mean.
Both are expert work done by me, with AI assistance. The automated benchmark designer further down this page is research and is not yet offered to clients.
Talk through your evaluation ↗What I am building
Five projects, one question: how would we know?
ActiveA team of AI agents that designs a custom benchmark for an AI system, checked the way a psychological test is checked.
Read more →ActiveTesting whether AI judges of code quality measure anything real, with preregistered checks that are able to fail.
Read more →FeaturedAn open playground of psychology-based patterns for controlling AI agent loops, and for measuring whether they help.
Read more →ActiveAn audit of interpretability measurements in a small language model: what survived correction, and what was withdrawn.
Read more →EarlyA science game where people and AI agents discover the hidden laws of a world nobody has seen before.
Read more → Where the evidence stands
What this work has shown, and what it has not.
Shown
- Study designs registered before the data existed, so the result could not shape the question.
- A full audit of my own interpretability work that withdrew the claims that did not survive it.
- A check on agreement between AI judges that came back low and retired one of my own earlier claims.
- Qualification gates that failed, published as failures in a public repository.
Not yet
- A benchmark run on a panel of models with items hard enough to tell them apart.
- An AI judge checked against more than one human coder.
- Any loop-control result on real agent traces; so far the data are simulated.
- Evidence that a team of agents designs a better benchmark than one strong model call.
Nothing on this page is described as validated, because nothing here has been independently validated yet.
What comes next
Closest to release first.
- A measurement-audit paper with a validation checklist for interpretability work. Drafted; awaiting my final read before submission.
- The first benchmark from the designer, once its items are hard enough to separate models.
- The third round of the AI-judge study, registered before it runs.
No dates are promised. Each of these appears in the Lab when it is public, with its limits attached.
The public record
What you can open today.
Working on evaluations yourself?
If you build benchmarks, study AI judges or evaluate agents, I would like to compare notes, review a protocol or have mine reviewed.
Write to me ↗