ActiveMachinery built · first two design passes ended as drafts · checked 13 Nov 2026

PsychForge v2

Can a question about an AI agent be turned into a benchmark that actually measures it?

A team of AI agents that designs a custom benchmark for an AI system, checked the way a psychological test is checked.

Why it matters
Public benchmarks test someone else's task. Teams need to know whether an agent does their work safely, and most have no way to build a test that would tell them.
What it does, and how
A team of specialised AI agents, each with one job, from scoping the question to writing test items. Two reviewer agents act as gates that can send work back. Ordinary code computes every score, every step is logged, and nothing is released without my sign-off.
Where it is up to
The machinery is built and tested. Its first two live design passes both ended as blocked drafts, not run-ready benchmarks: reviewers sent most items back, and an early screen found the surviving items too easy to tell models apart. It is not ready for a paying client, and whether the agent team beats a single strong model call has not been tested.

Where you come in

If you build or buy AI agents and have a question no public benchmark answers, or you make benchmarks and want a measurement review of one, the evaluation audit is the place to start.

Write to me about PsychForge v2