I measure the instruments that measure AI agents. A benchmark is an instrument; it has a noise floor, it drifts, and current reporting practice has no field that would show either. I build the small pieces that make those visible: a probe that runs every day, a format for recording what produced a number, and the analysis that reads a leaderboard against its own noise.
Poster, NeurIPS 2026 workshop "Who Verifies the Agents?"
We pinned every control a mobile-agent benchmark exposes and ran ten repetitions. Ten and a half hours in, the pinned endpoint's behaviour changed, and no field a leaderboard records would have shown it.
The sentinel sends the paper's probe, every day, to the paper's endpoint and to thirteen others.
live since 2026-09-04
The same fixed probe, every day, against hosted inference endpoints: determinism, output length, latency, served provider, and step changes.
Open source · v0.1.0
A small record of what produced a benchmark number. Run on the repeated runs eight public benchmarks publish, it gives each one's noise floor.
Log: one line per change to the site, the sentinel and the paper.