Case 001 · published 2 Oct 2026
A FOCS paper, tested two days after release.
The paper had no code and no experiments. One evening later, it had both.
The Power of Two-Choice Linear Probing
Amir Azarmehr, Michael A. Bender, William Kuszmaul, Rose Silver · FOCS 2026 · arXiv 2610.00688
Mostly, yes.
Checks to find an item · 1,048,576 slots · 5 runs
The swarm · 2 Oct 2026, IST
12 agents. 62 minutes.
Codex agents built and tested. Claude agents planned, reviewed and caught a mistake. Hover any bar.
Method
From PDF to verdict.
Five steps, every case. The checks are designed to catch our own mistakes first.
A new paper with no code and a testable claim.
Parallel agents implement every method, citing each section.
Reproduce a known answer before trusting any new one.
Many runs, fair baselines, a blind rebuild in another language.
A second AI audits. A human signs off. Authors get credit.
Instruments
Check the ruler before you measure.
Every case starts by reproducing a result that's already proven. If our ruler is off, nothing after it counts.
FIG A · CALIBRATION
Our baseline vs Knuth's 1962 formulas. On the line means the harness measures true.
FIG B · RUN-TO-RUN SPREAD AT 99%
Each dot is one run. Two choices make results nearly identical every time.
One table rises, the other stays low. That gap is the paper's idea, measured.
Two depths
One report. Read it your way.
The same facts for someone curious and for someone who reviews papers for a living.
Curious?
One glance tells you if it holds up.
Reviewer?
Flip to Expert view: log-log scaling, fitted exponents, seeds, ranges.
Rules
How we stay honest.
We publish our corrections next to our results.
Every case names them and links the paper.
Every number comes with its range.
One builds. Another audits.
Case 001 includes a claim we withdrew.
Published / cases 002–005
Four more tests. Read the evidence.
Each new case has a direct answer, a chart, source credit, claim verdicts and downloadable CSVs. Browse all five cases.
Yes: all 729 ordered coefficients match, and the scheme works over noncommutative rings. Direct Python is slower than NumPy; historical search claims remain untested.
Read findings and raw data ↗PUBLISHED CASE 003Are new shortest-path algorithms faster than Dijkstra?Not on four US road maps in our implementations: 1,600 certified quiet-server road runs, with median slowdowns about 6–75×. These timings do not test the papers' asymptotic bounds.
Read findings and raw data ↗PUBLISHED CASE 004Does SAMuon beat Muon? A small-scale testAt 0.056M–0.788M parameters SAMuon had lower median final loss at each size; post-hoc tokens-to-target were mixed, and the spectral stability mechanism did not reproduce.
Read findings and raw data ↗PUBLISHED CASE 005Does tokenizer search matter more than its objective?The published intrinsic measurements mostly reproduce (92/96 Table 1 values), but a tiny 1.32M-parameter test did not show the bottom-up BPB advantage. Paper-scale behavior remains open.
Read findings and raw data ↗Suggest a case
Got a paper you don't believe?
Public submissions open soon. Until then, we pick the queue.