How We Test a Research Paper

The route

Six gates before a verdict.

Each gate catches a different kind of mistake. A fast result is worthless if the implementation returns the wrong answer; a correct result is misleading if the comparison is unfair.

01 / EXTRACT

Write the claim first

We list each testable claim, quote the relevant sentence, record its location in the paper, and decide what measurement could support or challenge it. This happens before experiments, so we cannot quietly change the target after seeing results.

02 / VERIFY

Correctness before speed

We compare a new implementation with an independent oracle: brute force, exact arithmetic, a textbook method, or another trustworthy reference. Random seeds and edge cases run before any benchmark. A method that fails correctness is not timed as a valid result.

03 / CALIBRATE

Check the ruler

The benchmark must reproduce a known result using the same harness. In case 001, classic linear probing was checked against Knuth's expected-probe formulas. A mismatch sends us back to the harness before we interpret the new method.

04 / COMPARE

Make the baseline fair

The baseline gets comparable engineering and tuning effort. We document choices that affect cost, including data structures, hash families, stopping rules and what counts as an operation. A theoretical comparison and a practical implementation comparison answer different questions.

05 / REPEAT

Keep the spread

The protocol calls for at least five seeds or repeats. We report a median with min–max, or a mean with a 95% confidence interval, rather than promoting one lucky run. Raw measurements go into CSV files, and a script should regenerate tables and figures from them.

06 / REVIEW

Publish the limits

Each claim receives a verdict: confirmed, partly confirmed, not confirmed, or could not test. We show the evidence and what our setup cannot establish. Suspected paper errors go to a human and a second AI before public wording is settled.

Reading the output

What a number can say.

A benchmark is a measurement of one implementation under stated conditions. It does not turn a finite experiment into a theorem.

Seeds or repeats

Multiple runs show whether a result depends on one random draw. We still name the capacity, workload and hash model, because a wider or different workload may behave differently. For case 001, the main CSV records five seeds at each load.

CPU threads maximum

Our shared Windows 11 lab machine has an Intel i5-1235U, 16 GB RAM and no GPU. Experiments are limited to three CPU threads. Other jobs can run at the same time, so wall-clock timings are preliminary until repeated under quiet conditions. Counts such as probes remain useful when the machine is busy.

We distinguish correctness from performance. Correctness checks whether every tested input gets the right answer. Performance asks how many operations or how much time the method uses after correctness is established. Probe counts in case 001 mean conceptual table-slot reads; they do not include every metadata access or cache effect. The practical CPU question therefore needs a separate timing study.

We also distinguish what the paper proves from what our implementation shows. A paper may prove asymptotic behavior under ideal random hashing, while our test covers a finite table and a concrete pseudo-random family. Agreement supports the claim in that region. It cannot prove the theorem for all table sizes or every input.

Audit trail

Make the work inspectable.

A reader should be able to trace a headline figure back to rows, assumptions, and a correction if a claim changes.

The working protocol names the artifacts produced during a case: CLAIMS.md records claim IDs and exact source locations; test code checks an independent oracle; results/*.csv stores every number; RESULTS.md gives verdicts and limits; and PROGRESS.md records what ran and when. A case page presents the plain-English answer first and links the raw data. We keep the static page readable without JavaScript so browsers and crawlers can inspect the same evidence.

When a suspected error appears, we write down the precise claim and a minimal reproducer. We do not declare the paper wrong from an agent's first impression. In case 001's correction, another reviewer found that an agent had imported the wrong lookup stopping rule. The bug claim was withdrawn before publication. That event is visible because the review process itself is part of the result.

Our verdicts are local to their evidence. “Could not test” can be the most honest answer when a method needs unavailable hardware, data or time. An inconclusive timing remains inconclusive even if a probe count looks impressive. We credit original authors prominently, link their work, and invite readers to inspect what we actually ran. The first full report shows the measured table, independent replication, caveats and downloadable data together.