Case 005 / findings / 3 October 2026
The full findings
Published intrinsic results largely reproduce: 92/96 Table 1 values and the reported overlaps and examples match. Our 1.32M-parameter test found no bottom-up advantage: BottomUpLL was 0.0486 BPB worse than UnigramLM (95% CI +0.017 to +0.080). This scale cannot settle the paper.
Original authors: Ahmetcan Yavuz, Clara Meister, Tiago Pimentel · Read the paper ↗
Paper versus our measured bytes per token for all 12 English Table 1 tokenizers; diagonal means exact agreement. This is one of eight Table 1 metrics, and the full 96-value comparison is downloadable.
What the data says.
This table contains the measured comparison behind the answer. The expert sections below explain how the numbers were produced and what they cannot establish.
| Tokeniser | Mean tiny-model held-out BPB | Search |
|---|---|---|
| BPE | 3.3260 | Bottom-up |
| BottomUpLL | 3.4491 | Bottom-up |
| TopDownComp | 3.3175 | Top-down |
| UnigramLM | 3.4005 | Top-down |
Claim-by-claim verdicts.
Confirmed means supported within the stated test; Partly confirmed keeps a tested piece separate from an untested part. Not confirmed means the tested evidence did not show the claim. Could not test marks missing evidence or an inapplicable setup.
| Claim | Verdict | Evidence boundary |
|---|---|---|
| Search dominates objective downstream | Not confirmed | Tiny GRU contrasts point the other way; paper-scale transformers unavailable. |
| Bottom-up lower downstream BPB | Not confirmed | BottomUpLL was 0.0486 BPB worse than UnigramLM in the tiny check. |
| Table 1 intrinsic measurements | Partly confirmed | 92/96 within stated tolerances; four 128k utilisation misses. |
| Published vocabulary overlaps and examples | Confirmed | Fixed vocabularies and example totals reproduce to reported precision. |
| Multilingual and BLiMP outcomes | Could not test | Missing five-language test union, checkpoints and BLiMP data. |
Lab notes and sources.
Expert detail: measurements and interpretation
The Table 1 check used the shipped tokenizer JSONs on five disjoint windows totaling 10,000 FineWeb-Edu documents. Corpus totals were normalized to the paper's reported byte size; this does not recreate the authors' exact held-out split. The four tolerance misses concern 128k vocabulary utilisation, which depends strongly on test-set size. All published Figure 3 overlap cells and the ten illustrative segmentation totals matched the released assets at their printed precision.
For a downstream diagnostic, each of four independently trained tokenizers fed the same two-layer, 1,315,842-parameter GRU design over five paired seeds. Each seed used the same raw text chunks and 160 updates. We measured held-out bits per byte on the same five slices. BottomUpLL minus UnigramLM was +0.0486 BPB, with a paired-seed 95% t interval of +0.017 to +0.080. That interval describes seed variation conditional on this tiny setup; it says nothing direct about the paper's much larger transformers.
The authors' core claim concerns 100M–1B-parameter language models. Those checkpoints, BLiMP data, and a five-language test union were not supplied in this local case. Our tiny result did not show the claimed bottom-up advantage, but small scale cannot settle the paper's conclusion. The two paper/code differences below have since passed second review. Their measured effects were minor.
How we tested
We ran published tokenizer JSONs on five disjoint FineWeb-Edu windows, checked intrinsic metrics against independent arithmetic, and compared all 96 English Table 1 cells with predeclared tolerances. Vocabulary overlap and example outputs were cross-checked separately. We then trained four matched tiny GRU language models over five paired seeds and measured byte-weighted held-out bits per byte.
Limitations and next check
The exact paper training and test splits, 100M–1B transformer checkpoints, multilingual test union and BLiMP data were unavailable. Four 128k utilisation values missed tolerance on our smaller test set. The tiny GRU is a diagnostic, not a paper-scale replication; its paired-seed interval does not transfer to large models.
Where the released code differs from the paper
Our agents flagged these during the code audit. A second, independent reviewer then confirmed both by reading the code and retraining both ways. Both are minor: neither changes the paper’s results. We have not yet heard from the authors. Data: self-pair retraining, pruning schedule (50k→8k), pruning schedule (10k→2k).
Minor, results unaffected: Self-pair counting. When choosing merges, the released code counts overlapping repeats such as “...” or “www” (standard HuggingFace behaviour). It also does not apply the paper’s special formula for merging a token with itself. The paper describes the non-overlapping rule as what its implementation does. Retraining both ways changed fewer than 0.3% of vocabulary entries at 8k–32k (up to 3% at a toy 2k size), and compression by under 0.01%.
Minor, results unaffected: TopDownComp pruning rate. The paper says each round removes 10% of the current vocabulary. The released code removes 10% of the tokens still above the target size, which is a finer schedule. In our small test this changed about 4–6% of vocabulary entries and compression by under 0.04%. Mainly it makes training slower (roughly 2.5–6× more rounds); it does not change results.
Original sources, data and corrections
The ideas and original claims belong to the paper authors. Kael Research is responsible for these implementations, checks and interpretations. Original source: Objective vs. Search: Decomposing What Makes a Good Tokeniser (2026-09-16)
By Ahmetcan Yavuz, Clara Meister, Tiago Pimentel. Download each public CSV below; the full experiment repositories, including any larger uncopied raw artifacts, are available on request. For corrections, see our public log or contact us through the site.