Spectral Allocation: Why Muon Outperforms Adam, and How to Improve MuonXiaodong Wu, Wenyi Yu, Chao Zhang, Philip WoodlandarXiv preprint · arXiv 2608.25990 · tested 2–3 October 2026
Read paper ↗

Does SAMuon beat Muon?

A little, at small scale.

Tested ~1000× smaller than the paper

Median final loss was lower at 3 / 3 tiny sizes; tokens to a post-hoc target were 0.867 / 1.015 / 0.839 times Muon.

2 confirmed1 fine print2 not confirmed1 couldn't test

The idea, in 6 seconds

Three ways down the same loss valley. Lower is better.

AdamWMuonSAMuon
MEASURED RESULT · 0.788M PARAMETERS · FIVE SEEDSAdamW 4.837 · Muon 3.892 · SAMuon 3.851FINAL VALIDATION LOSS (BPB) · 25,600 TRAINING TOKENS

The verdicts

Tap any card

Watch the gap open up

Held-out validation loss · lower is better

Model size
AdamWMuonSAMuon

Median of five seeds; shaded bands show seed minimum to maximum. All runs had the same 25,600-token budget. Step 0 is above the detail range shown.

Tokens to the target

Post-hoc target · ratio to Muon

Below 1 means SAMuon needed fewer tokens. The target was chosen after early results, so this is exploratory; read the method.

Lab notebook

Expert view

Five seeds · 100 steps · 25,600 tokens per run

Fig. 1 Equal tuning grid

Seven learning rates per optimizer and size: 84 official trials. AdamW's selected LR was still at the high grid boundary for the first two sizes. Source: main_tuning.csv.

Fig. 2 Calibration

Muon beat AdamW in all five matched tinyshakespeare seeds. Median final loss: 3.622 vs 4.349 bpb. Source: calibration_summary.csv.

Fig. 3 Spectral probe

Bulk steps were much larger where the head probe was valid, but head validity and profiles changed across checkpoints. The proposed stable mechanism did not reproduce. Source: hvp_features.csv.

Fig. 4 Optimizer-only wall time

Interleaved, 30-repeat, one-thread CPU benchmark on a shared machine; preliminary and not a GPU-overhead estimate. Source: optimizer_bench_summary.csv.

Fig. 5 Paper-scale comparison

This test used byte tokens, short contexts and CPU kernels. The paper used 124M–1B-parameter GPU models; its large-scale result remains untested here. Source: paper_scale.csv.

Table 1 Fixed-budget results

Median final validation bits per byte (five seeds). Brackets give seed min–max. Source: main_summary.csv.
cd 004-samuon
python test_optimizers.py
python test_harness.py
python test_probe.py
python study.py
python analyze.py
python verify_results.py

How we checked

Six completed · author review not recorded

Case 004 / findings / 3 October 2026

What the measurements show

Answer: At 0.056M–0.788M parameters, SAMuon had a slightly lower median final loss at all three sizes. Tokens to a disclosed post-hoc target were mixed: 0.867, 1.015 and 0.839 times Muon. The paper's spectral-stability mechanism did not reproduce.

Original authors: Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland · Read the paper ↗

Tested 2–3 October 2026Published 2026-10-03Raw CSV below

Median held-out validation bits per byte over five seeds, at a common 25,600-token budget. Curves show steps 10–100; step zero has a different initial scale. The chart uses a derived, downloadable loss-curves CSV.

What the data says.

This table contains the measured comparison behind the answer. The expert sections below explain how the numbers were produced and what they cannot establish.

Non-embedding sizeMuon final bpbSAMuon final bpbTokens-to-target ratio
0.056M4.0174.0100.867
0.231M3.9343.9171.015
0.788M3.8923.8510.839

Claim-by-claim verdicts.

Confirmed means supported within the stated test; Partly confirmed keeps a tested piece separate from an untested part. Not confirmed means the tested evidence did not show the claim. Could not test marks missing evidence or an inapplicable setup.

ClaimVerdictEvidence boundary
SAMuon lower median final lossPartly confirmedYes at all three small sizes, not every seed or paper-scale configuration.
Consistent tokens saved versus MuonNot confirmedPost-hoc target ratios 0.867, 1.015, 0.839; middle size loses.
Muon better than AdamW calibrationConfirmedMatched tinyshakespeare seeds: 3.622 versus 4.349 bpb.
Stable spectral mechanismNot confirmedAnisotropic probes appeared, but head validity and profiles were unstable.
Boost 1 equals MuonConfirmedExact update and five-seed curve identity checks.
124M–1B paper-scale resultCould not testCPU models and training budget are about 1,000× smaller.

Expert detail: measurements and interpretation

The three bias-free byte-level GPTs have 55,536, 230,960, and 787,584 non-embedding parameters. Each optimizer received the same seven-learning-rate trial budget per size before five full-horizon seeds. Main runs used 100 steps and 25,600 training tokens. A matched Muon-versus-AdamW calibration passed on tinyshakespeare: Muon median 3.622 bpb and AdamW 4.349 bpb. AdamW's winning learning rate still touched the high end of its grid for two sizes, limiting the baseline comparison.

The tokens-to-target ratio uses the first downward crossing of a validation target. The target was chosen after inspecting initial results: 0.02 bpb above the worst final Muon loss at each size. It is a disclosed post-hoc diagnostic, not a preregistered effect estimate. A ratio below one means SAMuon used fewer tokens. The five-seed bootstrap intervals are coarse; the middle size's median ratio is above one. A separate paper-style per-seed target analysis includes censored non-crossings in the linked CSV.

Held-out spectral probes did show directional differences, but the head was not valid at every checkpoint and profile ranks varied across widths. The proposed stable spectral allocation mechanism did not reproduce here. These models use CPU kernels, short contexts, byte tokens, and different embedding treatment from the GPU study. They are about 1,000× smaller than the 124M–1B paper models, so these results neither confirm nor refute the large-model outcomes.

How we tested

We checked optimizer updates against numerical oracles, calibrated Muon against AdamW, gave each optimizer the same learning-rate sweep, then ran five full-horizon seeds per size. Validation was measured every ten steps. Separate held-out batches and Hessian-vector products probed the spectral explanation. Curves, final-loss summaries, calibration, ratios and probe features are downloadable.

Limitations and next check

This is a short, CPU-only byte-model diagnostic. The post-hoc target is exploratory and the five-seed intervals are coarse. The models, embedding treatment and compute budget differ materially from the 124M–1B GPU study; neither its losses nor its token-saving percentages are confirmed or refuted.

Original sources, data and corrections

The ideas and original claims belong to the paper authors. Kael Research is responsible for these implementations, checks and interpretations. Original source: Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon (2026-08-26)
By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland. Download each public CSV below; the full experiment repositories, including any larger uncopied raw artifacts, are available on request. For corrections, see our public log or contact us through the site.

main_summary.csv ↧token_ratio_summary.csv ↧calibration_summary.csv ↧width_profile_summary.csv ↧hvp_features.csv ↧paper_style_ratio_summary.csv ↧main_tuning.csv ↧loss_curves.csv ↧paper_scale.csv ↧

Got a paper? We'll test it.

Paper: Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland, “Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon”, arXiv 2608.25990 (PDF). All credit for the ideas belongs to the authors; the small-scale implementations and measurements here are ours. Tested 2–3 October 2026.