Grokking, reproduced

Every pair (a + b) mod 113 — 12,769 examples — written as four tokens, [a] [+] [b] [=], with the answer read off the last position. A one-layer decoder-only Transformer, 128-dimensional, four heads, written from scratch rather than imported. 30% of the pairs are used for training; the split is drawn once and frozen, so nothing leaks. Full-batch AdamW for 40,000 steps. Chance on a held-out question is 1 in 113 — 0.88%.

Training accuracy reaches 99% at step 81. If you stopped there — which is what you do when your stopping rule is “training loss stopped improving” — you would file this as a textbook overfit and move on. Held-out accuracy doesn't reach 90% until step 1,350. That is 16.7× later, and in between the model is sitting on the answer without being able to state it. A second seed does the same thing on its own schedule: memorized at 82, generalized at 1,100, a 13.4× delay.

This is the run worth staring at. Set weight decay to zero, change nothing else, and generalization never arrives — not late, not slowly, not at all inside 40,000 steps. Training is perfect from step 80 onward. Held-out accuracy ends at 36.0%. And the held-out loss climbs to 19.9, against 4.73 for a model that has learned nothing whatsoever — so it isn't merely failing to generalize, it's becoming catastrophically more confident about wrong answers, for 39,000 steps. The pressure toward smaller weights is what converts a memorized lookup table into the rule.

Holding everything else fixed and varying only how much of the dataset the model may see: at 50% it generalizes essentially as it memorizes (1.1×, steps 79 → 87). At 30%, a 16.7× delay. At 20%, a 77.1× delay — memorized at step 70, generalized at 5,400. The scarcer the examples, the longer memorization remains the better bargain.

My first attempt at this used 50% of the data on mod 97and reported a reproduction. It wasn't one. At that setting the model generalizes 8 steps after it memorizes, and held-out accuracy never drops below 81.9× chance while training holds — there is no plateau, so there is nothing to see and nothing to explain. I had matched the paper's modulus and its architecture but not its operation: Power et al.'s headline figure is division, which is not commutative and is much harder than the addition I was training. So I kept that run in this grid as frac0.5, deliberately. It reproduces the old null result exactly, which is what makes it a control rather than an embarrassment.

Scaling the initialization up by 8× produces the most extreme run in the grid: held-out accuracy falls to 0.2× chance — genuinely worse than guessing — while training accuracy is already perfect, and held-out loss peaks at 12.5against 4.73 for total ignorance. This is what a confidently-wrong memorizer looks like when you measure it instead of describing it. It still recovers to 100% by step 3,800. The direction matches Omnigrok's account, where the size of the weights you start with governs how long the detour through memorization takes.

The first grid logged every 100 steps, which put the first observation at step 100 — after memorization had already finished, so the event I was trying to time was invisible. It had no learning-rate warmup despite my own notes calling for one. It computed grokStepfrom the downsampled export rather than the full history, quantizing the headline number to the plot's resolution. And it applied weight decay uniformly, including to LayerNorm gains, which produced recurring collapses that I nearly wrote up as a finding about weight decay destroying the learned circuit. It wasn't: the run had simply been truncated mid-spike, and every earlier spike had recovered on its own. A measurement that is wrong in an interesting direction is the most expensive kind.