Non-record: 11L GEPA + 25k Steps + Pure Int6 + Legal TTT (val_bpb=1.0944) - unlimited compute category

[Christopher-Lee-McClendon]

Summary

• val_bpb = 1.0944 — new personal best with legal score-first TTT
• 11L GEPA architecture (27M params) trained for 25000 steps (12000 peak-LR + 13000 warmdown)
• Pure int6 per-row quantization with 15-candidate GPTQ-lite + zstd-22 compression
• Legal score-first TTT (SGD, momentum 0.9, 10 epochs): −0.014 BPP gain
• Artifact: 13.83 MB (14,496,936 bytes) — smallest in our series
• Includes model artifact (final_model.int6.ptz) for reproducibility

Key Result

Metric	Value
Float base (25k steps)	1.1088
After legal TTT	1.0944
Eval time	2,074s on 4×A100-40GB
Training wallclock	12,509s (~3h 28m)

Scaling Law (5 data points, warmdown is the dominant lever)

Steps	Peak-LR	Warmdown	Float Base	TTT BPP	Artifact
9,000	5,000	4,000	1.135	1.116	14.94 MB
12,000	7,000	5,000	1.127	1.108	14.79 MB
15,000	9,000	6,000	1.122	1.104	14.52 MB
20,000	12,000	8,000	1.115	1.098	14.22 MB
25,000	12,000	13,000	1.109	1.094	13.75 MB

All three metrics improve monotonically: float base, TTT BPP, and artifact size.

Key Insight: Warmdown Acceleration

The BPP improvement accelerates in the final warmdown steps despite the cosine LR schedule decelerating:

• Steps 20k→21k: −7.4 BPP/kstep
• Steps 22k→23k: −12.0 BPP/kstep
• Steps 22k→25k: −14.0 BPP/kstep

This suggests fine-grained optimization at low LR is disproportionately effective.

Non-record unlimited-compute submission (4×A100-40GB, ~3.5 hours).

Prior Submissions in This Series

• PR Non-record: 11L GEPA + 20k Steps + Pure Int6 + Legal TTT (val_bpb=1.0983): unlimited compute: 4×A100-40GB, ~2.8 hours #628: 20k steps → 1.0983 BPP
• PR Non-record: 11L GEPA + 12k Steps + Pure Int6 + Legal TTT (val_bpb=1.1079) #612: 12k steps → 1.1079 BPP
• PR Non-Record: BPB 1.1334 — 7000-Step Training + Mixed Int6/Int8 Quantization + Legal TTT #598: 7k steps → 1.1334 BPP

Acknowledgments

Builds on techniques from: @signalrush (PR #414, GPTQ-lite/EMA), @jfprincz (PRs #287/#315, XSA/Partial RoPE/LN Scale), @unnir (PR #265, Efficient XSA), raahilshah (PR #162, SmearGate/BigramHash), @aruniyer (PR #86, Int6 QAT), samacqua (LoRA TTT), @abaybektursun (PR #549, LeakyReLU²), and the OpenAI baseline.

[MatoTeziTanka]

Community Review — Non-record: 11L GEPA + 25k Steps + Pure Int6 + Legal TTT (val_bpb=1.0944) - unlimited compute category

BPB: 1.0944 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA a1fa11bd90a8, file records/track_non_record_16mb/2026-03-24_11L_GEPA_25kSteps_PureInt6_LegalTTT/train_gpt.py):

The TTT path at line 399 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.07s, dim=512, layers=11, vocab=1024, code=78689 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.07s, dim=512, layers=11, vocab=1024, code=78689 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.