Record: GDN-Hybrid (Gated DeltaNet + Sliding Window Attention) - quantized_bpb 1.02046

[joshkmartinez]

Summary

• quantized_bpb = 1.02045733 (3-seed mean, std 0.00553017) | 15.31–15.83 MB | 8×H100 SXM
• derived from the GDN-Hybrid line of PR Record: GDN-Hybrid (Gated DeltaNet + SWA) — val_bpb 1.028308 (3-seed cold-cache mean) #1545

3-Seed Results

Seed	Steps	EMA BPB	Quantized BPB	XSA BPB	Artifact bytes
42	1864	1.017723	1.026791	1.031731	15,313,984
1337	2239	1.007375	1.016586	1.020691	15,830,308
2024	2241	1.008736	1.017995	1.023138	15,820,201
Mean	—	1.011278	1.02045733	1.025187	15,654,831.00
Std (sample)	—	—	0.00553017	—	—

Architecture

This submission uses an SP1024-tokenized GDN-Hybrid backbone with the following high-level structure:

[GDN×5] → SWA → [GDN×5] → SWA_shared

Key components:

1. SP1024 tokenizer
2. Gated DeltaNet hybrid backbone
3. Sliding-window attention side path
4. MuonEq-R + AdamW training mix
5. EMA = 0.997
6. Late QAT threshold = 0.15
7. GPTQ int6 + zstd-22 packaging

Credits

• PR Record: GDN-Hybrid (Gated DeltaNet + SWA) — val_bpb 1.028308 (3-seed cold-cache mean) #1545 — GDN-Hybrid architecture line ([GDN×5] → SWA → [GDN×5] → SWA_shared)
• PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 — prior SP1024 / no-TTT / quantized record-track lineage
• PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 — earlier record-track baseline