Can a hybrid architecture beat the Transformer?
We trained two hybrid architectures — TAM v3 and Cortex-S — at 100 million parameters on 2 billion tokens and compared them head-to-head against a matched standard Transformer, on identical data, hardware, and seeds.
The architectures
Both TAM and Cortex-S add a learned recurrent state alongside standard attention — letting the model carry information across tokens without paying the cost of full quadratic attention at every layer.
TAM v3 — Temporal Associative Memory
Each block runs reduced-width causal attention and a diagonal affine scan (recurrent world-state) in parallel. A learned scalar gate mixes the two:
output = 2 × ((1−g) × attention + g × world_state)
The scan is parallel-associative, so training stays fast. The gate learns to route ~55% through the world-state branch and ~45% through attention.
Cortex-S — Safe Recurrent MoE
Sparse Mixture-of-Experts (8 experts, top-2 routing) with persistent recurrent state (128 per layer). Full attention runs only every 6th layer — the rest use the recurrent pathway.
Includes a deterministic safety kernel boundary — the architecture's compute is always auditable and bounded.
🔬 Matched experimental design
Every variable is controlled so the only difference is the architecture itself:
Parameters — matched within 0.003%
TAM: 101,806,616 · Transformer: 101,803,520
Tokens — identical byte-for-byte
FineWeb-Edu, FineMath, StackV2, Cosmopedia, ArXiv
Hardware — pinned GPU class
bfloat16 precision, torch.compile, AdamW
Same seed — identical initialization
Same optimizer, LR schedule, batch size
Results at 100M / 2B tokens
The headline numbers from the full pretraining run.
Full pretraining + post-training comparison
| Metric | TAM v3 | Transformer | Winner |
|---|---|---|---|
| Pretrain NLL ↓ | 2.6984 | 2.7116 | TAM |
| Pretrain perplexity ↓ | 14.86 | 15.05 | TAM |
| SFT assistant NLL ↓ | 1.7839 | 1.7947 | TAM |
| DPO reward accuracy ↑ | 62.0% | 62.6% | TFM |
| Final mixture NLL ↓ | 2.8439 | 2.8647 | TAM |
| Throughput (tok/s) ↑ | ~254.8k | ~319.9k | TFM |
| Wall-clock time ↓ | ~9,196s | ~6,917s | TFM |
NLL = negative log-likelihood (lower is better). PPL = perplexity (lower is better). Both post-trained with SFT on SmolSmolTalk (100K examples) + DPO on UltraFeedback (10K pairs). Training on 6-source data mixture at context length 512.
Scaling trend: TAM wins at every scale
TAM v3 was screened at 25M and 50M parameters (3 seeds each, 10M tokens) before the full 100M/2B run. It won on loss at every tested scale.
| Scale | TAM v3 NLL | Transformer NLL | TAM wins | Throughput ratio |
|---|---|---|---|---|
| 25M10M tokens · 3 seeds | 7.162 | 7.207 | 3 / 3 | ~86% |
| 50M10M tokens · 3 seeds | 6.889 | 7.064 | 3 / 3 | ~82% |
| 100M2B tokens · 1 seed | 2.698 | 2.712 | 1 / 1 | ~80% |
The NLL gap widened from 25M to 50M — a promising signal. At 100M the gap narrowed slightly due to the much larger token budget (2B vs 10M), but TAM still wins on loss.
Downstream benchmarks (100M / 2B)
Standard multiple-choice evaluation after post-training. The Transformer wins most downstream tasks despite losing on raw loss — a known "NLL ≠ capability" phenomenon at small scale.
| Benchmark | TAM v3 | Transformer | Delta |
|---|---|---|---|
| ARC-Easy | 31.5% | 35.5% | −4.0 pp |
| ARC-Challenge | 26.0% | 25.5% | +0.5 pp |
| PIQA | 54.0% | 58.0% | −4.0 pp |
| HellaSwag | 27.5% | 29.5% | −2.0 pp |
| OpenBookQA | 28.5% | 29.5% | −1.0 pp |
| GSM8K | 2.0% | 2.0% | 0.0 pp |
| Five-MCQ mean | 33.5% | 35.6% | −2.1 pp |
Synthetic state & memory probes
Four tasks (delayed recall, associative recall, state tracking, needle retrieval) tested at seven context lengths with 96 paired trials each. TAM shows its strongest advantage at 256 tokens — exactly where the recurrent state branch is most useful.
| Context length | TAM − Transformer | 95% CI | Signal |
|---|---|---|---|
| 128 | −2.08 pp | −7.03 to +2.60 | |
| 256 | +7.81 pp | +2.60 to +13.54 | Significant |
| 384 | +2.86 pp | −2.60 to +8.33 | |
| 512 | −0.52 pp | −5.73 to +4.69 | |
| 640 | −1.04 pp | −6.51 to +4.69 | |
| 768 | −0.78 pp | −6.25 to +4.69 | |
| 1000 | +1.04 pp | −4.43 to +6.51 | |
| Aggregate | +1.04 pp | −1.00 to +3.05 |
At 256 tokens, TAM's state tracking was +11.5 pp and needle retrieval +12.5 pp over the Transformer (CI: +2.1 to +22.9). The advantage fades at longer contexts where full attention already captures the dependencies.
Cortex-S: a second architecture confirms the signal
A separate sparse MoE + recurrent design, trained on the same frozen 2B corpus.
| Metric | Cortex-S | Transformer |
|---|---|---|
| Parameters | 101,778,112 | 101,803,520 |
| Final NLL ↓ | 2.7091 | 2.7116 |
| Final perplexity ↓ | 15.015 | 15.054 |
| Architecture | Sparse MoE + recurrent | Standard causal attention |
Cortex-S uses 8 experts (top-2 routing), 24 layers with full attention only every 6th layer, and persistent state of size 128 per layer. A deterministic safety kernel bounds every inference step.
Honest verdict
TAM v3 is not a breakthrough — but the signal is real.
- Lower NLL/PPL at every scale tested (25M, 50M, 100M) — the recurrent state genuinely helps language modelling.
- Significant state/memory advantage at 256 tokens — the world-state branch carries information that attention alone drops.
- Two independent architectures (TAM + Cortex-S) both beat the Transformer on loss — this is not a fluke of one design.
But:
- The Transformer is ~33% faster in wall-clock time. Per-FLOP, the Transformer may still win.
- The NLL advantage did not translate to downstream MCQ benchmarks — the Transformer wins 4 out of 5.
- No broad long-context advantage beyond 512 tokens — persistent state alone is not enough.
Next step: TAM v4 (persistent cross-chunk state, novelty write gate, dynamic routing) will be tested at small scale before committing GPU budget. Scaling TAM v3 past 100M is not justified.
Training protocol
Data
FineWeb-Edu (45%), FineMath (17.5%), StackV2 (15%), Cosmopedia (15%), ArXiv (7.5%) — 2B tokens, GPT-2 tokenizer, context 512
Optimizer
AdamW, β₁=0.9, β₂=0.95, LR 3×10⁻⁴ cosine, weight decay 0.1, gradient clip 1.0
Batch
Micro-batch 64 × gradient accumulation 2 = effective batch 128, bfloat16 mixed precision
Post-training
SFT on SmolSmolTalk (100K examples), then DPO on UltraFeedback (10K preference pairs)