Research · Small-scale programme

August 21, 2026 · 10 min read · Code included

Small models that actually reason: STLM and Mini-SLA

Not every insight needs a supercomputer. On a single consumer GPU, in minutes, we trained two tiny architectures — one that pins words to meaning, one that separates memory from thinking — and raced each against a matched Transformer baseline on the same data and seed. Both small models predict text better, and one of them solves negation with no help at all.

11–23M parameters Consumer GPU WikiText-2 + toy corpus Matched baselines Fully reproducible

⚡ The 2-minute version

STLM (Semantic Topology Language Model) gives every word a coordinate on meaning axes like hot–cold and true–false. Result: perplexity 80.89 vs 111.45 against its baseline, embeddings where opposites are genuinely opposite (correlation 0.957 vs −0.069), and 35/35 on negation with constraint guidance.
Mini-SLA loops a single Transformer layer 4 times around an explicit memory database with a symbolic negation router. At matched size (11.28M vs 11.22M params) it reaches perplexity 63.89 vs 111.45 — and cracks negation 35/35 with zero help at inference.
The honest trade-off, twice confirmed: the plain Transformer still wins subject-verb grammar (46/50 both times). Meaning up, syntax slightly down.
Everything below is reproducible — the full training code ships inline in this article. No hidden repositories.

The idea in one picture

A plain Transformer stores everything it knows smeared across its weights — including the fact that hot is the opposite of cold. Our two small models refuse to do that. Each gives meaning an explicit home:

🔁 Classic tiny Transformer

All knowledge lives in the weights. Hot and cold appear in similar sentences, so the model files them as similar — and completes "the door is not…" with open. It scores 0/35 on negation.

🧭 STLM + Mini-SLA

STLM pins each word to coordinates on 7 meaning axes (temperature, size, brightness, speed, door-state, truth, life-state). SLA keeps facts in an explicit memory database and routes not through a symbolic switch that flips what it retrieves.

The two challengers

STLM — meaning has coordinates

Same Transformer core as the baseline, plus three small additions: a topology classifier per word (which meaning axis?), a coordinate regressor (where on the axis?), and one reusable axis embedding per topology.

Training loss is joint: LM + 0.4 × topology + seed-coordinate. At inference an optional constraint bias nudges generation along the learned axes. Topology alignment reaches 98.3%.

Mini-SLA — memory, separated from thought

One Transformer layer, applied 4 times recurrently, wrapped around an explicit causal key-value memory with 10 entity slots: gated writes in, attention reads out.

A symbolic negation router blends each retrieval with its negation whenever not is in play — which is why SLA needs no constraint injection at all. Pure language-model loss, nothing else.

🔬 A fair fight: matched experimental design

Same size class, same corpus, same seed, same optimizer — the only difference is the architecture:

±0.6%

Parameters — SLA 11,281,469 vs baseline 11,217,337
STLM uses the same core plus a small topology overhead

25k+

Lines — WikiText-2 plus 214 hand-written state lines
plus seed-graph augmented negation templates

Same seed — identical data splits (85/5/10)
AdamW, lr 2e-4, batch 32, 8 epochs, clip 1.0

~9 min

Consumer GPU — baseline 498 s, SLA 568 s
held-out test split, never trained on

Results: matched comparisons

Lower perplexity (PPL) means better text prediction. Higher accuracy and correlation mean better understanding. Bold marks the winner.

63.89Mini-SLA Perplexity

111.45Baseline Perplexity

0.957STLM Meaning Correlation

−0.069Baseline Correlation

STLM vs matched Transformer baseline

Metric	STLM	Baseline	Winner
Test perplexity, plain ↓	80.89	111.45	STLM
Test perplexity, constrained ↓	69.09	—	STLM
Top-1 next-word ↑	15.30%	10.74%	STLM
Top-5 next-word ↑	34.37%	26.54%	STLM
Meaning correlation (Pearson) ↑	0.957	−0.069	STLM
Meaning correlation (Spearman) ↑	0.938	−0.053	STLM
Negation suite, plain (35)	0 / 35	0 / 35	Tie
Negation suite, constrained (35)	35 / 35	0 / 35	STLM
Grammar agreement (50)	40 / 50	46 / 50	TFM
Topology alignment ↑	98.3%	n/a	STLM

Constraints only work because STLM aligned its embeddings to coordinate axes during training — the same bias applied to the baseline changes nothing (0/35). Grammar is the one benchmark the baseline keeps.

Mini-SLA vs matched Transformer baseline

Metric	Mini-SLA	Baseline	Winner
Parameters	11,281,469	11,217,337	±0.6%
Depth	1 layer × 4 passes	4 stacked layers
Test perplexity ↓	63.89	111.45	SLA
Top-1 next-word ↑	24.18%	10.74%	SLA
Top-5 next-word ↑	48.72%	26.54%	SLA
Negation suite (35), no help	35 / 35	0 / 35	SLA
Grammar agreement (50)	41 / 50	46 / 50	TFM
Training time ↓	567.8 s	498.3 s	TFM

The standout cell is negation: SLA solves all 35 probes autonomously — plain evaluation, no constraint bias — while the baseline scores zero. Memory-computation separation costs about 14% extra training time.

Bonus: the saved checkpoints, three-way

Evaluating the actual saved checkpoints (different sizes, 8 epochs each) on identical test sentences — with one honest caveat attached:

Checkpoint	Params	Test PPL ↓	Top-1 ↑
STLM (full, d=256)	23,472,141	56.61	29.64%
Baseline (d=128)	10,813,213	210.79	21.20%
Mini-SLA (d=128)	10,252,969	230.36	20.09%

Caveat: at ~10M params and only 8 epochs, the simpler baseline optimizes faster on raw text prediction (210.79 vs 230.36) — memory networks need longer runs for their write/read heads to converge. All three checkpoints sit near chance on negation and grammar at this budget, which is exactly why the matched, fully-converged comparisons above are the headline results.

Honest verdict

Two tiny models, two real signals — and the same trade-off our 100M programme found.

✅Explicit meaning beats smeared meaning. Coordinates (STLM) and separate memory (SLA) both cut perplexity nearly in half versus matched baselines — 63.89 and 80.89 against 111.45.
✅Negation is solvable at tiny scale. SLA goes 35/35 with no inference help; STLM goes 35/35 with axis guidance. The baseline scores 0/35 either way.
✅Embeddings with geometry. STLM's 0.957 meaning correlation vs −0.069 says its vector space understands opposites — the baseline's does not.

But:

❌The Transformer keeps grammar both times (46/50 vs 40–41/50). Structuring for meaning costs a little syntax.
❌Memory networks are harder to optimize — at small budgets the plain baseline converges faster on raw prediction.
❌STLM still needs its constraint bias at inference for negation (0/35 plain). Only SLA reasons fully autonomously.

Next step: longer training runs for the memory heads, then scaling the SLA router idea upward. Small models are the perfect lab — an idea that works at 11M params earns its GPU budget at 100M.

Training protocol

Data

25,000 WikiText-2 lines + 214 hand-written state sentences + seed-graph negation templates across 7 topologies — word-level vocab, 38,941 tokens

Optimizer

AdamW, lr 2e-4, gradient clip 1.0 — identical for model and baseline, seed 42, 85/5/10 train/val/test split

Batch

Batch 32, context 96, 8 epochs for matched runs (12 for the full STLM) — one consumer GPU, minutes not days

Probes

35 negation minimal pairs, 50 grammar minimal pairs, 30-pair meaning-correlation suite, top-1/top-5 accuracy — all on held-out data

Replicate it: the full training code

The exact protocol from this article, condensed from the working sources. Same classes, same losses, same hyperparameters — train a model and its baseline, then run the probes yourself.

0Requirements and launch

terminalCPU or one consumer GPU

pip install torch numpy tqdm datasets

python src/compare_comprehensive.py   # STLM vs baseline, 5 benches
python src/compare_sla.py             # Mini-SLA vs baseline
python src/train.py --epochs 12       # full STLM checkpoint

1STLM core — baseline Transformer plus topology coordinates and the joint loss.

stlm_core.pymodel + loss

import torch
import torch.nn as nn
import torch.nn.functional as F

class STLMini(nn.Module):
    """Baseline Transformer core plus meaning coordinates:
    one axis per topology, one classifier and one
    coordinate per vocabulary word."""
    def __init__(self, vocab, word_to_id, topologies,
                 d_model=128, n_layers=4, n_heads=8,
                 max_len=96, constraint_strength=0.35):
        super().__init__()
        self.word_to_id = word_to_id
        self.topo_to_id = {n: i for i, n in enumerate(topologies)}
        self.token_emb = nn.Embedding(vocab, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)
        layer = nn.TransformerEncoderLayer(
            d_model, n_heads, dim_feedforward=d_model * 4,
            batch_first=True, dropout=0.1)
        self.encoder = nn.TransformerEncoder(layer, n_layers)
        self.lm_head = nn.Linear(d_model, vocab)
        self.topology_coord = nn.Embedding(len(topologies), 1)
        self.concept_topo = nn.Embedding(vocab, len(topologies))
        self.concept_coord = nn.Embedding(vocab, 1)
        self.strength = constraint_strength

    def forward(self, input_ids, constraint_bonus=None):
        seqlen = input_ids.size(1)
        pos = torch.arange(seqlen, device=input_ids.device).unsqueeze(0)
        x = self.token_emb(input_ids) + self.pos_emb(pos)
        mask = torch.triu(torch.ones(seqlen, seqlen,
                                     device=input_ids.device)
                          * float("-inf"), diagonal=1)
        logits = self.lm_head(self.encoder(x, mask=mask))
        if constraint_bonus is not None:
            bonus = constraint_bonus
            if bonus.dim() == 1:
                bonus = bonus.unsqueeze(0)
            logits = logits + self.strength * bonus.unsqueeze(1)
        return logits

    def topology_loss(self, input_ids):
        """Pull each token coordinate toward its topology axis."""
        probs = F.softmax(self.concept_topo(input_ids), dim=-1)
        coords = self.concept_coord(input_ids).squeeze(-1)
        axis = self.topology_coord.weight.squeeze(-1)
        target = torch.matmul(probs, axis)
        return F.mse_loss(coords, target)

# Joint training loss (baseline uses plain lm_loss only):
# loss = lm_loss + 0.4 * model.topology_loss(x) + seed_coord_loss

2Mini-SLA core — one layer looped 4 times around an explicit memory with a negation router.

sla_core.pymodel, plain LM loss

import torch
import torch.nn as nn
import torch.nn.functional as F

class MiniSLA(nn.Module):
    """One Transformer layer, looped recurrently, wrapped around
    an explicit key-value memory with 10 entity slots and a
    symbolic negation router. No constraint injection needed."""
    def __init__(self, vocab, word_to_id, d_model=128,
                 n_heads=8, max_len=96,
                 recurrent_steps=4, num_entities=10):
        super().__init__()
        self.word_to_id = word_to_id
        self.recurrent_steps = recurrent_steps
        self.num_entities = num_entities
        self.token_emb = nn.Embedding(vocab, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)
        self.encoder_layer = nn.TransformerEncoderLayer(
            d_model, n_heads, dim_feedforward=d_model * 4,
            batch_first=True, dropout=0.1)
        self.q_proj = nn.Linear(d_model, num_entities)
        self.w_proj = nn.Linear(d_model, d_model)
        self.g_proj = nn.Linear(d_model, 1)
        self.mem_combine = nn.Linear(d_model, d_model)
        self.router = nn.Linear(d_model, 1)
        self.lm_head = nn.Linear(d_model, vocab)

    def forward(self, input_ids, return_debug=False):
        bsz, seqlen = input_ids.shape
        pos = torch.arange(seqlen, device=input_ids.device).unsqueeze(0)
        h = self.token_emb(input_ids) + self.pos_emb(pos)
        mask = torch.triu(torch.ones(seqlen, seqlen,
                                     device=input_ids.device)
                          * float("-inf"), diagonal=1)
        for _ in range(self.recurrent_steps):
            h = self.encoder_layer(h, src_mask=mask)
        memory = torch.zeros(bsz, self.num_entities, h.size(-1),
                             device=input_ids.device)
        reads, routes = [], []
        for t in range(seqlen):
            h_t = h[:, t, :]
            probs = F.softmax(self.q_proj(h_t), dim=-1)
            probs = probs.unsqueeze(-1)
            read = (memory * probs).sum(dim=1)
            route = torch.sigmoid(self.router(h_t))
            routes.append(route)
            routed = (1.0 - route) * read + route * (-read)
            reads.append(routed)
            write = self.w_proj(h_t)
            gate = torch.sigmoid(self.g_proj(h_t)).unsqueeze(-1)
            memory = ((1.0 - gate * probs) * memory
                      + (gate * probs) * write.unsqueeze(1))
        combined = h + self.mem_combine(torch.stack(reads, dim=1))
        logits = self.lm_head(combined)
        if return_debug:
            return logits, memory, reads, routes
        return logits

3Train + probe — identical loop for model and baseline, then the held-out probes.

train_and_probe.pysame loop, both models

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

opt = torch.optim.AdamW(model.parameters(), lr=2e-4)
for epoch in range(1, epochs + 1):
    model.train()
    for input_ids, labels in train_loader:
        input_ids = input_ids.to(dev)
        labels = labels.to(dev)
        logits = model(input_ids)  # SLA: plain LM loss, no extras
        loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)),
                               labels.reshape(-1),
                               ignore_index=-100)
        # STLM instead: loss = lm + 0.4 * topology_loss + seed_loss
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    val_nll = evaluate(model, val_loader)  # mean NLL on held-out
    print("epoch", epoch, "val_ppl", round(math.exp(val_nll), 2))

# Probes (see src/compare_comprehensive.py for the full suites):
# - 35 negation pairs: hit when the top token is expected, not rejected
# - 50 grammar pairs: hit when P(correct) is above P(broken)
# - 30 meaning pairs: Pearson and Spearman over embedding similarity

Train it, beat it, tell us

The code above reproduces our runs end to end on a single consumer GPU. If your variant beats SLA on autonomous negation and keeps its perplexity — we want to hear about it at hej@belna.se.

Try Arche 1.0 while it trains →

Home · Research · Pricing