Bittensor subnet 100 · BASE · PRISM challenge

Train the best language model
in six hours, from nothing.

Every miner gets the same random start, the same 0.82 billion tokens, the same six hours, the same GPU. Only two files differ: the model and the training recipe. This is our measurement log.

Our best
Chain #1
Gap

The race

bits per byte · lower is better

Score is bpb — how many bits the model needs per byte of held-out text. The uniform-guess floor is 15.617; everything below is learning. Emission is winner-take-all, so second place earns exactly what last place earns.

our runs other miners

Every dot is a completed six-hour run. Hover for detail.

What actually moves the score

measured, not assumed

Each bar is a single change measured against an otherwise identical run. Negative is an improvement. The headline: the learning-rate schedule dominates, and model size does nothing at all.

improves score worsens score

Learning rate response

Orthogonalised-momentum group, full six-hour runs at 218.5M.

Monotonic and flattening — 0.04→0.02 buys 0.0054, 0.02→0.01 buys 0.0018. We are at the optimum.

Model size response

Four full runs, identical recipe, 218.5M → 295.6M.

Total spread 0.012 across a 35% range of size, and not monotonic. Chain #1 is a 130.7M model — smaller than anything here.

Every experiment we ran

Eight full six-hour runs and four short ablations. The position bar places each score on the same axis as the race chart above.

RunParamsTok/stepOrtho LRTokensPassesTok/sbpbPosition

"Passes" counts how many times the run swept the 0.82B-token shard. Over-repeating hurts: the best chain result sits near 2.4 passes, and a 3.1-pass run on the same architecture scored 0.11 worse.

Running right now

live at snapshot

Our GPUs (rented)

Five boxes testing whether a smaller model wins, since size proved flat above 218M.

RunParamsOrtho LRTok/sStepState

Our submissions on chain

Training-only entries — our recipe on other miners' architectures. These cost nothing; the subnet operator pays for the GPU.

HotkeyArchitectureStateQueued

Subnet queue

Running
Queued
Scored ever
Failed ever
Rejected

Fewer than one submission in five ever produces a score — the rest die to infrastructure faults or the anti-cheat gate. The queue runs roughly a day deep.

HotkeyArchitectureStateQueued

Borrowable architectures

the registry

Any miner can submit a training recipe against a published architecture. The architecture's owner is then credited with the best score anyone reaches on it — so a good design earns even while someone else trains it.

ArchitectureOwnerBest bpbPosition

How the subnet pays

mechanics worth knowing

Winner-take-all

PRISM's entire share of subnet emission collapses onto a single hotkey each epoch. Rank 2 earns zero. A run that improves our score but doesn't take first place is worth nothing in emission — only in information.

Owner credit, and the tie-break

Beating everyone on a borrowed architecture credits its owner with our exact score — a guaranteed tie. Ties break by lexicographically smallest hotkey, which is why we mined hotkeys beginning 0x000000 and sent them to the architectures whose owners have low-valued keys.

Scoring

metric
mean cross-entropy ÷ ln 2
floor
15.6170 bits (uniform over 50,257)
lattice
round(1,000,000 / (1 + bpb))
val set
256 held-out documents

Only rank matters under winner-take-all; the lattice value is a re-mapping so higher means better.

What gets you zeroed

  • Hardcoded metrics — printing a score instead of measuring one.
  • Missing telemetry — the recipe must report loss as it trains.
  • Non-causal models — any layer that lets position t see token t+1 is reading the answer.
  • Network access beyond the pinned dataset — which is what makes pretrained weights, and therefore fine-tuning, cheating.

A cheat or suspicious verdict is terminal: score 0, no retry.

What we got wrong

the expensive lessons