What five billion tokens look like

A language model becomes its training data — nothing more, nothing less. Everything it will ever know has to be on the reading list. So the very first act of this project is choosing what the model reads. This entry is that choice, and the actual dataset it produced.

The idea: knowledge density beats size

Phase 0's model is small (~135M parameters) and its data budget is deliberately tiny by modern standards: 5 billion tokens. When tokens are scarce, what matters is how much learnable knowledge each token carries. So instead of raw web crawl, the mix leans on the two most knowledge-dense open sources available:

Cosmopedia v2 (60%) — synthetic textbooks: millions of textbook-style articles generated specifically to pack facts and explanations into small models. Densest knowledge per token that exists in the open.

FineWeb-Edu (40%) — the real web, filtered hard: only pages a classifier judged genuinely educational survive (roughly the best tenth of Common Crawl). This keeps the model connected to how humans actually write, not just how textbooks read.

Later, for the big Phase-2 run, a third source joins: DCLM, filtered for text that explains and answers rather than lectures — because an assistant needs the shape of answering, not just the facts. Two filters, two different definitions of "good", and the blind spots of each covered by the other.

The recipe, executed

One script streams both sources from the Hub, interleaves them 60/40, tokenizes with a 49,152-piece BPE tokenizer (the same one the reference SmolLM models use, so our results stay comparable), and writes the token stream into fixed-size shards. Nothing is kept except tokens: 5.02 billion of them, as two-byte integers.

The shards

The dataset on disk: fifty training shards of 100M tokens each, plus one held-out shard the model will never train on — the honesty mechanism. A model can ace text it has memorized; only its score on text it has never seen tells the truth. Every loss curve this project publishes will be measured against that red square.

The birth certificate

Every dataset this project builds ships with a meta.json — tokenizer, mix, seeds, counts — so anyone (including future me) can rebuild it exactly:

{
  "tokenizer": "HuggingFaceTB/SmolLM2-135M",
  "vocab_size": 49152,
  "mix": [ cosmopedia-v2 × 0.6, fineweb-edu-dedup × 0.4 ],
  "segments": [ seed 20260905, seed 20260906 ],
  "total_tokens": 5020000000,
  "val_tokens": 20000000,
  "shard_tokens": 100000000,
  "dtype": "uint16"
}

What the model will actually read

The first document we decoded back out of the shards, exactly as the model will see it — a synthetic textbook being unmistakably itself:

Next: the model itself, and the trainer that will feed it these shards. Then the exam.