let's build an llm · learning in public
From zero weights to
a machine that answers
3 × DGX Spark
status: phase 0
What this is
One question, answered by building: what does it actually take to make a machine that answers you? Not by fine-tuning someone else's model — by starting from zero weights and doing every stage a frontier lab does, at a scale one person can run and fully understand: data engineering, pretraining, supervised fine-tuning, preference tuning.
The hardware is three NVIDIA DGX Sparks on my desk. The method is learning in public: build everything, understand everything, explain everything — and publish the failures alongside the wins. The end state is a small assistant, trained from scratch, that holds a conversation and knows some true things about the world.
How we keep ourselves honest
the 20-question exam
One fixed exam — simple facts, small tasks, one nonsense question — taken by every model this project trains. The same questions at every scale, so "how much data does sense require?" gets answered as a curve, not a claim. Every transcript published, failures included.
Every number on this site comes from a logged run — config, seed and data recipe published. When something breaks (things will break), the breakage gets written up too.
The stages
The smallest possible assistant
~135M parameters pretrained on just 5B tokens of the most knowledge-dense data available (synthetic textbooks + educational web), then taught to chat. Expectation, honestly set: perfect chat manners, some common facts, plenty of confident nonsense. It takes the exam first, and its score is the baseline everything else gets measured against. This phase also proves the whole pipeline — data, training, checkpoint-resume — before any long run.
The modernization ladder
Modern models all use the same handful of architecture upgrades — RoPE, RMSNorm, SwiGLU, grouped-query attention. Here each one gets earned: two identical training runs, with and without the upgrade, and the loss curves decide. No upgrade gets adopted on reputation.
The two-week run
The real pretraining: ~360M parameters on ~40B tokens, two Sparks in harness for about two weeks. Data mixing, learning-rate schedules, distributed training, crash recovery — the unglamorous craft that pretraining actually is, with a public training diary.
Base → assistant
A pretrained model doesn't answer questions — it continues text. Ask it something and it replies with more questions, because that's what such text looks like. This phase closes the gap: supervised fine-tuning on conversations, then preference tuning. The 360M assistant retakes the exam, and the score-versus-scale chart becomes the project's signature picture.
A language nobody curated for
Reuse the whole validated pipeline where the easy path doesn't exist: a from-scratch model for Tamil, a language with no curated pretraining sets, no instruction data, no ready-made benchmarks. Build the corpus, train our own tokenizer, pretrain, and judge the output the only way possible — by native readers, not leaderboards.