Build Your Own LLM
Over eight steps—one per lecture—you will build a complete language model from nothing. You begin with a counting model, pause to wire and inspect a small neural network exactly, then build an autodiff engine and train a neural language model. Attention, a full transformer, training, tokenization, and sampling follow. The end product is a “full-stack baby model”: a few-hundred-thousand- to few-million-parameter GPT that generates recognizable Shakespeare, built with your own hands at every layer of the stack.
The model is small. That is the point. Every mathematical phenomenon in the course — cross-entropy as log-likelihood, the geometry of softmax, the low-rank structure of attention heads, the variance bookkeeping of initialization, even rudimentary scaling laws — is visible in a model you can train in minutes and inspect completely.
Ground rules
- If we defined it in lecture, you implement it. No
torch.nn.TransformerDecoder, notorch.nn.MultiheadAttention, noF.scaled_dot_product_attention. You may use PyTorch tensors, autograd (from Step 3 on, after you’ve built your own),nn.Linear,nn.Embedding,nn.LayerNorm, and the optimizers (from Step 6, after you’ve written SGD yourself). The line is: primitives yes, the things this course is about, no. - Mathematics first. Each step opens with derivation exercises done on paper. The code is the check on your mathematics — when your handwritten gradient disagrees with finite differences, one of them is wrong, and it isn’t the finite differences.
- Small budget, real understanding. Each step is designed for 2–4 hours. If you are past 5, you’re stuck on something silly (we all do it); use the hints, ask, or peek at the solution for the previous step.
- AI assistants: off for the core implementation, fine for syntax and error messages. (See the note on the setup page.)
Rhythm and logistics
- Each step is released the day of its lecture and is designed to be finished before the next lecture — later lectures assume you have touched the code.
- Work in a single Jupyter notebook per step (Colab or local—see Setup). Keep them. Most steps extend the previous model; Step 2 is a self-contained construction, and Step 3 returns to the tokenized corpus from Step 1.
- Solutions are posted as notebooks (
solutions/step-01.ipynb…) one week after each step. They are written to be read: full prose between cells, and flagged checkpoint values (loss numbers, shapes, sample outputs) so you can diff your understanding, not just your output.
Joining late / catching up
The project is a chain, but every link is replaceable: each step’s
instructions begin from the previous step’s posted solution. If you join
at Lecture 5, download solutions/step-04.ipynb, run it top to bottom
(~5 minutes), skim its prose, and start Step 5 from there. You lose the
experience of the earlier steps, not the ability to continue — and you can
back-fill earlier steps any time, since they only require earlier lectures.
The eight steps
The last column gives a reference checkpoint: genuine sample text for a generative model, or a visible component-level result when the step builds one part of the system.
| Step | You build | New mathematics exercised | Reference checkpoint |
|---|---|---|---|
| 1 | Character bigram model from counts | MLE, cross-entropy, perplexity | Ths t hethalof finor: |
| 2 | A hand-wired one-hidden-layer ReLU classifier | units as matrix layers; nonlinear features; softmax geometry | exact diamond boundary |
| 3 | A scalar autodiff engine; then an -gram MLP in PyTorch | reverse-mode AD; embeddings; SGD | He halled A by lord-- |
| 4 | A causal self-attention head | attention matrix, masking, softmax geometry | (component) |
| 5 | The full GPT: multi-head + MLP blocks, residual stream, LayerNorm | architecture as composition; parameter counting | (untrained) |
| 6 | The training run: AdamW, schedules, train/val diagnostics | stochastic optimization in practice | WARWICK: / Nor I, the kind dark of that which he had |
| 7 | A BPE tokenizer and a proper sampler | greedy compression; tilted distributions | fluent-ish Shakespeare |
| 8 | Capstone: scale, fine-tune, or interpret | your choice | showcase |
What “done” looks like
By the end of Step 8 you will have, in notebooks written by you:
- a hand-wired neural classifier whose every unit you can trace;
- a reverse-mode automatic differentiation engine (~60 lines);
- a GPT — token embedding, positional embedding, blocks of (LayerNorm → causal multi-head attention → residual, LayerNorm → MLP → residual), final LayerNorm, unembedding — in ~200 lines;
- a training loop with AdamW, warmup/cosine schedule, and gradient clipping, with train/val loss curves you can explain feature by feature;
- a byte-pair-encoding tokenizer trained on your own corpus;
- a sampler with temperature, top-, and nucleus decoding;
- and one experiment of your own design (Step 8), presented in five minutes on the last day.
Every one of these artifacts corresponds to a section of the lecture notes. That correspondence — I can point to the line of code where this theorem acts — is the actual deliverable of the course.