The Mathematics of Large Language Models
An 8-lecture course for mathematics graduate students.
By Agnès Beaudry and Katherine E. Stange (University of Colorado Boulder), as well as Claude (Anthropic) and ChatGPT (OpenAI).
The purpose of the course is to describe the mathematical structure of an LLM to a mathematics graduate student audience. There is an attached project to build your own mini GPT, with support for learning Python along the way. Created for Math 5810 at University of Colorado Boulder in Fall 2026.
8 lectures + 8 project steps
Complete the installation instructions in advance.
Site map
- Setup & installation — do this before the projects
- Project: build your own LLM
- Resources: annotated readings and videos
Course outline
Lecture 1 — Language models are probability distributions
Autoregressive (text-predicting) language models. Tokens and tokenization. -gram models. Maximum likelihood estimation. Perplexity and other measures of loss.
Project Step 1: Set up, load Shakespeare, and build and test a bigram counting model
Lecture 2 — Neural networks and approximation
Sigmoid, softmax, perceptrons and feedforward neural networks.
Project Step 2: Wire a neural network by hand: an exact diamond classifier
Lecture 3 — Backpropagation and stochastic optimization
Computational graphs, automatic differentiation and gradient descent.
Project Step 3: Build reverse-mode autodiff yourself
Lecture 4 — Attention
Transformers, embeddings, self-attention, causal masking.
Project Step 4: Implement a causal self-attention head
Lecture 5 — The transformer
The full transformer architecture, including layernorm and feedforward layers.
Project Step 5: Assemble the full GPT
Lecture 6 — Training dynamics
Initialization, optimizers (Adam/AdamW), learning rate schedules, and why non-convex optimization works in practice.
Project Step 6: Train your GPT on Shakespeare
Lecture 7 — Tokenization, sampling, and scaling laws
Byte-pair encoding as greedy dictionary compression; the tokenizer as a change of alphabet, and its information-theoretic cost. Decoding as sampling from a tilted distribution: temperature, top-, nucleus sampling. Empirical scaling laws: loss as a power law in model size, data, and compute; the Chinchilla compute-optimal allocation as a constrained optimization problem you can solve in closed form.
Project Step 7: Write your own BPE tokenizer
Lecture 8 — Beyond next-token prediction
From base model to assistant: fine-tuning, RLHF, LoRA, and a glimpse of mechanistic interpretability.
Project Step 8: The whole enchilada