The Mathematics of Large Language Models

Python Primer

~90 minutes · Prerequisites: none — Section 0 below gets you a working notebook in under 5 minutes.

This page assumes you have never written a line of code. That’s fine — everything below is a small, deliberate vocabulary, and you already have the one prerequisite that actually matters: you’re comfortable reading a definition and immediately starting to compute with it. Programming is that, typed into a box that computes back.

How to use this page. Don’t just read the code blocks — type them into a notebook cell and press Shift+Enter to run the cell, for every single one, including the trivial ones. Text after a # is a comment: a note explaining the line for you, the reader, which Python ignores completely. You don’t need to type it — it’s there so you know what’s going on, not as part of the instructions. Then do every Try it exercise before moving on; each has an expected output so you can check yourself. This is the whole method: nothing here is hard, but it only sticks if your hands do it.

This primer covers exactly what you need and nothing more.

0. Open a notebook — and a tutor (~5 min)

  1. Go to colab.research.google.com and sign in with a Google account.
  2. Click New notebook.
  3. You’ll see one empty gray box — that’s a cell. Click inside it.
  4. Type 2 + 2 and press Shift+Enter. You should see 4 appear below the cell, and a fresh empty cell ready underneath it.

That loop — type code into a cell, Shift+Enter, read the output, move to the next cell — is the entire interface for this whole page. Keep adding new cells as you go; don’t worry about tidying up.

One Colab-specific trick you’ll meet in Step 1, worth knowing now so it doesn’t stop you cold later: a cell line starting with ! runs a shell command instead of Python — for example !wget -q <url> downloads a file from the internet straight into the notebook’s storage (§6). Nothing to practice here, just don’t be alarmed by the ! when you see it.

  1. Also open Claude (or Claude Code, if you have it installed) in a second tab, and keep it open the whole time you work through this page. Paste this page’s own URL into your very first message to Claude, so it can see the page itself — what it’s teaching and in what order — rather than guessing from your question alone. Treat it as a tutor sitting next to you, not an answer key: the most useful habit is, when a line confuses you, to first write out in your own words what you think it does, then ask Claude to correct you — you’ll learn faster from being wrong and corrected than from being told the answer cold. This is exactly the workflow the course itself recommends once you reach the real project (see the note at the top of Step 1), so you may as well build the habit now.

1. Variables, types, and running cells (~10 min)

A notebook is a sequence of cells. You run them one at a time (Shift+Enter), in whatever order you choose, and variables persist between cells — the whole notebook shares one running program. This is different from a math paper, where “let x=5x = 5” and “let x=7x = 7” a page later are just two uses of a symbol; in a notebook they are two events, and order matters because each one overwrites the last.

x = 5
y = 7
x = x + 1        # not an equation — an instruction: recompute, then overwrite x
print(x, y)

= is assignment, not equality — read x = x + 1 as “the new value of x is the old value of x, plus one,” which is a perfectly good instruction and a terrible equation. print(...) shows you a value — without it, a cell only displays its very last expression, which is convenient but easy to forget.

Try it — variables. Predict the output on paper first, then run it.

a = 3
b = a + 2
a = 10
print(a, b)

Expected: 10 5b was computed once, from a’s value at that moment (3 + 2), and does not change retroactively when a changes later. This is the single biggest difference from math notation: assignment is an event in time, not a standing equation.

Every value has a type, and Python has four basic ones you’ll use constantly: int (whole numbers, 5), float (decimals, 5.0), str (text, in quotes, "hi"), and bool (exactly two values, True and False — more on these below). type(x) tells you which one you’re holding.

Try it — types.

print(type(5))
print(type(5.0))
print(type("hello"))
print(type(True))

Expected: <class 'int'>, <class 'float'>, <class 'str'>, <class 'bool'> — four different boxes for four different kinds of value. type(...) is your go-to tool whenever you’re not sure what you’re holding, and you’ll reach for it constantly while debugging.

Each of these type names is also callable, like a function — calling one builds a value of that type out of whatever you hand it, which is exactly what type(x) was reporting back to you above: which of these callables would have produced x.

print(int("5"))          # 5 — text, parsed as a number
print(str(5))              # '5' — a number, turned into text
print(list("abc"))          # ['a', 'b', 'c'] — a string broken into a list of characters

You’ll see this same pattern again with list(...) and dict(...) in §3.

The usual arithmetic works as you’d expect (+ - * /), plus three you may not have seen: ** for exponentiation (2102^{10} is 2 ** 10), // for integer (floor) division, and % for the remainder.

Try it — arithmetic.

print(2 ** 10)      # exponentiation
print(17 // 5)       # floor division: how many times does 5 fit into 17?
print(17 % 5)         # remainder: what's left over after that?
print(5 * (17 // 5) + 17 % 5)   # sanity check: quotient*divisor + remainder...

Expected: 1024, 3, 2, 17 — the last line is just 17=53+217 = 5\cdot 3 + 2 written back out, confirming // and % agree with each other.

Comparisons (==, !=, <, <=, >, >=) don’t hand back a number — they hand back one of exactly two values, True or False. That result itself has a type, and its type is bool. This is what “a comparison produces a bool” means, concretely — the comparison’s output is a value of type bool, the same way 2 + 2’s output is a value of type int:

Try it — comparisons and bool.

print(5 > 3)                  # a comparison...
print(type(5 > 3))             # ...and the *type* of what it produced
print(5 == 5.0)                 # int and float can compare equal
print(5 != 3)                    # 'not equal to' — the opposite of ==
print(5 > 3 and 2 > 10)          # 'and' combines two bools into one bool
print(not (2 > 10))               # 'not' flips a bool

Expected: True, <class 'bool'>, True, True, False, True. The second line is the important one: it shows that 5 > 3 isn’t just “truthy text” — it’s a real value living in a box labeled bool, exactly like 5 lives in a box labeled int. You’ll use bool values constantly starting in §4 below, inside if statements.

2. Strings (~10 min)

Text lives in str values, in single or double quotes (no difference — useful when the text itself contains a quote mark). A string is a sequence of characters, so you can pull out one character by position (indexing) or a range of characters (slicing), and you can glue two strings together with +.

name = "Shakespeare"
print(name[0])        # 'S'  — indexing starts at 0
print(name[:5])        # 'Shake' — slice: "up to, not including, index 5"
print(name[-1])        # 'e' — negative indices count from the end
print(name[-3:])        # 'are' — combine the two: "from 3-from-the-end, to the end"
print(len(name))       # 11
print(name.lower())    # 'shakespeare' — a method call

name.lower() is called a method call. lower is the name of the action; the dot means “do this action to name”; the () runs it. So name.lower() means: lowercase name. Every type comes with its own small set of built-in actions like this, written the same way — value, dot, action name, () — and .lower(), .upper(), .split() below are three of the ones strings have.

.lower() and .upper() change case; .split() cuts a string into a list of pieces at whitespace (or at a character you specify); .join() is the reverse — glue a list of strings back together with a separator between each piece.

print(name.upper())                      # 'SHAKESPEARE'
words = "to be or not to be".split()      # cut at whitespace
print(words)                               # ['to', 'be', 'or', 'not', 'to', 'be']
print(", ".join(words))                     # 'to, be, or, not, to, be'

That last line reads backwards from what you’d expect: the method is attached to the separator ", ", not to the list words. Read it as: “using ", " as glue, stick together every item in words.” The separator sits between each pair of items — the result has one fewer comma than words has entries.

These four are the ones you’ll use constantly, and every one of them returns a new string rather than modifying the original — name itself is still "Shakespeare" after all of the calls above. The most useful construction is the f-string: put an f right before the opening quote, and anything written inside {} is no longer plain text — it’s an instruction to insert a value there.

n = 65
print(f"the alphabet has {n} characters")   # {n} is replaced by n's current value

Here {n} gets replaced by whatever n currently holds (65), so the whole thing prints as the alphabet has 65 characters. Without the f, {n} would just print as the four literal characters {n}.

Try it. Given s = "Mathematics", print its length, its last three characters, and an f-string reporting both: f"{s} has {len(s)} letters". Expected: 11, 'ics', Mathematics has 11 letters.

3. Lists, tuples, and dictionaries (~20 min)

A list is an ordered, mutable sequence — think of it as a vector you can grow, shrink, and index into, written in square brackets:

primes = [2, 3, 5, 7, 11]
primes.append(13)          # mutates in place
print(primes[0], primes[-1], len(primes))   # 2 13 6
print(primes[1:4])          # [3, 5, 7] — slicing, same rule as strings

Notice there’s no = on the .append() line — nothing gets reassigned. primes still refers to the exact same list it always did; .append() reaches into that list and changes its contents, in place. This is called mutation, and it’s the opposite of what strings do (§2): name.upper() couldn’t change name itself, it had to hand back a brand-new string, because strings are immutable. Lists are mutable — their methods typically change the object you already have, rather than building you a new one.

A tuple, written in parentheses, is exactly the ordered tuple you already know — (3,4)R2(3,4) \in \mathbb{R}^2 is (3, 4) — and like a mathematical tuple it’s immutable: once built, its entries don’t change. Its main use is assigning several variables in one line, which reads the same way as the math notation:

point = (3, 4)
a, b = point       # exactly "(a, b) = (3, 4)" — so a = 3, b = 4

zip is a function that takes two sequences and lines them up entrywise — the 1st of one with the 1st of the other, the 2nd with the 2nd, and so on — producing a sequence of tuples. zip(...) on its own isn’t in a form you can print or look at directly, so wrap it in list(...), which collects whatever’s inside into an actual list:

print(list(zip([1, 2, 3], [10, 20, 30])))   # [(1, 10), (2, 20), (3, 30)]

Since each item zip produces is a tuple, the unpacking pattern above applies directly inside a for loop, unpacking one pair per iteration:

for a, b in zip([1, 2, 3], [10, 20, 30]):
    print(a, b)      # prints (1, 10), then (2, 20), then (3, 30)

A dictionary is a finite mapping from keys to values — exactly the mathematical object, and its type is called dict. The usual way to write one is the same way you’d write a function table: {key: value, ...}.

prime_index = {2: 0, 3: 1, 5: 2, 7: 3}   # a dict, a finite function
print(prime_index[5])                      # 2 — look up like f(5)
print(5 in prime_index)                    # True — is 5 a key?

You can also call dict(...) as a function, to build one from a sequence of (key, value) pairs — for instance, the pairs zip produces:

print(dict(zip([2, 3, 5, 7], [0, 1, 2, 3])))   # {2: 0, 3: 1, 5: 2, 7: 3}

One more collection type: a set, written set(...) — exactly the mathematical object, an unordered collection with no duplicates:

letters = set("mississippi")   # every distinct character, no duplicates, no fixed order
print(letters)                    # {'m', 'i', 's', 'p'} — order may vary; sets aren't ordered
print(len(letters))                # 4

Since a set has no order, it’s usually paired with sorted(...), which takes any sequence and hands back a new, ordered list:

print(sorted(letters))    # ['i', 'm', 'p', 's'] — alphabetical, as a list now
print(sorted([3, 1, 2]))    # [1, 2, 3] — works on any sequence, not just sets

sorted(set(text)) — a combination you’ll see in Step 1 — reads as: “every distinct character in text, alphabetically.” That’s exactly how Step 1 builds its vocabulary.

Next, comprehensions — just set-builder notation with different brackets. Compare {x2:x{1,,5}}\{x^2 : x \in \{1,\dots,5\}\} with:

squares = [x**2 for x in range(1, 6)]        # list comprehension: [1, 4, 9, 16, 25]
evens = [x for x in range(20) if x % 2 == 0]  # a "such that" clause, same as set-builder
lookup = {x: x**2 for x in range(5)}          # a *dict* comprehension: {0:0, 1:1, 2:4, ...}
print(type(lookup))                             # <class 'dict'> — the {...} really did build a dict

(range(1, 6) generates 1,2,3,4,51,2,3,4,5 — stop is exclusive, same rule as slicing.) You will write exactly this shape of code in Step 1, building a lookup table between characters and integer codes.

Drop the brackets entirely, and a comprehension becomes a generator expression instead — it produces the same values one at a time, without ever building the whole list. You’ll mainly see this as an argument handed directly to another function, especially .join() (§2):

print(sum(x**2 for x in range(1, 6)))     # 55 — no brackets needed inside sum(...)
print(''.join(str(x) for x in range(5)))    # '01234' — same idea, inside .join()

Rule of thumb: storing the result in a variable, use [...]; handing it straight to another function as that function’s one argument, the brackets can come off.

One more building block for that: enumerate, similar in spirit to zip above — instead of pairing two sequences together, it pairs one sequence with its own position (0, 1, 2, …):

print(list(enumerate("abc")))   # [(0, 'a'), (1, 'b'), (2, 'c')]

Try it. The goal: a dictionary where 'a' maps to 0, 'b' maps to 1, and so on through 'e' mapping to 4 — each letter of "abcde" paired with its position. Type this in and run it:

letter_to_index = {ch: i for i, ch in enumerate("abcde")}
print(letter_to_index)

Expected: {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4}.

How it works, piece by piece: enumerate("abcde") produces the pairs (0, 'a'), (1, 'b'), ... as shown just above; for i, ch in enumerate(...) unpacks each pair into two names, i and ch, the same way a, b = point did earlier in this section (i and ch are names I chose, short for “index” and “character” — nothing special about them, you could call them banana and apple instead); and {ch: i for ...} is the dict comprehension from just above, using ch as each entry’s key and i as its value.

4. Control flow: for, while, if (~15 min)

A for loop runs a block once per element of a sequence — it is \sum or \prod, with the summand written out as a statement instead of an expression:

total = 0
for x in range(1, 11):    # x = 1, 2, ..., 10
    total = total + x      # same instruction-not-equation reading as §1
print(total)                # 55

compare this line-for-line with x=110x\sum_{x=1}^{10} x: total starts at the identity for + (namely 0), and each loop iteration is one term of the sum. if / elif / else branch on a condition, and Python uses indentation (not braces) to mark which lines belong to which block — this is the single most common source of early bugs, so when something doesn’t run “the way it’s written,” check the indentation first.

Try it — indentation. Predict how many times each line will print, then run it.

for x in range(3):
    print("inside the loop:", x)
print("after the loop")

Expected:

inside the loop: 0
inside the loop: 1
inside the loop: 2
after the loop

The indented line is part of the loop body, so it runs once per iteration — three times. The unindented line is not part of the loop (it’s back at the same indentation as for itself), so it only runs once, after the loop has completely finished. Now indent the second print to match the first, so both lines sit inside the loop, and run it again: "after the loop" (misnamed now!) prints three times too, once per iteration. Indentation is not decoration — it is the only thing telling Python which lines belong to the loop.

for x in range(1, 11):
    if x % 2 == 0:
        print(x, "even")
    elif x == 7:
        print(x, "lucky")
    else:
        print(x, "odd")

A while loop repeats for as long as a condition holds — unlike for, which always loops a fixed, already-known number of times, while is for when you don’t know the count in advance. One more piece of notation first: steps += 1 is shorthand for steps = steps + 1 — “take the current value, add 1, and store that back” — and it works the same way with any operator (-=, *=, and so on).

n = 100
steps = 0
while n > 1:            # keep looping as long as this comparison is True
    n = n // 2             # halve n (floor division, from §1)
    steps += 1               # shorthand for steps = steps + 1 — count one more halving
print(steps)                   # how many times can 100 be halved before reaching 1?

Try it. Compute k=1100k\sum_{k=1}^{100} k with a for loop, and check it against the closed form n(n+1)/2n(n+1)/2.

total = 0
for k in range(1, 101):
    total += k
print(total, 100 * 101 // 2)

Expected: 5050 5050 — both numbers equal, which is the check.

5. Functions (~15 min)

A function is defined with def, takes parameters, and (optionally) returns a value — the same object as f(x)=f(x) = \dots, just with the body written as statements instead of one expression:

def square(x):
    return x ** 2

def is_prime(n):
    if n < 2:
        return False
    for d in range(2, n):
        if n % d == 0:
            return False
    return True

print(square(5), is_prime(17), is_prime(18))   # 25 True False

Trace how is_prime reaches its last line, return True — this is the subtle part. First, return doesn’t just supply a value, it exits the function immediately: no line after it runs, no matter how much code is left. Now look at return True’s indentation: it lines up with for, so it is not inside the loop body — it’s simply the next statement after the loop ends. The loop itself tries each candidate divisor d in turn, and the moment it finds one that divides n evenly, return False fires and the function exits right there, on the spot — return True never runs in that case. So return True is reached only if the loop makes it all the way through every d from 2 to n-1 without ever finding a divisor. No divisor found is exactly what “prime” means — the function doesn’t compute “is prime” directly, it searches for a counterexample and reports whether that search came up empty.

A parameter can have a default value — used automatically whenever the caller doesn’t supply one:

def greet(name, greeting="Hello"):
    return f"{greeting}, {name}!"   # f-string from §2: splices in the current values

print(greet("Kate"))                  # 'Hello, Kate!' — greeting takes its default
print(greet("Kate", greeting="Hi"))    # 'Hi, Kate!' — a keyword argument overrides it by name

Passing an argument by name like that — greeting="Hi" — is called a keyword argument, and it isn’t limited to parameters with defaults. You’ll see the term constantly from Step 1 onward, e.g. torch.zeros((V, V), dtype=torch.int64), where dtype= is a keyword argument on a function that has nothing to do with this one.

A function can also hand back more than one value at once, by returning a tuple — literally the comma-separated tuple from §3, which the caller then unpacks the same way:

def divide(a, b):
    return a // b, a % b   # a tuple: (quotient, remainder)

q, r = divide(17, 5)         # unpacked immediately, same pattern as §3
print(q, r)                    # 3 2

Last, when a function’s whole body is one expression, there’s a one-line shorthand called a lambda — the same idea as def, just condensed onto a single line, with the expression’s value automatically returned:

double = lambda x: 2 * x     # exactly equivalent to: def double(x): return 2 * x
print(double(21))              # 42

You’ll see this constantly in the project — for instance Step 1 builds encode = lambda s: [stoi[c] for c in s].

A function is allowed to call itself — recursion — which is induction, computationally: a base case, plus a step that reduces to a smaller instance.

def factorial(n):
    if n == 0:               # base case
        return 1
    return n * factorial(n - 1)   # inductive step: n! = n * (n-1)!

print(factorial(5))            # 120

One more statement, useful now that you’re writing code worth checking: assert condition does nothing at all if condition is True, and stops your program with an error, AssertionError, if it’s False. It’s a one-line way to make Python verify something for you, instead of eyeballing the output yourself:

assert 2 + 2 == 4            # True — nothing happens, silently passes
assert factorial(5) == 120     # also True — nothing happens
assert factorial(5) == 121      # False — stops here with an AssertionError

You’ll use this constantly starting in Step 1, to make Python confirm code you just wrote actually does what you meant, rather than trusting it by eye.

Try it. Write is_prime as above (or copy it), then use a list comprehension (§3) to build the list of primes below 30, using is_prime as the filter condition.

primes_below_30 = [n for n in range(2, 30) if is_prime(n)]
print(primes_below_30)

Expected: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29].

Notice the filter clause is if is_prime(n), not if is_prime(n) == True. is_prime(n) already is a bool (§1) — it’s either True or False — so if can use it directly as the condition. Comparing it to True with == would still work, but it’s redundant: you’d be asking “is this True-or-False value equal to True?” when the value already answers that question by itself.

6. Reading a text file (~10 min)

The goal: get the contents of a text file into one Python string you can work with. This is literally the first thing Step 1 does, loading the complete works of Shakespeare. Two pieces of syntax are new — open, and the with statement around it — so try them first on a tiny file you create yourself, rather than one that has to already exist on disk.

Write a file:

with open('demo.txt', 'w') as f:    # 'w' = write mode: create (or overwrite) the file
    f.write('hello, world')

No upload, no separate save step: this line creates demo.txt itself, directly in the notebook’s own temporary storage, the instant it runs. (This storage doesn’t survive Colab disconnecting you for being idle — not a concern here, since we only need demo.txt for the next few minutes.)

Now read it back:

with open('demo.txt', 'r') as f:   # 'with' guarantees the file gets closed
    text = f.read()                  # a method call on f, same idea as name.lower() in §2
print(text)                            # 'hello, world'

Not to be confused with the f in f"..." strings from §2 — same letter, unrelated meaning. Here f is just an ordinary variable name (as arbitrary as ch and i in §3); it happens to be the standard convention for “the open file,” which is why you’ll see it constantly, including in Step 1’s own code.

as f names the open file f — the same job = does everywhere else, just written this way because it’s part of a with statement. Everything indented under it can then call methods on f, which is why f.read() works. with itself closes the file automatically the moment the indented block ends — so you’ll almost never open a file any other way.

Two more pieces of dict syntax (§3) for the exercise below. First, counts[w] = ... creates or updates an entry — the opposite direction from the lookup prime_index[5] you saw in §3:

counts = {}
counts['a'] = 1              # no 'a' key yet, so this creates one: {'a': 1}
counts['a'] = counts['a'] + 1   # look up the old value, add 1, store it back
print(counts)                     # {'a': 2}

Second, .get(key, default) is a safe lookup: counts[key] errors if key isn’t there yet, but counts.get(key, 0) hands back 0 instead — no error. Continuing right on from counts above (still {'a': 2}, in the same notebook, same cell or the next one):

print(counts.get('a', 0))    # 2 — 'a' is a key, so its existing value comes back
print(counts.get('z', 0))     # 0 — 'z' isn't a key, so the default comes back instead

Try it — putting it all together. This is a miniature of Step 1, with words instead of characters. Run it on any short passage — even a sentence you type in directly, no file needed:

passage = "the quick brown fox jumps over the lazy dog the fox runs"
words = passage.split()                       # str.split(): list of words
counts = {}
for w in words:
    counts[w] = counts.get(w, 0) + 1            # w's count so far (or 0), plus 1, stored back
print(counts)
print(counts['the'])

Expected: counts is a dictionary with one entry per distinct word, and counts['the'] is 3'the' is the only word repeated (it appears three times in passage; every other word appears once).

Worth tracing by hand, since counts[w] = counts.get(w, 0) + 1 does two different jobs depending on whether w has shown up before. words is ['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', 'the', 'fox', 'runs'], and counts starts empty. First time w is 'the': counts.get('the', 0) finds no 'the' key yet, so it returns the default, 0, and counts['the'] = 0 + 1 creates a new entry at 1. Second time w is 'the' (further down the list): 'the' is now a key with value 1, so .get returns that real stored value, 1, and counts['the'] = 1 + 1 increments it to 2. Third time, the same line increments it again, to 3. The default 0 stands in for “you’ve never seen this word, so its count so far is zero” — true exactly the first time, false every time after — which is why one line, with no if statement, both creates entries and updates them.

This is the same add-one bookkeeping you’ll do with character pairs, at real scale, in Step 1.

One last piece of syntax, saved for the end because nothing above needed it: import. Tools like print, len, and dict are always available, but most tools live in separate libraries you have to explicitly bring in first:

import math
print(math.sqrt(16))    # 4.0 — sqrt lives inside the math library, reached with a dot

import math makes everything in that library reachable as math.something — the same dot notation as a method call (§2), just reaching into a library instead of a value. A library can also be renamed on import — almost always seen for exactly one library, NumPy, as import numpy as np. You’ll see this constantly starting in Step 1: import torch, import matplotlib.pyplot as plt.

You’re ready

That’s genuinely the whole primer. If you can comfortably do all the Try it exercises above without looking back at the explanations, you’re ready for Step 1 — A Bigram Model from Counts.

If you’re stuck