Lecture 1 — Language Models are Probability Distributions

Project connection. Project Step 1 constructs the bigram model studied below. Numerical values in this lecture refer to the course’s 1,115,394-character Shakespeare corpus.

0. What is a language model

Let $\mathcal{V}$ be a set, called the vocabulary, of finite cardinality $V$. Elements of $\mathcal V$ are called tokens. We write $\mathcal{V}^*$ for finite sequences from $\mathcal{V}$, and for a sequence $x = (x_1, \ldots, x_T)$, we write $x_{<t} = (x_1, \ldots, x_{t-1})$ for its prefix before position $t$. The empty sequence $\varnothing$ is allowed as an element of $\mathcal{V}^*$.

Define

$$\Delta(\mathcal{V}) := \left\{ q \in \mathbb{R}^{\mathcal V}_{\ge 0} : \sum_{a \in \mathcal{V}} q_a = 1 \right\}.$$

This is called the probability simplex. Its elements can be thought of as probability distributions: an element $q$ is the function

$$q(\cdot): \mathcal{V} \rightarrow [0,1] \qquad a \mapsto q_a,$$

such that $\sum_{a \in \mathcal{V}} q(a) = 1$. Since $|\mathcal V| = V$, a topologist would notice this is homeomorphic to $\Delta^{V-1}$.

Let $\Theta$ be a subset of $\mathbb{R}^d$, which we call the parameter space. Then a parametrized autoregressive language model on $\mathcal{V}$, or a language model (LM) for short, is a function

$$P: \Theta \times \mathcal{V}^* \rightarrow \Delta(\mathcal{V}), \quad (\theta, c) \mapsto p_\theta( \cdot | c) := P(\theta,c)$$

that is differentiable in $\theta$ for any fixed $c$. The notation $p_\theta( \cdot | c)$ foreshadows the roles of $\theta$ and $c$, which we will explain shortly. Here, $\theta$ is called the parameter and $c$ is called the context. Fixing $\theta \in \Theta$, we obtain a function $P(\theta, \cdot): \mathcal{V}^* \rightarrow \Delta(\mathcal{V})$ which is called a trained language model.

A trained language model is a function $P(\theta, \cdot)$ that takes in a string of tokens, and returns a probability distribution on the next token: informally, we call it a token predictor. Above, we wrote

$$P(\theta, c) = p_\theta( \cdot | c).$$

This is because the probability distribution on next tokens can be viewed as a conditional probability, conditional on the input string $c$, of the possible next tokens. For each $T \geq 1$, the chain rule therefore gives a probability distribution $q_\theta^{(T)}$ on $\mathcal{V}^T$, the space of sequences of length $T$, defined by

$$q_\theta^{(T)}(x_1, \ldots, x_T) := \prod_{i=1}^T p_\theta(x_i \mid x_1, \ldots, x_{i-1}).$$

When the length is clear from the argument, we will often write simply $q_\theta(x_1,\ldots,x_T)$. Thus the next-token distributions $p_\theta(\cdot \mid c)$ determine the model’s probability for every fixed-length token string.

---

1. Corpus and tokenization

A corpus is a body of text used to build a language model: the entire internet, more or less, in the case of large language models like ChatGPT; the complete works of Shakespeare in our toy models.

Typically when building a model, one gathers a corpus and divides it into three sets of data to work with:

1. A training set, which is usually most of the corpus. This is used to set the parameters initially.
2. A dev set, used during model development to compare choices and tune the model without looking at the test set. We will return to this role later in the course.
3. A test set, held out from training and model development and used to evaluate, or benchmark, the final model. We will discuss this evaluation later in this lecture, when we define perplexity.

The corpus arrives as raw text, so the first task is to turn it into tokens. Tokens might be letters or words or phonemes or letter combinations, etc. But on a computer, you probably want things to be encoded differently, for example, you want your information written in ASCII strings.

Tokenization is the process of changing text into tokens. Let $\mathcal{S}$ be a set of raw texts or strings (the stuff you type in). A tokenizer is an injection (“encoding”)

$$\operatorname{enc} : \mathcal S \rightarrow \mathcal{V}^*$$

with a left inverse $\operatorname{dec}$ (“decoding”), i.e. $\operatorname{dec} \circ \operatorname{enc} (s) = s$. This left-inverse property is referred to as losslessness.

It makes sense to choose your vocabulary $\mathcal{V}$ based on the corpus, and there are algorithms to determine the best tokens to use for a given corpus. That is, the text itself can naturally dictate $\mathcal{V}$ by the nature of its statistics. But for the moment, we’ll just do the dumbest thing in our Shakespeare example.

As an aside, for the transformer models studied later in the course, the cost of standard self-attention grows quadratically with sequence length. A tokenizer that represents text using fewer tokens is therefore computationally valuable; character-by-character encoding is very uncompressed.

---

2. $n$-gram models

The simplest language models are obtained by requiring that the distribution of the next token depend only on a fixed number of preceding tokens. These are called $n$-gram models and defined as follows:

In other words, the model output depends only on the previous $n-1$ tokens. So we need only consider contexts from $\mathcal V^{n-1} \subseteq \mathcal{V}^*$. For a short context of length $k < n-1$, we simply prepend $n-1-k$ $\texttt{*}$ tokens (the special token that says “I haven’t started talking yet, stand by!”).

Warning: we are really hoping to model a corpus which does not contain $\texttt{*}$. This definition is appropriate if we assume that our corpus and our language do not themselves contain the token $\texttt{*}$. This extra token is merely a practical formality.

The model $P$ is parametrized by a matrix $\theta \in \Theta \subseteq M_{V^{n-1} \times V}(\mathbb{R})$ with entries

$$\theta_{c,b} = p_\theta(b \mid c) = P(\theta, c)(b)$$

indexed by contexts $c \in \mathcal V^{n-1}$ and next tokens $b \in \mathcal{V}$, subject to the condition that $\theta_{c,\cdot} \in \Delta(\mathcal V)$ for every $c$, i.e. each $c$-th row is a probability distribution, so the entries are non-negative with row sum $1$. A matrix with this property is called a row-stochastic matrix.

The case $n=1$ is a unigram model. Here the context is empty ($\mathcal{V}^{n-1} = \mathcal{V}^0$ contains only the empty sequence), so the model ignores context entirely: it is a single fixed probability distribution on the next token. Its parameter matrix $\theta$ is a $1 \times V$ row-stochastic matrix, i.e. a single row, which is just an element of $\Delta(\mathcal{V})$.

The case $n=2$ is a bigram model; its parameter matrix $\theta$ is a $V\times V$ row-stochastic matrix. A bigram model defines a Markov chain: a stochastic process in which the probabilities of the next state depend only on the current state. In this case, the probability distribution of the next token depends only on the previous one, and not on any earlier history. The only contexts needed to define the model are single tokens, i.e. $\mathcal{V}^{n-1} = \mathcal{V}$.

---

3. Maximum likelihood

Training is the process of incrementally adjusting $\theta$ to achieve some “best” value. Maximum likelihood is the mathematical definition of “best”. (This is true during the training phase. There are later phases where we do other types of adjustments to the parameters.)

When we encode the training set, it becomes a sequence of tokens, which we will also call the training set.

Fix a training set $(x_1, \ldots, x_T) \in \mathcal{V}^*$. The likelihood of $\theta \in \Theta$ for this training set is

$$L(\theta)=L(\theta : x_1, \ldots, x_T) := q_\theta^{(T)}( x_1, \ldots, x_T).$$

We are thinking of the training set as fixed, so that this is a function on $\theta \in \Theta$ with values in $[0,1]$:

$$L( \cdot ) : \Theta \rightarrow [0,1].$$

The log likelihood is the function

$$\log L \colon \Theta \to [-\infty,0].$$

Don’t worry about $\log 0$ for right now.

A maximum likelihood estimator is any maximizer $\widehat{\theta}$ of the log likelihood as a function of $\theta$. The log is just more convenient to work with. That is, $\widehat{\theta}$ is an element of $\Theta$ at which a maximum occurs.

For any token string $x=(x_1,\ldots,x_T)$, define the model’s average negative log-likelihood, or average log loss, on $x$ by

$$\mathcal L(\theta : x_1,\ldots,x_T) :=-\frac1T\log L(\theta : x_1,\ldots,x_T) =-\frac1T\sum_{t=1}^T\log p_\theta(x_t\mid x_1,\ldots,x_{t-1}).$$

For the fixed training set, we abbreviate

$$\mathcal L(\theta):=\mathcal L(\theta : x_1,\ldots,x_T),$$

so that

$$\mathcal L(\theta) =-\frac1T\log L(\theta).$$

Thus maximizing likelihood is equivalent to minimizing average log loss. The latter is often more convenient because the product of next-token probabilities becomes an average of their negative logarithms. If $\log$ means logarithm to base $2$, the units are bits per token; natural logarithms give nats per token.

To fix ideas, let’s determine the maximum likelihood estimator for the bigram model. This means finding a row-stochastic matrix $\widehat{\theta}$ that maximizes the log likelihood $\log L$, or, equivalently, finding its entries $\widehat{\theta}_{a,b}$ for all $a,b \in \mathcal{V}$.

Let $\texttt{*}\in\mathcal V$ denote the beginning of sequence token, and set $x_0=\texttt{*}$. Recall that the training set $(x_1,\ldots,x_T)$ does not contain the token $\texttt{*}$. For $a,b\in\mathcal V$, let $N_{a,b}$ count the occurrences of the sequence $ab$ in the training set, including the initial transition $(x_0,x_1)=(\texttt{*},x_1)$, i.e.

$$N_{a,b}=\#\{i\in\{1,\ldots,T\}:(x_{i-1},x_i)=(a,b)\}, \qquad N_a=\sum_{b\in\mathcal V}N_{a,b}.$$

Then for $a\neq\texttt{*}$, $N_a$ counts the occurrences of $a$ in the training set (except in the last token), while

$$N_{\texttt{*},x_1}=1, \qquad N_{\texttt{*},b}=0 \quad\text{for }b\neq x_1, \qquad N_{\texttt{*}}=1.$$

Recall that for a bigram model, a context is just a single token.

In other words, when $N_a=0$, the $a$-th row may be chosen arbitrarily in $\Delta(\mathcal V)$. Note that $N_{\texttt{*}}=1$, so the theorem also determines the $\texttt{*}$-th row: $\widehat\theta_{\texttt{*},x_1}=1$, i.e. the model is certain to begin with the same token the training set begins with.

Proof

We want to maximize

$$\log L(\theta:x_1,\ldots,x_T).$$

For a bigram model, the conditional distribution of $x_i$ depends only on the preceding token. With the convention $x_0=\texttt{*}$, this gives

$$p_\theta(x_i\mid x_1,\ldots,x_{i-1}) = p_\theta(x_i\mid x_{i-1}) = \theta_{x_{i-1},x_i}$$

for every $1\leq i\leq T$. Therefore

$$\begin{aligned} L(\theta:x_1,\ldots,x_T) &= q_\theta^{(T)}(x_1,\ldots,x_T)\\ &= \prod_{i=1}^T p_\theta(x_i\mid x_1,\ldots,x_{i-1})\\ &= \prod_{i=1}^T \theta_{x_{i-1},x_i}\\ &= \prod_{a,b\in\mathcal V} \theta_{a,b}^{N_{a,b}}. \end{aligned}$$

Taking logarithms, we obtain

$$\log L(\theta:x_1,\ldots,x_T) = \sum_{a,b\in\mathcal V} N_{a,b}\log\theta_{a,b}.$$

If $N_a=0$, then $N_{a,b}=0$ for every $b\in\mathcal V$, so the $a$-th row contributes nothing to the log-likelihood. Hence that row may be chosen arbitrarily.

Now suppose that $N_a>0$ and define

$$\widehat\theta_{a,b} = \frac{N_{a,b}}{N_a}.$$

We claim that this is the unique choice of the $a$-th row which maximizes its contribution to the log-likelihood. If $\theta_{a,b}=0$ for some $b$ with $N_{a,b}>0$, then the log-likelihood is $-\infty$, so we may assume that

$$\theta_{a,b}>0$$

whenever $N_{a,b}>0$.

Using the inequality

$$\log x\leq x-1,$$

we have

$$\begin{aligned} &\sum_{b\in\mathcal V} N_{a,b}\log\theta_{a,b} - \sum_{b\in\mathcal V} N_{a,b}\log\widehat\theta_{a,b}\\ &\qquad= \sum_{\substack{b\in\mathcal V\\N_{a,b}>0}} N_{a,b} \log\left( \frac{\theta_{a,b}}{\widehat\theta_{a,b}} \right)\\ &\qquad\leq \sum_{\substack{b\in\mathcal V\\N_{a,b}>0}} N_{a,b} \left( \frac{\theta_{a,b}}{\widehat\theta_{a,b}}-1 \right)\\ &\qquad= \sum_{\substack{b\in\mathcal V\\N_{a,b}>0}} \left( N_a\theta_{a,b}-N_{a,b} \right)\\ &\qquad= N_a \sum_{\substack{b\in\mathcal V\\N_{a,b}>0}} \theta_{a,b} - N_a\\ &\qquad\leq 0. \end{aligned}$$

Here we used

$$\sum_{b\in\mathcal V}\theta_{a,b}=1 \qquad\text{and}\qquad \sum_{b\in\mathcal V}N_{a,b}=N_a.$$

It follows that

$$\log L(\theta:x_1,\ldots,x_T) \leq \log L(\widehat\theta:x_1,\ldots,x_T).$$

Moreover, equality in $\log x\leq x-1$ occurs only when $x=1$. Hence equality forces

$$\theta_{a,b} = \widehat\theta_{a,b} = \frac{N_{a,b}}{N_a}$$

whenever $N_{a,b}>0$. Summing over those values of $b$, we obtain

$$\sum_{\substack{b\in\mathcal V\\N_{a,b}>0}} \theta_{a,b} = \sum_{\substack{b\in\mathcal V\\N_{a,b}>0}} \frac{N_{a,b}}{N_a} = \frac{1}{N_a} \sum_{b\in\mathcal V}N_{a,b} = 1.$$

On the other hand, the $a$-th row of $\theta$ is a probability distribution, so

$$\sum_{b\in\mathcal V}\theta_{a,b}=1$$

and every $\theta_{a,b}\geq 0$. Therefore

$$\sum_{\substack{b\in\mathcal V\\N_{a,b}=0}} \theta_{a,b}=0.$$

Since each term in this sum is nonnegative, it follows that

$$\theta_{a,b}=0$$

whenever $N_{a,b}=0$. Thus, for every $a$ with $N_a>0$, the unique maximizing row is

$$\widehat\theta_{a,b} = \frac{N_{a,b}}{N_a}.$$

This proves the result. $\square$

Remark 3.2 (parameter sharing). The proof separates the likelihood question into $V$ independent multinomial estimation problems, one for each context. Neural models deliberately destroy this separation by sharing parameters across contexts; that sharing permits evidence from one context to affect predictions in another.

We will see later that zero entries in $\widehat{\theta}$ are problematic. This motivates the following adjustment to $\widehat{\theta}$.

Definition 3.3 (Laplace smoothing). For $\alpha>0$, the Laplace-smoothed estimator is

$$\widehat\theta^{(\alpha)}_{a,b} =\frac{N_{a,b}+\alpha}{N_a+\alpha V}.$$

In a Laplace-smoothed estimator, every coordinate is strictly positive.

---

4. Perplexity

We need a way to compare trained models (compare different values of $\theta$). To do this, we will define perplexity, which is a sort of measure of how surprising the trained model finds a string of text.

When we tokenize the test set, we obtain a sequence of tokens $(x_1, \ldots, x_N)$, also called the test set. Then the perplexity of the model on the test set is

$$PPL_\theta(x_1, \ldots, x_N) := q_\theta^{(N)}( x_1, \ldots, x_N)^{-1/N} = \prod_{i=1}^N p_\theta( x_i \mid x_1, \ldots, x_{i-1})^{-1/N}.$$

For an $n$-gram model, the probabilities of the first $n-1$ tokens are computed using the $\texttt{*}$-padding convention of Definition 2.1. Thus the same perplexity formula applies to every token in the test string; no initial factors need to be dropped.

The uniform unigram model is the unigram model which applies the same probability to every token in the vocabulary.

Proof

Every next-token probability is $1/V$, so $q_\theta^{(N)}(x_1, \ldots, x_N) = V^{-N}$ and $PPL = (V^{-N})^{-1/N} = V$. $\square$

---

5. Entropy

Perplexity is closely related to entropy and cross-entropy, two fundamental notions from information theory that will come up repeatedly.

We begin with entropy. Let $p$ be a probability distribution on a finite set $\mathcal X$.

The quantity

$$-\log p(x)$$

is sometimes called the information content or surprisal of $x$. An unlikely outcome has large surprisal, while a likely outcome has small surprisal. Thus entropy is the average surprisal of an outcome drawn according to $p$:

$$H(p) = \sum_{x\in\mathcal X}p(x)\bigl(-\log p(x)\bigr).$$

If the logarithm is base $2$, entropy is measured in bits; if we use the natural logarithm, it is measured in nats.

For language, it is useful to apply this idea not to a single token, but to long strings. Fix a trained language model. For every $n$, recall that the chain rule gives a probability distribution on $\mathcal V^n$:

$$q_\theta^{(n)}(x_1,\ldots,x_n) := \prod_{t=1}^n p_\theta(x_t\mid x_1,\ldots,x_{t-1}).$$

This is the fixed-length distribution introduced in Section 0; here we keep the superscript $n$ explicit because we will compare distributions for different string lengths.

Its entropy is

$$H(q_\theta^{(n)}) = -\sum_{(x_1,\ldots,x_n)\in\mathcal V^n} q_\theta^{(n)}(x_1,\ldots,x_n) \log q_\theta^{(n)}(x_1,\ldots,x_n).$$

This measures the average information contained in a string of length $n$. To obtain an average per token, we divide by $n$.

Thus $H(q_\theta^{(n)})$ measures the average information in an entire length-$n$ string, while $h(q_\theta)$ measures the asymptotic average information per token.

The entropy-rate discussion concerns an idealized process that continues indefinitely. It is therefore separate from the optional $\dashv$-stopping convention used to model finite responses in the appendix.

There is another way to understand the entropy rate which is particularly useful for language models. Suppose that the token sequence

$$X_1,X_2,X_3,\ldots$$

is a stationary ergodic stochastic process with joint distributions $q_\theta^{(n)}$.

Now sample one infinite sequence

$$s=(x_1,x_2,x_3,\ldots)$$

from the process. The quantity

$$-\log q_\theta^{(n)}(x_1,\ldots,x_n)$$

measures how surprising its first $n$ tokens are under the model, and hence

$$-\frac{1}{n}\log q_\theta^{(n)}(x_1,\ldots,x_n)$$

is the average surprisal per token along this particular string.

The Shannon—McMillan—Breiman theorem says that, under the stationary and ergodic hypotheses, almost every sufficiently long sample is representative in exactly this sense:

There is an immediate connection with perplexity. For any token sequence $(x_1,\ldots,x_N)$, Section 4 defined

$$PPL_\theta(x_1,\ldots,x_N) = q_\theta^{(N)}(x_1,\ldots,x_N)^{-1/N}.$$

Taking logarithms gives

$$\log PPL_\theta(x_1,\ldots,x_N) = -\frac{1}{N} \log q_\theta^{(N)}(x_1,\ldots,x_N).$$

Thus the logarithm of perplexity is exactly the average surprisal per token. In particular, under the hypotheses of the Shannon—McMillan—Breiman theorem, the log perplexity of a typical long sample converges to the entropy rate:

$$\log PPL_\theta(X_1,\ldots,X_N) \longrightarrow h(q_\theta).$$

We have now also seen two quantities used to evaluate a model: during training we maximize likelihood, while during testing we hope for low perplexity. These goals are not at odds. When likelihood and perplexity are computed on the same sequence of tokens, they are exactly the same measurement in different forms.

Proof

By definition,

$$PPL_\theta(x_1,\ldots,x_N) = q_\theta^{(N)}(x_1,\ldots,x_N)^{-1/N}.$$

Therefore

$$\begin{aligned} \log PPL_\theta(x_1,\ldots,x_N) &= -\frac{1}{N} \log q_\theta^{(N)}(x_1,\ldots,x_N)\\ &= -\frac{1}{N} \sum_{t=1}^N \log p_\theta(x_t\mid x_1,\ldots,x_{t-1})\\ &= \mathcal L(\theta:x_1,\ldots,x_N). \end{aligned}$$

The middle equality is the autoregressive factorization of the string probability.

Now fix the training set and regard these quantities as functions of $\theta$. Maximizing its likelihood

$$L(\theta:x_1,\ldots,x_N) = q_\theta^{(N)}(x_1,\ldots,x_N)$$

is equivalent to minimizing

$$-\frac{1}{N} \log q_\theta^{(N)}(x_1,\ldots,x_N) = \log PPL_\theta(x_1,\ldots,x_N).$$

Since the logarithm is strictly increasing, this is in turn equivalent to minimizing $PPL_\theta(x_1,\ldots,x_N)$. $\square$

The theorem concerns likelihood and perplexity evaluated on the same data. In particular, maximizing likelihood on the training set necessarily minimizes perplexity on that training set. But this does not tell us that the model will have low perplexity on new text. What we really want is for the probabilities learned from the training set to perform well on new samples from the underlying language. To formulate this distinction mathematically, we need to compare two probability distributions: the distribution generating the data and the distribution predicted by the model. This leads to the notion of cross-entropy, which we see in the next section.

---

6. Cross-entropy

When we evaluate a language model on new text, there are really two probability distributions in the background. At a context $c \in \mathcal V^*$, imagine that the language itself has some true, but unknown, next-token distribution

$$p_*(\,\cdot\mid c)\in\Delta(\mathcal V),$$

while our trained model predicts

$$p_\theta(\,\cdot\mid c)\in\Delta(\mathcal V).$$

The next token is generated by the language, but we want to evaluate how well our model predicted it.

Suppose, for example, that the next token $X$ is drawn according to $p_*(\,\cdot\mid c)$. If $X=x$, the model incurs the log loss

$$-\log p_\theta(x\mid c).$$

Averaging this loss over the possible tokens that the language might actually produce gives

$$-\sum_{x\in\mathcal V} p_*(x\mid c)\log p_\theta(x\mid c).$$

This is the next-token version of cross-entropy; after defining the general notion, we will return to whole strings.

Indeed, we will prove in the next section:

Let’s return to language models and long strings. The true next-token distributions $p_*(\cdot\mid c)$, when combined by the same chain rule as the model, determine joint distributions $q_*^{(n)}$ on length-$n$ strings. Thus $q_*^{(n)}$ is the true, unknown distribution of length-$n$ text, while $q_\theta^{(n)}$ is the distribution assigned by the model. The cross-entropy

$$H\bigl(q_*^{(n)},q_\theta^{(n)}\bigr) = -\sum_{(x_1,\ldots,x_n)\in\mathcal V^n} q_*^{(n)}(x_1,\ldots, x_n) \log q_\theta^{(n)}(x_1,\ldots, x_{n})$$

is the expected log loss of the model on a length-$n$ string generated by the true language. Dividing by $n$ gives the expected log loss per token.

When the limit exists, we call

$$h(q_*,q_\theta) := \lim_{n\to\infty} \frac{1}{n} H\bigl(q_*^{(n)},q_\theta^{(n)}\bigr)$$

the cross-entropy rate of the model relative to the source.

The cross-entropy rate is the quantity we would ideally like to know, but it involves the true language distribution $q_*$, which is unknown. To see how we can nevertheless estimate it from data, rewrite the finite-length cross-entropy per token as

$$\frac{1}{N}H\bigl(q_*^{(N)},q_\theta^{(N)}\bigr) = \sum_{(x_1,\ldots,x_N)\in\mathcal V^N} q_*^{(N)}(x_1,\ldots,x_N) \left( -\frac{1}{N} \log q_\theta^{(N)}(x_1,\ldots,x_N) \right).$$

This formula separates the roles of the true language and the model very clearly. The true distribution $q_*^{(N)}$ tells us how likely each length-$N$ string is to occur, and therefore supplies the weights in the average. For each possible string, the quantity

$$-\frac{1}{N} \log q_\theta^{(N)}(x_1,\ldots,x_N)$$

measures how surprising that string is according to the model, per token.

In practice, we do not know the weights $q_*^{(N)}$, so we cannot perform the sum over all possible strings. What we do have is a held-out string

$$(x_1,\ldots,x_N)$$

of actual language that was not used to train the model. For this observed string we can compute the quantity appearing inside the average:

$$\begin{aligned} -\frac{1}{N}\log q_\theta^{(N)}(x_1,\ldots,x_N) &= -\frac{1}{N} \sum_{t=1}^N \log p_\theta(x_t\mid x_1,\ldots,x_{t-1})\\ &=\mathcal L(\theta:x_1,\ldots,x_N). \end{aligned}$$

This is exactly the average log loss introduced in Section 3, now evaluated on held-out text rather than on the training set.

The connection with our discussion of entropy can be made precise. Recall that the Shannon—McMillan—Breiman theorem says that if $q_*$ is a stationary ergodic source and

$$(X_1,X_2,\ldots)$$

is sampled from $q_*$, then, with probability $1$,

$$-\frac{1}{N}\log q_*^{(N)}(X_1,\ldots,X_N) \longrightarrow h(q_*).$$

Thus the entropy rate, which is defined by averaging over all possible strings, can be recovered from the probability of one typical long string.

For cross-entropy, the same principle holds with the string scored by a second distribution $q_\theta$. Under suitable stationary and ergodic hypotheses on the source and model, a relative form of the Shannon—McMillan—Breiman theorem gives

$$-\frac{1}{N}\log q_\theta^{(N)}(X_1,\ldots,X_N) \longrightarrow h(q_*,q_\theta) \qquad\text{with probability 1,}$$

where the sequence $X_1,X_2,\ldots$ is generated according to the true source $q_*$.

The distinction between the two formulas is worth emphasizing. In the entropy formula, the source $q_*$ both generates and scores the string. In the cross-entropy formula, the source $q_*$ generates the string, but the model $q_\theta$ scores it. Consequently, for one sufficiently long held-out sample,

$$-\frac{1}{N}\log q_\theta^{(N)}(X_1,\ldots,X_N)$$

approximates the cross-entropy rate of the model relative to the source. This is why the model’s average log loss on held-out text is the practical quantity used to estimate cross-entropy: it can be computed without knowing $q_*$.

But we have already encountered exactly the quantity

$$-\frac{1}{N}\log q_\theta^{(N)}(x_1,\ldots,x_N).$$

By the definition of perplexity,

$$PPL_\theta(x_1,\ldots,x_N) = q_\theta^{(N)}(x_1,\ldots,x_N)^{-1/N},$$

and therefore

$$\log PPL_\theta(x_1,\ldots,x_N) = -\frac{1}{N} \log q_\theta^{(N)}(x_1,\ldots,x_N).$$

So perplexity on held-out text is simply an exponential version of the quantity used to estimate cross-entropy. The relationship can be summarized as follows:

• Cross-entropy averages the model’s log loss over all possible text generated by the true language.
• Held-out log loss computes that same loss on one observed sample of real text.
• Perplexity is the exponential of the held-out log loss.

Thus low perplexity on new text is evidence that the model has low cross-entropy relative to the language generating that text.

---

7. KL divergence and Gibbs’ inequality

In the preceding section, we compared two distributions on strings of length $n$: $q_*^{(n)}$, the true distribution of the language, and $q_\theta^{(n)}$, the distribution predicted by the model. Their cross-entropy

$$H\bigl(q_*^{(n)},q_\theta^{(n)}\bigr)$$

measures the average log loss when strings are generated according to the true language but scored using the model.

If we knew the true distribution and used it to score the same strings, the corresponding quantity would instead be the entropy

$$H\bigl(q_*^{(n)}\bigr).$$

Thus the difference

$$H\bigl(q_*^{(n)},q_\theta^{(n)}\bigr) - H\bigl(q_*^{(n)}\bigr)$$

measures the extra average log loss caused by using the model distribution $q_\theta^{(n)}$ instead of the true distribution $q_*^{(n)}$. This difference is an instance of the Kullback—Leibler divergence.

We first define KL divergence for arbitrary probability distributions.

Indeed,

$$\begin{aligned} H(p,q)-H(p) &= -\sum_{x\in\mathcal X}p(x)\log q(x) + \sum_{x\in\mathcal X}p(x)\log p(x)\\ &= \sum_{\substack{x\in\mathcal X\\p(x)>0}} p(x)\log\frac{p(x)}{q(x)}. \end{aligned}$$

As usual, terms with $p(x)=0$ contribute $0$.

The interpretation above suggests that KL divergence should never be negative: using the true probabilities should be at least as good, on average, as using some other probabilities. This is Gibbs’ inequality. The proof is elementary so we leave it out.

Combining Gibbs’ inequality with

$$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q)$$

immediately gives

$$H(p,q)\geq H(p),$$

with equality if and only if $p=q$. This proves the cross-entropy lower bound from the preceding section.

For the language model distributions, we obtain for every $n$

$$H\bigl(q_*^{(n)},q_\theta^{(n)}\bigr) = H\bigl(q_*^{(n)}\bigr) + D_{\mathrm{KL}} \bigl(q_*^{(n)}\|q_\theta^{(n)}\bigr).$$

Thus the cross-entropy of the model on length-$n$ text decomposes into two parts:

$$\boxed{ \text{cross-entropy} = \text{source entropy} + \text{model mismatch}. }$$

The first term is uncertainty inherent in the source itself. The second term measures the additional loss caused by the model’s failure to reproduce the true distribution.

There is a corresponding statement for the rates introduced in the preceding section. When the relevant limits exist, define the KL divergence rate by

$$d_{\mathrm{KL}}(q_*\|q_\theta) := \lim_{n\to\infty} \frac{1}{n} D_{\mathrm{KL}} \bigl(q_*^{(n)}\|q_\theta^{(n)}\bigr).$$

Dividing the preceding identity by $n$ and taking limits gives

$$h(q_*,q_\theta) = h(q_*) + d_{\mathrm{KL}}(q_*\|q_\theta).$$

In particular,

$$h(q_*,q_\theta)\geq h(q_*).$$

Thus, at the level of long strings, the excess cross-entropy rate of the model over the entropy rate of the source is precisely its KL divergence rate from the true language.

Looking ahead to Lecture 2. There are two equivalent levels at which to view these quantities. In this section we have compared the distributions $q_*^{(n)}$ and $q_\theta^{(n)}$ on complete strings of length $n$. Recall, however, that these distributions are built from the next-token distributions by

$$q_\theta^{(n)}(x_1,\ldots,x_n) = \prod_{t=1}^n p_\theta(x_t\mid x_1,\ldots,x_{t-1}).$$

Consequently,

$$-\log q_\theta^{(n)}(x_1,\ldots,x_n) = \sum_{t=1}^n -\log p_\theta(x_t\mid x_1,\ldots,x_{t-1}).$$

Thus the loss on a string is the sum of the losses of its individual next-token predictions. In Lecture 2 we will return to this local viewpoint: the problem will be to construct and parametrize the map

$$c\longmapsto p_\theta(\,\cdot\mid c)\in\Delta(\mathcal V)$$

from contexts to next-token probability distributions.

---

8. Summary: measuring the quality of a language model

Let $q_*^{(n)}$ denote the true distribution of length-$n$ text, and let $q_\theta^{(n)}$ denote the distribution predicted by the language model.

The main quantities introduced above fit together as follows:

• For an observed string $(x_1,\ldots,x_N)$, the model’s average log loss is $\mathcal L(\theta:x_1,\ldots,x_N) = -\frac1N\log q_\theta^{(N)}(x_1,\ldots,x_N)$. Smaller is better.

• The perplexity of the model on this string is $PPL_\theta(x_1,\ldots,x_N) = q_\theta^{(N)}(x_1,\ldots,x_N)^{-1/N}$. Thus $\log PPL_\theta(x_1,\ldots,x_N) = \mathcal L(\theta:x_1,\ldots,x_N)$. Perplexity is therefore just a monotone rescaling of the same measurement.

• The cross-entropy rate $h(q_*,q_\theta)$ is the idealized average log loss of the model on text generated by the true language. Since $q_*$ is unknown, in practice we estimate this quantity using average log loss, or equivalently perplexity, on held-out text.

• The entropy rate $h(q_*)$ measures the irreducible uncertainty in the language itself.

• The difference between these two quantities is the KL divergence rate: $h(q_*,q_\theta) = h(q_*) + d_{\mathrm{KL}}(q_*\|q_\theta)$. Thus the KL term measures the additional predictive loss caused by the model’s failure to match the true language distribution.

In short, training and evaluation revolve around the same basic quantity: the probabilities that the model assigns to the tokens that actually occur. On a fixed held-out string, assigning larger probability is exactly the same as having lower average log loss and lower perplexity. On sufficiently representative held-out text, these quantities estimate the model’s cross-entropy rate on the underlying language.

---

9. A language model as a compression scheme

For this subsection, we take all logarithms in base $2$. We have just seen that the average log loss

$$\mathcal L(\theta:x_1,\ldots,x_N) = -\frac1N\log_2 q_\theta^{(N)}(x_1,\ldots,x_N)$$

measures how well the model assigns probabilities to an observed string. There is another useful interpretation of exactly the same quantity: it is closely related to the number of bits per token needed to compress the string using the probabilities supplied by the model.

Suppose that the sender and receiver both have access to the same language model. At the context

$$(x_1,\ldots,x_{t-1}),$$

the model assigns the next token $x_t$ probability

$$p_\theta(x_t\mid x_1,\ldots,x_{t-1}).$$

A probability $p_\theta(x_t\mid x_{<t})$ corresponds to an ideal code length of approximately

$$-\log_2 p_\theta(x_t\mid x_{<t})$$

bits. Thus likely continuations receive short descriptions, while unlikely continuations require longer descriptions.

The code depends on the context, but this causes no problem: once the receiver has decoded $x_1,\ldots,x_{t-1}$, they know the same context as the sender and therefore know the same next-token distribution.

For a string

$$(x_1,\ldots,x_N),$$

the total idealized code length is therefore approximately

$$\begin{aligned} \sum_{t=1}^N -\log_2 p_\theta(x_t\mid x_1,\ldots,x_{t-1}) &= -\log_2 \prod_{t=1}^N p_\theta(x_t\mid x_1,\ldots,x_{t-1})\\ &= -\log_2 q_\theta^{(N)}(x_1,\ldots,x_N). \end{aligned}$$

Dividing by $N$, the idealized bitrate is

$$-\frac1N \log_2 q_\theta^{(N)}(x_1,\ldots,x_N)$$

bits per token.

But this is exactly the average log loss of the model when logarithms are taken in base $2$. Equivalently,

$$\log_2 PPL_\theta(x_1,\ldots,x_N) = -\frac1N \log_2 q_\theta^{(N)}(x_1,\ldots,x_N).$$

Thus lower perplexity corresponds to a shorter probabilistic description of the text.

The code lengths

$$-\log_2 p_\theta(x_t\mid x_{<t})$$

need not be integers, so they should be understood as ideal code lengths rather than literal lengths of individual binary codewords. Coding methods such as arithmetic coding make this probabilistic picture precise up to small coding overheads.

The main conceptual point is that the same quantity has now appeared in several forms:

$$\boxed{ \text{average log loss} \quad=\quad \log_2(\text{perplexity}) \quad\approx\quad \text{bits per token}. }$$

For long text generated by the source, the corresponding population quantity is the cross-entropy rate, and the KL divergence rate measures the extra compression cost due to model mismatch.

Dividing a log loss in nats by $\ln 2$ converts it to bits, and multiplying by the ratio of tokens to characters gives bits per character, a tokenizer-independent unit. Measured on the course corpus:

model	nats/token	bits/character	perplexity
uniform	4.1744	6.0224	65.000
bigram (add-one)	2.4549	3.5417	11.646
trigram (add-one)	1.9532	2.8178	7.051

Historical aside. In Prediction and Entropy of Printed English (1951), Shannon estimated English at roughly one bit per character using human guessing experiments. His number concerns a language source’s long-run entropy rate; the course’s progression from the uniform baseline, through $n$-grams, to the transformer is most readily compared in bits per character.

---

Appendix. Normalization and the end-of-sequence token

In the main text, we explained informally how an end-of-sequence token turns next-token probabilities into probabilities of finite sentences. We give the details here.

Let $\mathcal V$ be a finite vocabulary containing a distinguished end-of-sequence token $\dashv$, and write

$$\mathcal V_0:=\mathcal V\setminus\{\dashv\}$$

for the set of ordinary tokens. Fix a trained language model

$$P(\theta,\cdot):\mathcal V^*\longrightarrow\Delta(\mathcal V), \qquad c\longmapsto p_\theta(\,\cdot\mid c).$$

Thus, for every context $c$,

$$\sum_{a\in\mathcal V_0}p_\theta(a\mid c) +p_\theta(\dashv\mid c)=1.$$

To sample from the model, we begin with the empty context. We draw a token according to $p_\theta(\,\cdot\mid\varnothing)$. If that token is $\dashv$, we stop. Otherwise we append it to the context and repeat. In this way the model may produce

$$x_1,x_2,\ldots,x_T,\dashv, \qquad x_i\in\mathcal V_0.$$

We call such a sequence a complete sentence. The corresponding sentence, with the stopping symbol suppressed, is the finite string $(x_1,\ldots,x_T)\in\mathcal V_0^*$.

The chain rule suggests assigning to this complete sentence the probability

$$q_\theta(x_1,\ldots,x_T,\dashv) := \left( \prod_{i=1}^T p_\theta(x_i\mid x_{<i}) \right) p_\theta(\dashv\mid x_1,\ldots,x_T).$$

For $T=0$, the empty product is $1$, so this formula says

$$q_\theta(\dashv)=p_\theta(\dashv\mid\varnothing).$$

There is a small normalization issue. Although each $p_\theta(\,\cdot\mid c)$ is a probability distribution, it does not automatically follow that the probabilities of all finite sentences sum to $1$. There could, in principle, be positive probability that the model continues producing ordinary tokens forever and never emits $\dashv$.

The following proposition makes this precise.

Proof

For $T\geq0$, let $H_T$ be the event that the model produces exactly $T$ ordinary tokens and then emits $\dashv$. For a fixed sequence $x_1,\ldots,x_T\in\mathcal V_0$, the probability of producing

$$x_1,\ldots,x_T,\dashv$$

is, by the chain rule,

$$q_\theta(x_1,\ldots,x_T,\dashv).$$

The possible strings $(x_1,\ldots,x_T)$ are mutually exclusive, so

$$\Pr(H_T) = \sum_{x_1,\ldots,x_T\in\mathcal V_0} q_\theta(x_1,\ldots,x_T,\dashv).$$

The events $H_0,H_1,H_2,\ldots$ are also mutually exclusive, and their union is exactly the event that the model eventually emits $\dashv$. Therefore

$$\begin{aligned} \Pr(\text{the model eventually emits }\dashv) &= \sum_{T=0}^{\infty}\Pr(H_T)\\ &= \sum_{T=0}^{\infty} \ \sum_{x_1,\ldots,x_T\in\mathcal V_0} q_\theta(x_1,\ldots,x_T,\dashv). \end{aligned}$$

The left-hand side is a probability, so it is at most $1$.

Finally, either the model eventually emits $\dashv$, or it continues forever without emitting $\dashv$. Thus

$$\Pr(\text{the model eventually emits }\dashv) = 1-\Pr(\text{the model never emits }\dashv).$$

The displayed sum is therefore equal to $1$ precisely when the latter probability is $0$. $\square$

A collection of nonnegative numbers whose total is at most $1$ is sometimes called a sub-probability distribution. Thus the language model always gives a sub-probability distribution on finite sentences. The “missing mass” has a concrete interpretation:

$$1- \sum_{T=0}^{\infty} \ \sum_{x_1,\ldots,x_T\in\mathcal V_0} q_\theta(x_1,\ldots,x_T,\dashv) = \Pr(\text{the model never emits }\dashv).$$

For example, if the probabilities of all finite sentences sum to $0.97$, the remaining $0.03$ is exactly the probability that the model generates ordinary tokens forever.

When the probability of continuing forever is $0$, we say that the sampling process halts almost surely. Here “almost surely” is standard probability terminology: it means “with probability $1$.” In that case, appending $\dashv$ gives a bijection

$$\mathcal V_0^* \longleftrightarrow \{\text{complete sentences}\}, \qquad (x_1,\ldots,x_T) \longleftrightarrow (x_1,\ldots,x_T,\dashv),$$

and $q_\theta$ therefore gives an ordinary probability distribution on finite sentences in $\mathcal V_0^*$.

There is a simple sufficient condition which guarantees this normalization. Suppose there is some $\varepsilon>0$ such that

$$p_\theta(\dashv\mid c)\geq\varepsilon$$

for every context $c$ that can arise before the model stops. At each step, the probability of producing another ordinary token is then at most $1-\varepsilon$. Consequently, the probability that the model produces at least $T$ ordinary tokens without stopping is at most

$$(1-\varepsilon)^T.$$

Since

$$(1-\varepsilon)^T\longrightarrow0 \qquad\text{as }T\longrightarrow\infty,$$

the probability that the model continues forever is $0$. Hence the model halts almost surely, and its probabilities on finite sentences sum to $1$.

---

Exercises (paired with Step 1)

1. ★ Extend Theorem 3.1 to a general $n$-gram model, implement the bigram estimator, and confirm an average log loss (cross-entropy) of 2.4549 nats for add-one smoothing.
2. Show that the Laplace-smoothed estimator (Definition 3.3) is the mean of the posterior obtained from a $\operatorname{Dirichlet}(\alpha,\ldots,\alpha)$ prior on the simplex and the observed counts. Determine the limits as $\alpha\to\infty$ and $\alpha\to0^+$.
3. ★ Compute perplexity for the uniform, bigram, and trigram models, and verify Proposition 4.2 numerically.
4. Sampling versus scoring. Construct a distribution on $\{\texttt{a},\texttt{b}\}^T$ with excellent training perplexity but poor samples.
5. Held-out zeros. Fit an unsmoothed bigram model on the first 90% of the corpus. Estimate the probability that a uniformly selected held-out position receives probability zero, and relate it to missing mass.
6. Context length and training loss. Prove that the training loss of the $k$th-order MLE is non-increasing in $k$, and explain why this does not establish better held-out performance.
7. Prove that $\operatorname{CE}(p,q)$ is convex in $q$ and that $D_{\mathrm{KL}}(p\|q)$ is jointly convex in $(p,q)$.

---

Pointers

Shannon, A Mathematical Theory of Communication (1948) and Prediction and Entropy of Printed English (1951); Cover and Thomas, Elements of Information Theory, Chapters 2 and 5. See the resources page and Project Step 1.