Lecture 7 — Tokenization, Sampling, and Scaling Laws

Project connection. Project Step 7 trains a byte-pair-encoding tokenizer, retrains the model on its tokens, implements temperature, top-$k$, and nucleus sampling, and pools model sweeps into a small-scale scaling-law plot. BPE measurements below use the course’s 1,115,394-character Shakespeare corpus.

Chapter overview. Tokenization determines the random variables modeled by the network; sampling determines how a modeled distribution is turned back into text; scaling laws describe empirical changes in loss as model, data, and compute grow. In each case, an engineering choice has a precise mathematical formulation.

0. Notation

A tokenizer has a finite token set, called its vocabulary, and maps raw text to a finite sequence of token identifiers. The token count of a corpus is the length of this sequence. Throughout Sections 1 and 2, $N_{\mathrm{ch}}$ denotes the number of raw characters and $N_{\mathrm{tok}}$ the number of tokens.

1. Byte-pair encoding

Suppose the current corpus representation is a token sequence $(u_1,\ldots,u_N)$. An adjacent pair occurrence of $(a,b)$ is an index $i$ with $(u_i,u_{i+1})=(a,b)$. Replacing occurrences is performed from left to right and uses non-overlapping pairs.

Definition 1.1 (byte-pair encoding). Byte-pair encoding (BPE) starts from a base character or byte vocabulary and the corresponding tokenized corpus. For each of $M$ rounds:

1. count every adjacent token pair;
2. choose a pair with maximal count;
3. introduce a new token representing its concatenation; and
4. replace its non-overlapping occurrences from left to right.

The ordered list of chosen pairs is the merge list. To encode new text, start from base tokens and apply the learned merges in training order. Decoding recursively expands every merged token into base tokens and then into text.

Proposition 1.2 (token-count reduction). If one merge replaces $r$ non-overlapping occurrences of a selected pair, then it reduces the corpus token count by exactly $r$. For a pair of distinct tokens, adjacent occurrences cannot overlap, so its adjacent-pair count equals $r$ and BPE’s choice maximizes the immediate token-count reduction among such pairs.

Proof. Each replacement turns two tokens into one and therefore removes one token. Non-overlap makes the $r$ reductions disjoint. If $a\neq b$, two occurrences of $(a,b)$ cannot share a token, because a shared middle token would have to equal both $a$ and $b$. $\square$

Remark 1.3 (self-overlap). For a pair $(a,a)$, the adjacent count can exceed the number of non-overlapping replacements: aaa has two adjacent occurrences of aa but only one left-to-right replacement. Thus BPE is exactly greedy for the adjacent-count surrogate and usually, but not literally in every self-overlapping case, for one-step token reduction. There is no global optimality theorem analogous to Huffman coding. A Huffman code is a prefix code minimizing expected codeword length for a fixed symbol distribution; BPE instead changes the segmentation of an entire corpus, so it solves a different combinatorial problem.

Remark 1.4 (merge order and losslessness). Later tokens are defined in terms of earlier tokens, so applying available merges in a different order can produce a different segmentation. Retaining the learned order makes encoding deterministic. Because every new token stores the pair from which it was formed, recursive expansion proves exact decoding and hence losslessness.

For this corpus, the first ten merges are

 0 'e '   1 'th'   2 't '   3 's '   4 'd '
 5 ', '   6 'ou'   7 'er'   8 'in'   9 'y '

followed by an, :\n, or, o , en, \n\n, and others.

merges $M$	vocabulary size	tokens	characters/token
64	129	743,756	1.500
256	321	568,210	1.963
512	577	487,961	2.286
1024	1089	414,322	2.692

Corpus aside 1.5. Frequent word-final pairs, the play-format tokens :\n and \n\n, and eventually whole words appear without linguistic annotation. The table also shows diminishing returns: increasing from 256 to 1,024 merges improves characters per token by about 37%, not by a factor of four. Tokens are frequency artifacts, so they may split digits, whitespace, or rare words in ways that do not match human categories.

2. Tokenization and the unit of loss

Suppose a model assigns average log loss (cross-entropy) $\mathcal L_{\mathrm{tok}}$ nats per token to an encoded corpus. By the coding interpretation of Lecture 1, its total ideal code length is $N_{\mathrm{tok}}\mathcal L_{\mathrm{tok}}$ nats.

Proposition 2.1 (bits-per-character conversion). The same corpus code has rate

$$\operatorname{bpc} =\frac{\mathcal L_{\mathrm{tok}}}{\ln2} \frac{N_{\mathrm{tok}}}{N_{\mathrm{ch}}}$$

bits per character.

Proof. Divide the total code length by $\ln2$ to convert nats to bits, then divide by the number of raw characters. Losslessness ensures that the token sequence and raw corpus describe the same object. $\square$

Corollary 2.2. If a tokenizer halves the token count, its nats-per-token loss may approximately double without worsening bits per character. Consequently, per-token perplexities are not comparable across different tokenizers.

Project remark 2.3. At $M=256$, the BPE corpus has 568,210 tokens and vocabulary size 321. Each token carries about 1.96 characters. With the same $T_{\max}$, its context window spans roughly twice as many characters, but each prediction is correspondingly more information-rich. Step 7 compares the character and BPE models only after applying Proposition 2.1.

3. Decoding from a predictive distribution

Let $z\in\mathbb R^V$ be the next-token logits and $p=\operatorname{softmax}(z)$. A decoding rule maps this conditional distribution to a selected or sampled next token.

For $\tau>0$, define the temperature-$\tau$ distribution by

$$p_\tau=\operatorname{softmax}(z/\tau).$$

For $a\in\Delta^{V-1}$, let $H(a)=-\sum_ba_b\log a_b$ be its entropy.

Proposition 3.1 (Gibbs variational characterization). For every $\tau>0$,

$$p_\tau =\operatorname*{arg\,max}_{a\in\Delta^{V-1}} \{\langle a,z\rangle+\tau H(a)\}.$$

Proof. Put $Z=\sum_be^{z_b/\tau}$. Since $\log(p_\tau)_b=z_b/\tau-\log Z$, every $a\in\Delta^{V-1}$ satisfies

$$\langle a,z\rangle+\tau H(a) =\tau\log Z-\tau D_{\mathrm{KL}}(a\|p_\tau).$$

Gibbs inequality makes the right side at most $\tau\log Z$, with equality exactly when $a=p_\tau$. Thus the maximizer exists and is unique. $\square$

Remark 3.2 (temperature limits). As $\tau\to\infty$, entropy dominates and $p_\tau$ tends to uniform. As $\tau\to0^+$, it concentrates uniformly on the indices attaining $\max_bz_b$; with a unique maximizer it tends to that one-hot distribution. Temperature is therefore an entropy—score tradeoff, exactly as in attention.

Two truncation rules remove part of a distribution’s tail before sampling.

Definition 3.3 (top-$k$ sampling). For an integer $1\leq k\leq V$, keep the $k$ indices having the largest logits, set all other logits to $-\infty$, apply softmax, and sample from the resulting distribution.

Definition 3.4 (nucleus sampling). Sort tokens in non-increasing order of probability. For $p_0\in(0,1]$, keep the smallest prefix whose cumulative probability is at least $p_0$, discard the remaining tokens, renormalize, and sample. This is also called top-$p$ sampling.

Nucleus sampling adapts the number of retained tokens to the concentration of the current conditional distribution.

Probability aside 3.5 (typical sets and entropy rate). Let $X_1,X_2,\ldots$ be independent and identically distributed (iid) with token law $q$, and put $Z_t=-\log q(X_t)$. Then $\mathbb E[Z_t]=H(q)$. The weak law of large numbers states that averages of iid finite-variance variables converge in probability to their common mean; explicitly, for every $\varepsilon>0$,

$$\mathbb P\left( \left|\frac1T\sum_{t=1}^TZ_t-H(q)\right|>\varepsilon \right)\longrightarrow0.$$

Since $q(X_1,\ldots,X_T)=\prod_tq(X_t)$, this is equivalent to

$$-\frac1T\log q(X_1,\ldots,X_T)\longrightarrow H(q)$$

in probability. Hence most mass lies on a typical set of roughly $e^{TH(q)}$ sequences, each with probability about $e^{-TH(q)}$.

For a dependent source, the long-run quantity replacing $H(q)$ is its entropy rate

$$H_{\mathrm{rate}} =\lim_{T\to\infty}\frac1T H(X_1,\ldots,X_T).$$

A source is stationary if its finite-dimensional laws are invariant under time shifts. For a stationary finite-alphabet source, the rate also equals

$$H_{\mathrm{rate}} =\lim_{T\to\infty}H(X_T\mid X_{<T});$$

these conditional entropies decrease as the available context grows. The source is ergodic if long-run sample averages converge to their corresponding expectations. The Shannon—McMillan—Breiman theorem gives stationary ergodic sources the analogous asymptotic equipartition property:

$$-\frac1T\log p(X_1,\ldots,X_T) \longrightarrow H_{\mathrm{rate}}$$

almost surely. For an iid source the rate is $H(q)$; for a stationary first-order Markov chain with transition matrix $\theta$ and stationary distribution $\pi$, it is $\sum_a\pi(a)H(\theta_{a,\cdot})$.

Entropy rate is a property of the source, not another model score. Lecture 1’s $\mathcal L_{\mathrm{best}}$ is the empirical one-step conditional estimate obtained from a fixed corpus. Its bigram value, 3.54 bits per character, lies well above Shannon’s roughly one-bit estimate because one character of context leaves much uncertainty. Increasing empirical context on a fixed corpus eventually drives the training floor toward zero through memorization, not toward the true source rate. The single highest-probability sequence likewise need not resemble a typical draw.

Decoding remark 3.6. Greedy decoding chooses an argmax at every step. Beam search retains a fixed number of prefixes having the largest accumulated log-probabilities and expands only those prefixes; it is a heuristic for finding a high-probability complete sequence. Language models often make these modes repetitive, while moderate sampling yields more typical text. Step 7 records this degeneration directly. The training objective remains likelihood; the decoding rule determines which region of the learned distribution is exposed.

4. Empirical scaling laws

A positive quantity $f(s)$ obeys a power law in scale $s$ if $f(s)\approx cs^{-\alpha}$ for constants $c,\alpha>0$. Taking logarithms gives

$$\log f(s)\approx\log c-\alpha\log s,$$

so a power law appears as a straight line on log—log axes.

Let $N$ denote non-embedding parameter count, $D$ the number of training tokens, and $C$ the training compute.

Observation 4.1 (Kaplan scaling laws). Across substantial empirical ranges in which other resources are not limiting, language-model loss was fit by relations of the form

$$L(N)\approx\left(\frac{N_c}{N}\right)^{\alpha_N}, \qquad L(D)\approx\left(\frac{D_c}{D}\right)^{\alpha_D}, \qquad L(C)\approx\left(\frac{C_c}{C}\right)^{\alpha_C}.$$

Here $N_c,D_c,C_c>0$ are fitted scale constants and $\alpha_N,\alpha_D,\alpha_C>0$ are fitted exponents. The fitted exponents are small; Kaplan et al. reported, for example, $\alpha_N\approx0.076$ in one regime.

Remark 4.1 (status). These are empirical fits rather than universal theorems. The straightness over many orders of magnitude is the notable phenomenon. Small exponents mean that a fixed multiplicative loss reduction requires a large multiplicative increase in scale.

To allocate a fixed compute budget, consider a joint loss model. Let $E,A,B,\alpha,\beta$ be positive constants, with $E$ representing an irreducible loss floor.

Proposition 4.2 (compute-optimal Chinchilla allocation). Suppose

$$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}$$

and the dominant training-compute constraint is $C=6ND$. Then the unique positive minimizer satisfies

$$N^*\propto C^{\beta/(\alpha+\beta)}, \qquad D^*\propto C^{\alpha/(\alpha+\beta)}.$$

More precisely,

$$(N^*)^{\alpha+\beta} =\frac{\alpha A}{\beta B}\left(\frac C6\right)^\beta.$$

Proof. Substitute $D=C/(6N)$ and omit the constant $E$:

$$f(N)=AN^{-\alpha}+B\left(\frac{6N}{C}\right)^\beta.$$

The stationary equation is

$$-\alpha AN^{-\alpha-1} +\beta B6^\beta C^{-\beta}N^{\beta-1}=0,$$

which rearranges to the displayed formula. The first term dominates as $N\to0^+$ and the second as $N\to\infty$, so this stationary point is the unique global minimizer. Substituting it into $D=C/(6N)$ gives the second scaling relation. $\square$

When $\alpha\approx\beta$, both $N^*$ and $D^*$ grow approximately as $C^{1/2}$.

Historical aside 4.3. Hoffmann et al. found exponents close enough to equal that compute-optimal model size and data should grow together. GPT-3 used about 175 billion parameters and 300 billion tokens; Chinchilla used about 70 billion parameters and 1.4 trillion tokens and performed better at a comparable training budget. Their fitted constants led to the approximate rule of twenty training tokens per parameter and motivated training smaller models on substantially more data than earlier practice.

Caveat 4.4. The constants and exponents depend on architecture, data, and optimizer. The law must bend near the irreducible floor and when data are repeatedly reused. The objective also omits inference cost; a model serving many queries may rationally be smaller and trained for longer than the training-only optimum.

5. A course-scale scaling law

Step 7 plots the best validation loss against parameter count for controlled Step 6 sweeps, for example $d\in\{32,64,128,256\}$ at fixed depth. A line is fit only through the non-saturated points on log—log axes.

Project remark 5.1. The fitted exponent need not resemble the Kaplan value: the dataset, architecture, and scale differ. The useful questions are whether a straight segment is visible and where it bends. Very small models are capacity-limited; very large models exhaust the 1.1 MB corpus and widen the training—validation gap. This is a laptop-scale instance of the model—data allocation problem in Proposition 4.2.

Summary

BPE is a deterministic ordered merge scheme that greedily compresses token count, subject to a small self-overlap qualification. Tokenizer changes alter the unit of per-token loss, so bits per character is the appropriate cross-tokenizer metric. Temperature sampling is an entropy-regularized soft argmax; top-$k$ and nucleus rules truncate its tail. Typical-set and entropy-rate reasoning explains why a modal sequence may be less representative than a sample. Empirical loss curves often follow power laws, and the Chinchilla loss model yields a closed-form compute-optimal split between parameters and data.

Exercises (paired with Step 7)

A star marks a Project Step 7 task.

1. ★ Implement BPE training, encoding, and decoding. Verify exact round trip on unseen text and reproduce the compression table.
2. Give a short merge list and input for which applying learned merges out of training order changes the segmentation.
3. ★ Prove Proposition 2.1 and compare the character and BPE models in bits per character.
4. ★ Prove Proposition 3.1, including both temperature limits. Implement temperature, top-$k$, and nucleus sampling.
5. Find a greedy-decoding repetition loop and compare the probability scale of the greedy sequence with the typical $e^{-TH}$ scale.
6. ★ Derive Proposition 4.2 and fit a power-law exponent to the class sweep. Explain where the fitted line must bend at both ends.
7. Construct a corpus on which two-step lookahead produces a smaller final BPE token count than the greedy pair choice.
8. For $\gamma>0$, add an inference-cost term $\gamma NQ$ for $Q$ expected queries to the compute-allocation problem and determine qualitatively how the optimal $N^*$ changes.

Pointers

Sennrich, Haddow, and Birch (2016) for BPE in NLP; Holtzman et al., The Curious Case of Neural Text Degeneration (2020); Kaplan et al., Scaling Laws for Neural Language Models (2020); Hoffmann et al., Training Compute-Optimal Large Language Models (2022); Cover and Thomas, Chapter 3, for the asymptotic equipartition property. See the resources page and Project Step 7.