Lecture 2 — Neural Networks and Approximation

Project connection. Project Step 2 asks you to wire a one-hidden-layer ReLU network from explicit matrices. Four hidden units build the nonlinear feature $|x_1|+|x_2|$; two output logits and softmax turn it into a classifier with a diamond-shaped decision boundary.

Chapter overview. Lecture 1 described a language model as a map from contexts to probability distributions. We now replace an explicit matrix of probabilities by a parametrized function. We begin with binary logistic regression, because every calculation fits on one line, and then pass to softmax regression and neural networks. The final sections ask what such networks can represent, why depth can help, and what those representation theorems do—and do not—say about training a useful model.

0. The sigmoid function and logistic regression

We begin with a new type of model. We will connect this with language models (from the last lecture) later in this lecture.

A classification problem asks us to predict which of several categories an observation belongs to.

Let $\mathcal{C}$ be a finite set whose members are called classes or labels. In binary classification there are two possible classes, typically $\mathcal{C}=\{0,1\}$. Let $\Delta(\mathcal{C})$ be the probability simplex on $\mathcal{C}$, i.e. the set of probability distributions on $\mathcal{C}$.

The input space is a set $\mathcal{X}\subseteq\mathbb R^d$ whose elements are called feature vectors. Each of the $d$ standard coordinates of $\mathbb R^d$ is called a feature, so that a feature vector

$$\mathbf{x}=(x_1,\ldots,x_d)^\top\in\mathbb R^d$$

has a value $x_i$ for the $i$-th feature.

For a classification problem, we use the same notation as for a language model. Let $\Theta\subseteq\mathbb R^k$ be the parameter space. A probabilistic classifier for $\mathcal X$ and $\mathcal C$ is a function

$$P:\Theta\times\mathcal X\longrightarrow\Delta(\mathcal C), \qquad (\theta,\mathbf{x})\longmapsto p_\theta(\,\cdot\mid\mathbf{x}):=P(\theta,\mathbf{x}),$$

that is differentiable in $\theta$ for each fixed $\mathbf{x}$. Fixing $\theta$ gives a trained classifier $P(\theta,\cdot)$. We write $p_\theta(c\mid\mathbf{x})$ for the probability it assigns to class $c$ on input $\mathbf{x}$.

In the binary classification case, we can give the output probability distribution $p_\theta$ on $\{0,1\}$ as a single number $\widehat y\in[0,1]$:

$$p_\theta(y=1\mid\mathbf{x})=\widehat y, \qquad p_\theta(y=0\mid\mathbf{x})=1-\widehat y.$$

Both a classification model and a language model are instances of a more general conditional probabilistic model: something that maps an input to a probability distribution on possible outputs.

Analogously to the last lecture, we will discuss training and maximum likelihood in this new context. In the case of a training set, we need to train the model on feature vectors which are accompanied by the “correct” labels. For example, many pictures of cats labelled as cats and many other pictures labelled as non-cats.

For a classification model, the training set consists of pairs $(\mathbf{x},c)\in\mathcal{X}\times\mathcal{C}$ called training samples. In the context of the training set, the feature vector $\mathbf{x}$ is called the training observation and $c$ is the gold label. The value $c$ is the supposed “correct answer,” recorded along with the observation (perhaps $\mathbf{x}$ is derived from an image, and the label $c$ was assigned by a human being who knows how to recognise cats well). The term comes from “gold standard.”

It will be convenient to have a function that turns an arbitrary real number into a probability. The first useful elementary example is the sigmoid.

The two functions are inverses: $\operatorname{logit}(\sigma(z))=z$ and $\sigma(\operatorname{logit}(p))=p$. Indeed,

$$\frac{\sigma(z)}{1-\sigma(z)} = \frac{1/(1+e^{-z})}{e^{-z}/(1+e^{-z})} =e^z,$$

and taking logarithms gives the first identity. The second follows in the same way.

Now we introduce a simple binary classification model which we will use as a building block for a neural network.

Aside (why “regression”?). The adjective “regression” is historical. In statistics, regression refers to fitting a function to data. The name descended from Galton’s nineteenth-century studies of “regression toward the mean” in heredity. Logistic regression earns the name because it is linear regression of the log-odds.

A model produces a probability distribution. We typically want to turn the probability into a label prediction (a “yes” or “no”). One common rule is to predict $1$ exactly when its probability is at least $1/2$. Since $\sigma(0)=1/2$ and $\sigma$ is increasing, this is equivalent to

$$\mathbf{w}^\top\mathbf{x}+b\geq 0.$$

The map $\mathbf{x}\mapsto\mathbf{w}^\top\mathbf{x}+b$ is affine: it is a linear map plus a constant. Its zero set is an affine hyperplane, and we define this set to be the decision boundary:

$$\{\mathbf{x}\in\mathbb R^d:\mathbf{w}^\top\mathbf{x}+b=0\},$$

where the prediction switches from one label to the other. The equation above describes a translated hyperplane normal to $\mathbf{w}$. As the parameters $(\mathbf{w},b)$ vary, the decision boundary can translate and rotate, but it is always a hyperplane. This flatness is the limitation that hidden layers will remove in Section 4.

As a summary of binary logistic regression, here is a schematic:

Now, how to train our model (choose a good $\theta$)? For a binary classification model, the training set consists of pairs called training samples

$$(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_N,y_N), \qquad \mathbf{x}_i\in\mathbb R^d,\quad y_i\in\{0,1\}.$$

Compare the following definition to the likelihood and average negative log likelihood of Lecture 1, Section 3: it is the same recipe, with the (context, next-token) pairs of a language model replaced by (feature vector, label) pairs.

Let us apply this to the binary classification model. From its definition, our model with parameter $\theta=(\mathbf{w},b)$ returns a probability typically denoted

$$\widehat y:=\sigma(\mathbf{w}^\top\mathbf{x}+b).$$

This $\widehat y$ is a compact way to describe the probability distribution, since

$$p_\theta(y\mid\mathbf{x}) = \begin{cases} \widehat y,&y=1,\\ 1-\widehat y,&y=0 \end{cases} \;=\; \widehat y^{\,y}(1-\widehat y)^{1-y}$$

(a Bernoulli distribution with parameter $\widehat y$).

Therefore the log loss for an individual training pair $(\mathbf{x},y)$ is

$$\ell(\theta;\mathbf{x},y) =-\log\left(\widehat y^{\,y}(1-\widehat y)^{1-y}\right) =-\left[ y\log\widehat y+(1-y)\log(1-\widehat y) \right].$$

When $y=1$, the second term vanishes and $\ell=-\log\widehat y$. When $y=0$, the first term vanishes and $\ell=-\log(1-\widehat y)$. We use the convention $0\log0=0$ in formulas of this kind.

Notationally, $\widehat y$ is typically used for a model prediction (here, the predicted probability of label $1$), and $y, y_i \in \mathcal C$ for gold labels. Training involves comparing these two, and loss is a measure of how badly the model predicted reality.

In short: correct predictions correspond to smaller losses. Falsely assigning a small probability $p_\theta(y_i | \mathbf{x}_i)$ to the gold label $y_i$ incurs a large loss. In particular, the loss diverges as the probability assigned to the observed label tends to zero.

---

1. Gradient descent for logistic regression

The definition of the maximum likelihood estimator $\widehat\theta \in \Theta \subseteq \mathbb R^k$ (Definition 0.4) tells us what we want, but not how to find it. For a differentiable function $F:\mathbb R^k\to\mathbb R$, its gradient at $\theta$ is the vector of partial derivatives

$$\nabla F(\theta) = \begin{pmatrix} \frac{\partial F}{\partial\theta_1}(\theta)\\ \vdots\\ \frac{\partial F}{\partial\theta_k}(\theta) \end{pmatrix}.$$

The directional derivative in direction $v \in \mathbb R^k$ is

$$\left.\frac{d}{dt}F(\theta+tv)\right|_{t=0} = \nabla F(\theta)^\top v.$$

Proof

Recall a classical fact from linear algebra, the Cauchy–Schwarz inequality:

For all $u,v\in\mathbb R^k$,

$$|u^\top v|\le\|u\|\,\|v\|,$$

with equality if and only if $u$ and $v$ are linearly dependent.

Apply this with $u=\nabla F(\theta)$. For any unit vector $v$,

$$\nabla F(\theta)^\top v \;\geq\; -\left|\nabla F(\theta)^\top v\right| \;\geq\; -\|\nabla F(\theta)\|,$$

and by the equality case of Cauchy–Schwarz, the lower bound $-\|\nabla F(\theta)\|$ is attained only when $v$ is a scalar multiple of $\nabla F(\theta)$ — for a unit vector, only when $v=\pm \nabla F(\theta)/\|\nabla F(\theta)\|$. Trying both sign cases, we find that among unit vectors $v$, the directional derivative is smallest when $v$ points opposite the gradient. $\square$

For one logistic-regression training sample $(\mathbf{x},y)$, we consider the log loss $\ell(\theta) := \ell(\theta; \mathbf{x}, y)$ of our model $p_\theta(y\mid\mathbf{x})$ as a function of $\theta = (\mathbf{w}, b)$. Recall that our goal is to choose $\theta$ to result in small loss. The method of gradient descent will be defined formally below, but in brief, the idea is this. We will compute the gradient of $\ell$ at the position $\theta$, and use the result to estimate an improved $\theta$ by taking a step in the direction of decrease of loss. In equations:

$$\theta_{\text{new}} = \theta-\eta\,\nabla\ell(\theta),$$

where $\eta>0$ is a small fixed number called the learning rate.

For logistic regression we take $F=\mathcal L$, the average cross-entropy loss, as a function on $\Theta = \mathbb R^k$; here $k=d+1$, viewing $\theta=(\mathbf{w},b)\in\mathbb R^d\times\mathbb R$ as a single vector.

We now demonstrate one step of gradient descent explicitly for logistic regression. Let us assume we have a single training sample $(\mathbf{x},y)$. The method of gradient descent views the log loss $\mathcal{L}= \ell$ as a function of $\theta$, for our fixed training sample. So $\mathbf{x}$ and $y$ are constants, while $\theta = (\mathbf{w},b)$ are the variables. Recall that we write

$$z=\mathbf{w}^\top\mathbf{x}+b$$

for the logit and $\widehat y = \sigma(z)$ for the probability, as in Section 0, both functions of $\theta$. One step of gradient descent is

$$\theta_{\text{new}} = \theta-\eta\,\nabla\ell(\theta).$$

Proof

Recall from Definition 0.4 that $\ell$ is the log loss of our training sample,

$$\ell(\theta) = -\log p_\theta(y\mid\mathbf{x}) = -\left[y\log\widehat y+(1-y)\log(1-\widehat y)\right],$$

where $\widehat y=\sigma(\mathbf{w}^\top\mathbf{x}+b)$ is the probability the model assigns to the label $1$.

We wish to compute the vector

$$\nabla\ell(\theta) = \left( \frac{\partial\ell}{\partial w_1},\, \ldots,\, \frac{\partial\ell}{\partial w_d},\, \frac{\partial\ell}{\partial b} \right)^\top.$$

The loss depends on $\theta$ only through the logit $z$, so the chain rule expresses every one of these partial derivatives in terms of the single scalar $\partial\ell/\partial z$:

$$\frac{\partial\ell}{\partial w_j} = \frac{\partial\ell}{\partial z}\, \frac{\partial z}{\partial w_j}, \qquad \frac{\partial\ell}{\partial b} = \frac{\partial\ell}{\partial z}\, \frac{\partial z}{\partial b}.$$

From $z=w_1x_1+\cdots+w_dx_d+b$ we read off $\partial z/\partial w_j=x_j$ and $\partial z/\partial b=1$, so

$$\nabla\ell(\theta) = \frac{\partial\ell}{\partial z} \,(x_1,\ldots,x_d,1)^\top = \frac{\partial\ell}{\partial z} \begin{pmatrix}\mathbf{x}\\1\end{pmatrix}.$$

It remains to compute the scalar $\partial\ell/\partial z$. The loss depends on $z$ through the probability $\widehat y=\sigma(z)$, so we apply the chain rule once more:

$$\frac{\partial\ell}{\partial z} = \frac{\partial\ell}{\partial \widehat y}\, \frac{d\widehat y}{dz}.$$

Differentiating the binary cross-entropy $\ell = -\left[y\log\widehat y+(1-y)\log(1-\widehat y)\right]$ with respect to $\widehat y$ gives

$$\frac{\partial\ell}{\partial \widehat y} = -\frac{y}{\widehat y}+\frac{1-y}{1-\widehat y},$$

and the sigmoid has the pleasant derivative

$$\frac{d\widehat y}{dz} = \frac{d\sigma}{dz}(z) = \frac{e^{-z}}{(1+e^{-z})^2} = \sigma(z)(1-\sigma(z)) = \widehat y(1-\widehat y).$$

Multiplying the two,

$$\begin{aligned} \frac{\partial\ell}{\partial z} &= \left( -\frac{y}{\widehat y}+\frac{1-y}{1-\widehat y} \right)\widehat y(1-\widehat y)\\ &=-y(1-\widehat y)+(1-y)\widehat y\\ &=\widehat y-y. \end{aligned}$$

Assembling the pieces gives the boxed formula. $\square$

Let us pause to interpret our computation. Recall that $\widehat y$ is the predicted probability of our model, as a function of $\theta$. And $y$ is the gold label for our training sample (a constant). The factor $\widehat y-y$ is called the prediction error. If $y=1$ and the model assigns too small a probability $\widehat y$, then $\widehat y-y<0$: the negative-gradient update increases the logit. If the target is $0$, then $\widehat y-y>0$: the update decreases it.

Now suppose we have a larger training set $(\mathbf{x}_i, y_i)_{i=1}^N$. Let $X\in\mathbb R^{N\times d}$ have row $\mathbf{x}_i^\top$, and let

$$\widehat y_i=\sigma(\mathbf{w}^\top\mathbf{x}_i+b).$$

If $\widehat{\mathbf{y}}=(\widehat y_1,\ldots,\widehat y_N)^\top$ and $\mathbf{y}=(y_1,\ldots,y_N)^\top$ are the vectors of probabilities and gold labels, and $\mathbf1=(1,\ldots,1)^\top\in\mathbb R^N$ denotes the all-ones vector, then

$$\nabla_{\mathbf{w}}\mathcal L = \frac1N X^\top(\widehat{\mathbf{y}}-\mathbf{y}) \qquad \frac{\partial\mathcal L}{\partial b} = \frac1N\mathbf1^\top(\widehat{\mathbf{y}}-\mathbf{y})$$

Writing this out in matrix form, with the two blocks stacked into a single vector,

$$\nabla\mathcal{L}(\theta) = \frac{1}{N} \begin{pmatrix} x_{11} & x_{21} & \cdots & x_{N1}\\ x_{12} & x_{22} & \cdots & x_{N2}\\ \vdots & \vdots & & \vdots\\ x_{1d} & x_{2d} & \cdots & x_{Nd}\\ 1 & 1 & \cdots & 1 \end{pmatrix} \begin{pmatrix} \widehat y_1 - y_1\\ \widehat y_2 - y_2\\ \vdots\\ \widehat y_N - y_N \end{pmatrix},$$

where $x_{ij}$ denotes the $j$th coordinate of $\mathbf{x}_i$, so that the first $d$ rows of the matrix form $X^\top$ and the final row of ones collects the bias derivatives.

This is the same calculation for every observation, collected into two matrix operations. The choice of transposes is simply further evidence that machine learning has a tendency to strange conventions. As before, the vector $\widehat{\mathbf{y}} - \mathbf{y}$ is the prediction error.

Another way to present these formulas is as an average. The log loss $\mathcal L$ is an average of the per-sample log losses $\ell(\theta;\mathbf{x}_i,y_i)$, and differentiation is linear, so the gradient of the log loss is the average of the per-sample gradients:

$$\nabla\mathcal L(\theta) = \frac1N\sum_{i=1}^N\nabla\ell(\theta;\mathbf{x}_i,y_i) = \frac1N\sum_{i=1}^N(\widehat y_i-y_i) \begin{pmatrix}\mathbf{x}_i\\1\end{pmatrix}.$$

A gradient descent step

$$\theta^{(t+1)} := \theta^{(t)}-\eta\nabla \mathcal{L}(\theta^{(t)}), \quad t \ge 0,$$

that uses this average over the whole training set (all $N$ samples) is called a full-batch step. The training set, used all at once in this way, is called the batch.

One run of gradient descent on the full training set is called an epoch of training. An alternative way to perform an epoch is minibatch, which averages over a small subset of the training set at each step, because for large training sets the full batch is too expensive to revisit at every training step. Note that different batch sizes do result in different gradient descent paths, hence different training outcomes.

The gradient descent algorithm for binary logistic regression becomes the following.

Now we turn to asking whether gradient descent is well-behaved. If the function we are minimizing is convex, every local minimum is a global minimum, so we can hope that gradient descent will get us to a global minimum. Gradient descent is nice in this way in the case of logistic regression.

Geometrically: the chord joining any two points of the graph lies on or above the graph. A convex function has no separate valleys for gradient descent to get trapped in.

How do we check that a function is convex? For a twice-differentiable function $F$ of one variable, convexity is equivalent to $F''\geq0$ everywhere: the slope never decreases. In $m$ variables, the role of the second derivative is played by a matrix.

The quadratic form $\mathbf{u}^\top\nabla^2F(\theta)\,\mathbf{u}$ is the second derivative of $F$ along the line through $\theta$ in direction $\mathbf{u}$, so the theorem says exactly that $F$ is convex if and only if its restriction to every line has nonnegative second derivative.

Let us apply this test to the logistic-regression loss.

Proof

We recap the setting from above: the parameter is $\theta=(\mathbf{w},b)$, the training samples are $(\mathbf{x}_i,y_i)_{i=1}^N$, and the probabilities are $\widehat y_i=\sigma(\mathbf{w}^\top\mathbf{x}_i+b)$. In the gradient we computed in Section 1, the parameter and each feature vector appear with one extra coordinate appended, the bias $b$ in one case and the constant $1$ in the other, so it is worth naming the stacked vectors. Write

$$\mathbf{v}_i= \begin{pmatrix} \mathbf{x}_i\\ 1 \end{pmatrix}, \qquad \theta= \begin{pmatrix} \mathbf{w}\\ b \end{pmatrix} \in\mathbb R^{d+1},$$

so that $z_i=\theta^\top\mathbf{v}_i$, $\widehat y_i=\sigma(z_i)$, and the gradient reads

$$\nabla\mathcal L(\theta) = \frac1N\sum_{i=1}^N(\widehat y_i-y_i)\mathbf{v}_i.$$

Differentiate the gradient once more. The only dependence on $\theta$ on the right side of the gradient formula is through $\widehat y_i=\sigma(\theta^\top\mathbf{v}_i)$, so by the chain rule and the sigmoid derivative $\sigma'=\sigma(1-\sigma)$,

$$\nabla \widehat y_i = \sigma'\bigl(\theta^\top\mathbf{v}_i\bigr)\,\nabla\bigl(\theta^\top\mathbf{v}_i\bigr) = \widehat y_i(1-\widehat y_i)\,\mathbf{v}_i.$$

The $i$th summand of the gradient is the fixed vector $\mathbf{v}_i$ scaled by $\widehat y_i-y_i$, so its derivative in $\theta$ is $\mathbf{v}_i$ times the row vector $(\nabla \widehat y_i)^\top$, that is, the outer product $\widehat y_i(1-\widehat y_i)\mathbf{v}_i\mathbf{v}_i^\top$. Averaging the summands,

$$\nabla^2\mathcal L(\theta) = \frac1N\sum_{i=1}^N \widehat y_i(1-\widehat y_i)\mathbf{v}_i\mathbf{v}_i^\top.$$

For any $\mathbf{u}$,

$$\mathbf{u}^\top\nabla^2\mathcal L(\theta)\,\mathbf{u} = \frac1N\sum_{i=1}^N \widehat y_i(1-\widehat y_i)(\mathbf{u}^\top\mathbf{v}_i)^2 \geq0.$$

Thus the Hessian is positive semidefinite at every $\theta$, and by the second-order test, $\mathcal L$ is convex.

For the upper bound, we estimate each summand twice over. First, the function $s\mapsto s(1-s)$ on $[0,1]$ attains its maximum $1/4$ at $s=1/2$, so $\widehat y_i(1-\widehat y_i)\leq1/4$. Second, by the Cauchy–Schwarz inequality,

$$(\mathbf{u}^\top\mathbf{v}_i)^2 \leq \|\mathbf{u}\|^2\|\mathbf{v}_i\|^2 \leq R^2\|\mathbf{u}\|^2.$$

Each summand is therefore at most $\tfrac{R^2}{4}\|\mathbf{u}\|^2$, and so is their average. $\square$

Recall from the discussion of the second-order test that $\mathbf{u}^\top\nabla^2\mathcal L(\theta)\,\mathbf{u}$ is the second derivative of $\mathcal L$ along the line through $\theta$ in direction $\mathbf{u}$. The upper bound in the theorem therefore says that the gradient cannot change arbitrarily fast: along any unit direction, the second derivative of the loss is at most $R^2/4$.

Proof sketch

A differentiable convex function with Hessian bounded above by $MI$ satisfies the standard smooth-convex inequality

$$\left\langle \nabla\mathcal L(\theta), \theta-\widehat\theta \right\rangle \geq \mathcal L(\theta)-\mathcal L(\widehat\theta) + \frac{1}{2M}\|\nabla\mathcal L(\theta)\|^2.$$

Insert the update $\theta^+=\theta-\eta\nabla\mathcal L(\theta)$ into $\|\theta^+-\widehat\theta\|^2$. Expanding the square and using the displayed inequality, together with $\eta\leq1/M$, gives

$$2\eta\left[ \mathcal L(\theta)-\mathcal L(\widehat\theta) \right] \leq \|\theta-\widehat\theta\|^2 - \|\theta^+-\widehat\theta\|^2.$$

Apply this at iterations $0,\ldots,t-1$ and sum. The right side telescopes. The loss is non-increasing for this step size, so its final gap is at most the average of the preceding gaps, which gives the claimed bound. $\square$

Let us make a few notes on the hypotheses.

First, the restriction $\eta\leq1/M$ is necessary: a learning rate that is too large can overshoot. For a simple example, consider linear regression (not logistic regression). In that case, the loss looks like a parabola $\mathcal{L}(\theta)=\tfrac{M}{2}\theta^2$. One gradient step looks like

$$\theta^{(t+1)}=\theta^{(t)}-\eta M\theta^{(t)}=(1-\eta M)\,\theta^{(t)}.$$

In this example, if $\eta<2/M$, the factor $1-\eta M$ has absolute value less than $1$ and the iterates converge to the minimum at $0$; but if $\eta>2/M$, the factor has absolute value greater than $1$, and each step jumps past the minimum to a point farther away than where it started. The iterates oscillate in sign with growing amplitude, and the loss diverges.

Second, the theorem assumes that a finite minimizer $\widehat\theta$ exists. This can fail. Call a training pair $(\mathbf{x},y)$ a positive sample if its gold label is $y=1$, and a negative sample if $y=0$. A binary dataset is linearly separable if some affine hyperplane places the feature vector of every positive sample strictly on one side and the feature vector of every negative sample strictly on the other. It turns out that if the data are separable, the infimum is never attained by any finite parameter, and the theorem does not apply. The converse is nearly true: a classical dichotomy of Albert and Anderson says that a finite minimizer exists if and only if the data cannot even be weakly separated, i.e. there is no nonzero $(\mathbf{w},b)$ with $\mathbf{w}^\top\mathbf{x}+b\geq0$ for every positive sample and $\leq0$ for every negative sample. In particular, strictly separable data never admit a finite minimizer.

The standard repair, when a finite minimizer is wanted, is to add a penalty, or regularizer: an extra term, added to the loss, that grows as the parameters grow, so that large parameters are penalized. The most common choice is the penalty $\lambda\|\theta\|^2$ with $\lambda>0$ — that is, we minimize $\mathcal L(\theta)+\lambda\|\theta\|^2$ instead of $\mathcal L(\theta)$. This works because the penalized objective tends to $+\infty$ as $\|\theta\|\to\infty$, so it attains a finite minimizer even on separable data, at the price of a small bias toward small weights. The penalty also improves the curvature. Its Hessian, as a function of $\theta$, is $2\lambda I$, where $I$ denotes the identity matrix, so the penalized objective has Hessian bounded below by $2\lambda I$; a function whose Hessian is bounded below by $\mu I$ for some $\mu>0$ is called strongly convex. Strong convexity guarantees a unique minimizer, and for such objectives gradient descent converges geometrically — the error shrinks by a constant factor each step — rather than at the slower $1/t$ rate of Theorem 1.11.

---

2. Softmax and the simplex

Binary logistic regression produces a distribution on two labels. A language model needs a distribution on an entire vocabulary $\mathcal{V}$ (Lecture 1, Section 0). Therefore, let $\mathcal{C} := \mathcal{V}$, so that the number of possible labels is $V=|\mathcal V|$, and write

$$\Delta^{V-1} = \Delta(\mathcal{V}) = \left\{ p\in\mathbb R^V: p_b\geq0,\ \sum_{b=1}^Vp_b=1 \right\}.$$

This is the probability simplex from Lecture 1. Its relative interior is

$$\operatorname{relint}\Delta^{V-1} = \left\{ p\in\Delta^{V-1}:p_b>0\text{ for every }b \right\}.$$

It is called a relative interior because the simplex lies in the affine hyperplane $\sum_b p_b=1$, which has dimension $V-1$ rather than $V$. So we are taking its interior with respect to this hyperplane.

For $V=2$, write $z=(z_1,z_2)^\top$. The second softmax coordinate is a sigmoid of the logit difference:

$$\operatorname{softmax}(z)_2 = \frac{e^{z_2}}{e^{z_1}+e^{z_2}} = \frac{1}{1+e^{-(z_2-z_1)}} = \sigma(z_2-z_1).$$

The first coordinate is $1-\operatorname{softmax}(z)_2$, which carries the same information. Thus two-class softmax gives exactly the binary distribution of Section 0 if one uses $z_2-z_1$ as the binary logit and discards the redundant first coordinate in favour of the second one.

Each coordinate $e^{z_b}/\sum_c e^{z_c}$ is positive, and the coordinates sum to $1$, so softmax does land in $\operatorname{relint}\Delta^{V-1}$, as the stated codomain asserts. The entries $z_b$ of the input vector are called logits, extending the terminology of Definition 0.2.

Recall from Definition 0.1 that for an event of probability $p$, the odds are $p/(1-p)$ and the logit is the log-odds $\log\bigl(p/(1-p)\bigr)$. With $V$ labels, the natural analogue compares two labels at a time: given a distribution $p\in\operatorname{relint}\Delta^{V-1}$, the ratio $p_b/p_c$ is the odds of label $b$ against label $c$, and $\log(p_b/p_c)$ is their pairwise log-odds. For softmax the pairwise log-odds have an extremely simple form:

$$\log \frac{\operatorname{softmax}(z)_b} {\operatorname{softmax}(z)_c} = z_b-z_c.$$

Thus only differences between logits matter to the output distribution: the pairwise odds depend on $z$ only through the differences $z_b-z_c$, and since a probability vector is determined by its coordinate ratios (the coordinates must sum to $1$), so does $\operatorname{softmax}(z)$ itself. In particular, adding the same constant to every logit leaves the output unchanged.

The preceding example reveals that softmax has one redundant direction. We now describe the geometry precisely. Let $\mathbf1=(1,\ldots,1)^\top\in\mathbb R^V$, let

$$\mathbf1^\perp = \{z\in\mathbb R^V:\mathbf1^\top z=0\},$$

and for $u\in\mathbb R^V$ write

$$\operatorname{mean}(u)=\frac1V\sum_{b=1}^Vu_b.$$

That is, $\mathbf1^\perp$ is the kernel of the mean: those vectors $z$ whose mean is zero.

A diffeomorphism is a smooth bijection with a smooth inverse.

Proof

We already observed that softmax has positive coordinates summing to $1$, so its image is contained in the relative interior. In the other direction, take $p\in\operatorname{relint}\Delta^{V-1}$. Every logarithm is defined, and

$$\operatorname{softmax}(\log p)_b = \frac{e^{\log p_b}}{\sum_c e^{\log p_c}} = \frac{p_b}{\sum_c p_c} = p_b.$$

This proves the first assertion.

For any scalar $c$,

$$\begin{aligned} \operatorname{softmax}(z+c\mathbf1)_b &= \frac{e^{z_b+c}}{\sum_r e^{z_r+c}}\\ &= \frac{e^ce^{z_b}}{e^c\sum_r e^{z_r}}\\ &= \operatorname{softmax}(z)_b. \end{aligned}$$

This proves shift invariance. Every affine line $z+\mathbb R\mathbf1$ contains exactly one zero-mean representative,

$$z^0=z-\operatorname{mean}(z)\,\mathbf1.$$

Indeed, $z^0\in\mathbf1^\perp$, while $z+c\mathbf1$ has mean $\operatorname{mean}(z)+c$.

It remains to verify the inverse. For $p \in \operatorname{relint}\Delta^{V-1}$, the vector $g(p)$ has mean zero. Shift invariance and the first calculation give

$$\operatorname{softmax}(g(p)) = \operatorname{softmax}(\log p) = p.$$

Conversely, take $z\in\mathbf1^\perp$ and set $S=\sum_c e^{z_c}$. Coordinatewise,

$$\log(\operatorname{softmax}(z)) = z-(\log S)\mathbf1.$$

Since $z$ has mean zero, the mean of the vector on the right is $-\log S$. Subtracting this mean recovers $z$, so

$$g(\operatorname{softmax}(z))=z.$$

Softmax is smooth because it is built from exponentials with a positive denominator, and $g$ is smooth where every $p_b>0$. The restricted map is therefore a diffeomorphism. $\square$

Restricting to zero-mean logits is one way to choose a unique representation. Fixing a specified logit to be zero is another.

The term identifiable means that different parameter values produce different modeled distributions. Unrestricted softmax logits are not identifiable: $z$ and $z+c\mathbf1$ represent the same distribution. Their differences are identifiable.

---

3. Multinomial logistic regression

We now repeat binary logistic regression with $V$ possible labels. The parallel is exact:

The logits will again be affine functions of the features.

The most probable label is the one with largest logit. The decision boundary between labels $j$ and $r$ is

$$z_j(\mathbf{x})=z_r(\mathbf{x}),$$

or equivalently

$$(\mathbf w_j-\mathbf w_r)^\top\mathbf{x} +(\beta_j-\beta_r)=0.$$

It is a hyperplane. The pairwise boundaries divide $\mathcal X=\mathbb R^d$ into polyhedral regions. Within each region, the ordering of the logits—and hence the ordering of the class probabilities—is fixed.

For next-token prediction, take $\mathcal C=\mathcal V$. The feature vector $\mathbf{x}$ is a numerical representation of the context, and the observed class $y$ is the next token.

For a target class $j\in\{1,\ldots,V\}$, let $e_j\in\mathbb R^V$ denote its one-hot vector: coordinate $j$ is $1$ and every other coordinate is $0$ (a pure mathematician would call this a standard basis vector 🙃). We will use the indicator notation

$$\mathbf1_{\{A\}} = \begin{cases} 1,&\text{if the statement }A\text{ is true},\\ 0,&\text{if it is false}. \end{cases}$$

Thus $(e_y)_j=\mathbf1_{\{j=y\}}$.

These are the quantities of Definition 0.4, specialized to the softmax model. They follow the same maximum-likelihood recipe as Lecture 1, Section 3.

Fix one sample $(\mathbf{x},y)$. We next differentiate through the logits $z=W^\top\mathbf{x}+\beta$.

Proof

The loss depends on $(W,\beta)$ only through the $V$ logits $z_j=\mathbf w_j^\top\mathbf{x}+\beta_j$. As a function of $z$,

$$\ell=-z_y+\log\sum_{r=1}^V e^{z_r}.$$

For each $j$,

$$\begin{aligned} \frac{\partial\ell}{\partial z_j} &= -\mathbf1_{\{j=y\}} + \frac{e^{z_j}}{\sum_r e^{z_r}}\\ &= p_j-\mathbf1_{\{j=y\}}. \end{aligned}$$

Hence $\nabla_z\ell=p-e_y$. Since $\nabla_{\mathbf w_j}z_j=\mathbf{x}$ and $\partial z_j/\partial\beta_j=1$,

$$\nabla_{\mathbf w_j}\ell = (p_j-\mathbf1_{\{j=y\}})\mathbf{x}, \qquad \frac{\partial\ell}{\partial\beta_j} = p_j-\mathbf1_{\{j=y\}}.$$

Assembling the weight derivatives column by column gives $\nabla_W\ell=\mathbf{x}(p-e_y)^\top$. $\square$

The logit-gradient coordinates sum to zero:

$$\mathbf1^\top(p-e_y)=1-1=0,$$

where $\mathbf1=(1,\ldots,1)^\top\in\mathbb R^V$. This had to happen: the loss is unchanged when every logit receives a common shift, so its directional derivative along $\mathbf1$ vanishes.

The weight gradient is an outer product. Its $j$th column is $(p_j-\mathbf1_{\{j=y\}})\mathbf{x}$: prediction error for class $j$, times the feature vector. In the two-class setting of Aside 3.1a, the two coordinates are opposites. If $q=p_\theta(y=2\mid\mathbf{x})$ and the target is class $2$, then

$$p-e_2=(1-q,q-1)^\top,$$

which is the binary derivative $q-1$ expressed in the redundant two-logit coordinates.

For Example 3.3,

$$\nabla_z\ell = \left(\frac16,-\frac23,\frac12\right)^\top.$$

A negative-gradient step raises the target logit relative to the other two: it adds $2\eta/3$ to $z_2$ while subtracting $\eta/6$ from $z_1$ and $\eta/2$ from $z_3$.

Over a training set, the empirical gradient is the average of the per-sample gradients. We can compute all samples at once by collecting them row by row.

A minibatch is a small randomly chosen subset of the training samples. Minibatch methods use the same formulas with the average over a fresh minibatch in place of the average over all $N$ samples. Each step becomes much cheaper, at the price of a noisy estimate of the full gradient.

Proof

Differentiating $p_j=e^{z_j}/\sum_r e^{z_r}$ gives

$$\frac{\partial p_j}{\partial z_k} = p_j(\mathbf1_{\{j=k\}}-p_k).$$

This is the $(j,k)$ entry of $\operatorname{diag}(p)-pp^\top$. For every $u\in\mathbb R^V$,

$$\begin{aligned} u^\top\left(\operatorname{diag}(p)-pp^\top\right)u &= \sum_jp_ju_j^2-\left(\sum_jp_ju_j\right)^2\\ &= \operatorname{Var}_{J\sim p}(u_J) \geq0. \end{aligned}$$

Thus the loss is convex in $z$. Since $z$ is affine in $(W,\beta)$, composition with this affine map preserves convexity, as does averaging over samples. $\square$

---

4. From individual units to feedforward networks

To increase expressiveness, we can combine “units” of logistic regression into networks.

4.1 Artificial neurons

We have already met a unit: binary logistic regression (Definition 0.2) computes exactly one unit, with the sigmoid as its activation function. The activation function is $\phi=\sigma$; the weights $\mathbf{w}$ and bias $b$ of the unit are the weights and bias of the logistic regression model; the preactivation $a=\mathbf{w}^\top\mathbf{x}+b$ is the logit $z_\theta(\mathbf{x})$; and the activation $h=\sigma(a)$ is the predicted probability $p_\theta(y=1\mid\mathbf{x})=\widehat y$.

(Softmax regression is close to being a collection of $V$ units, one per logit, but not quite: the softmax is applied jointly to the whole logit vector, rather than to each preactivation separately.)

One historically important activation is the step function

$$H(t) = \begin{cases} 0,&t<0,\\ 1,&t\geq0. \end{cases}$$

A unit $\mathbf{x}\mapsto H(\mathbf{w}^\top\mathbf{x}+b)$, i.e. a unit using the step function activation, is called a perceptron. It makes a hard binary decision separated by a hyperplane in $\mathbb R^d$.

The first question to ask is how expressive such units are. This is the question of what space of possible functions they represent.

The Boolean exclusive-or function, or XOR, is the function $\operatorname{XOR} : \{0,1\} \times \{0,1\} \rightarrow \{0,1\}$ given by

$x_1$	$x_2$	$\operatorname{XOR}(x_1,x_2)$
$0$	$0$	$0$
$1$	$0$	$1$
$0$	$1$	$1$
$1$	$1$	$0$

Proof

Correct classification at $(0,0)$ requires

$$b<0.$$

Correct classification at $(1,0)$ and $(0,1)$ requires

$$w_1+b\geq0, \qquad w_2+b\geq0.$$

Adding these two inequalities gives $w_1+w_2+2b\geq0$. Since $b<0$,

$$w_1+w_2+b\geq-b>0.$$

But correct classification at $(1,1)$ requires $w_1+w_2+b<0$, a contradiction. $\square$

4.2 Combining units into networks

The rectified linear unit is

$$\operatorname{ReLU}(t)=\max\{0,t\}.$$

Unlike the perceptron, it retains the size of a positive preactivation. Consider two units

$$h_1=\operatorname{ReLU}(x_1+x_2), \qquad h_2=\operatorname{ReLU}(x_1+x_2-1),$$

and a third unit, taking $(h_1,h_2)$ as its input:

$$f(x_1,x_2)=\operatorname{ReLU}(h_1-2h_2).$$

We will now string these together. The diagram below shows how the units should be imagined as being arranged in layers. The final unit(s) are the output unit(s) — those whose results are not fed into further units. Units whose outputs do feed into other units are called hidden units. Here, $h_1$ and $h_2$ are hidden units and $f$ is the single output unit.

On the four possible inputs, $s=x_1+x_2$ is $0$, $1$, or $2$, and

$$f(x_1,x_2) = \operatorname{ReLU}\bigl( \operatorname{ReLU}(s) - 2\operatorname{ReLU}(s-1) \bigr).$$

The complete calculation is

(The output unit’s ReLU changes nothing here, since $h_1-2h_2$ is already nonnegative at all four inputs.)

Building a network with hidden units has changed the expressivity: now we can express XOR.

A scalar activation $\phi:\mathbb R\to\mathbb R$ acts on vectors coordinatewise:

$$\phi(u_1,\ldots,u_n) = (\phi(u_1),\ldots,\phi(u_n)).$$

A feedforward network $f_\theta$ is a deterministic function, not by itself a probabilistic model. When $n_L=V$, softmax gives the associated classifier

$$P(\theta,\mathbf{x}) = \operatorname{softmax}(f_\theta(\mathbf{x})).$$

Some authors count depth differently, for example by counting only hidden layers. Any depth statement should therefore declare its convention. We will care mainly about whether depth grows linearly or stays fixed, so a difference of one is harmless.

At a hidden unit $j$ in layer $\ell$, the node-and-arrow calculation is

$$(h_\ell)_j = \phi_\ell \left( \sum_{k=1}^{n_{\ell-1}} (W_\ell)_{jk}(h_{\ell-1})_k + (b_\ell)_j \right).$$

Writing all units at once gives $h_\ell=\phi_\ell(W_\ell h_{\ell-1}+b_\ell)$. The diagram and the matrix formula are two views of exactly the same computation.

In fact, if we use the identity function as the activation, and zero biases, then the fancy diagram above is literally a diagram of matrix multiplication 🙃.

Proof

The composition of two affine maps is affine:

$$W_2(W_1\mathbf{x}+b_1)+b_2 = (W_2W_1)\mathbf{x}+(W_2b_1+b_2).$$

Applying this identity repeatedly collapses all layers to one map

$$f_\theta(\mathbf{x})=W_{\mathrm{eff}}\mathbf{x}+b_{\mathrm{eff}}.$$

Here “eff” abbreviates effective: $W_{\mathrm{eff}}$ and $b_{\mathrm{eff}}$ are the single weight matrix and bias vector obtained after all the affine layers have been composed.

Thus additional linear layers may change the parametrization, but not the set of functions represented. $\square$

The nonlinear activation is what allows a network to bend decision boundaries and build new features. Matrix multiplication supplies rich linear mixing; activation functions prevent all of those matrices from collapsing into one.

4.3 Common activation functions

The following plots use the same horizontal scale. Their vertical scales are chosen to show the characteristic shape rather than to compare numerical magnitudes exactly.

activation	formula	strengths, limitations, and common use
step	${H(t)=\mathbf1_{\{t\geq0\}}}$	It is discontinuous and has derivative $0$ away from the threshold, so ordinary gradient descent cannot train it effectively.
sigmoid	${\sigma(t)=(1+e^{-t})^{-1}}$	Smooth, bounded. It can saturate: for large $\lvert t\rvert$, its derivative is near $0$, weakening gradients.
hyperbolic tangent	$\tanh t$	Smooth, bounded, centered at $0$. Historically common and used in Project Step 3; it also saturates.
ReLU	$\max\{0,t\}$	Cheap, non-saturating on the positive half-line, piecewise linear. It is not differentiable at $0$, and a unit stuck on the negative side can receive zero gradient.
GELU	$t\Phi(t)$	A smooth version of ReLU. Common in transformers, used in Project Step 5; it costs more to evaluate than ReLU. Here $\Phi$ denotes the cumulative distribution function of a standard
normal random variable.

Replacing the step function by a smooth or piecewise-linear activation lets a unit retain how far its preactivation lies from zero. Differentiability matters for training (next lecture), although an isolated corner such as ReLU’s at zero is easily handled. For a single unit, changing the activation does not by itself bend the level sets of the preactivation; the nonlinear geometry comes from combining units.

An activation saturates where changing its input has almost no effect on its output, equivalently where its derivative is close to zero. Sigmoid and $\tanh$ saturate in both tails. During backpropagation, derivatives from successive layers are multiplied, so many saturated units can make a gradient very small before it reaches an early layer. ReLU does not saturate on its positive half-line, but on its negative half-line its derivative is zero; a ReLU that remains there on all relevant inputs is sometimes called a dead unit.

Softmax is different from every entry in the table. It acts on a whole vector, not coordinatewise, and its coordinates must interact so that they sum to $1$. For this reason it is commonly used in the output (last) layer (where we are trying to classify something), while ReLU, GELU, or $\tanh$ is used in hidden layers.

4.4 Neural language models

We can now put the pieces together. A context must first be represented by numbers, a feedforward network turns those numbers into one logit per vocabulary token, and softmax turns the logits into probabilities.

---

5. Universal approximation

The XOR network is one exact construction. The main result of this section says that such examples are not isolated: even a network with only one hidden layer can approximate every continuous function on a compact set, provided its activation is not a polynomial.

Let $K\subset\mathbb R^n$ be compact, and let $C(K)$ denote the real-valued continuous functions on $K$. For $f\in C(K)$, define the uniform norm

$$\|f\|_\infty = \sup_{\mathbf{x}\in K}|f(\mathbf{x})|.$$

We use this norm because it controls the worst error anywhere on $K$. An average-error norm could hide a large mistake on a small part of the domain; a uniform bound cannot. Compactness ensures that a continuous function is bounded and attains its maximum error, so the displayed quantity is finite.

A family $\mathcal F\subseteq C(K)$ is dense in $C(K)$ under the uniform norm if, for every $f\in C(K)$ and every $\varepsilon>0$, some $g\in\mathcal F$ satisfies

$$\|f-g\|_\infty<\varepsilon.$$

Here is the principal theorem.

ReLU, sigmoid, $\tanh$, and GELU are all continuous and non-polynomial, so each meets the hypothesis. The theorem permits vector outputs as well: place finitely many scalar networks side by side, one for each output coordinate.

Read the quantifiers carefully. For every target $f$ and tolerance $\varepsilon$, the theorem promises that some finite width and some parameters exist. It does not give a useful width, construct the parameters, show that gradient descent finds them, or show that finitely many training observations determine the target. In the worst case even an $L$-Lipschitz function on $[0,1]^n$ may require resolution on the scale $\varepsilon/L$ in every coordinate—roughly

$$\left(\frac{L}{\varepsilon}\right)^n$$

local regions. Thus existence alone does not make the network economical, trainable, or likely to generalize.

5.1 Examples covered by the theorem

The same idea constructs every continuous piecewise-linear function on an interval: a ReLU placed at a breakpoint changes the slope from that point onward.

The theorem also applies directly to probabilistic models.

Compactness keeps the target uniformly away from the boundary: each continuous coordinate $p_j$ has a positive minimum on $K$. If a target allows exact zero probabilities, it can first be moved by an arbitrarily small amount toward the uniform distribution.

5.2 Why the theorem is true

We first prove the interval result explicitly.

Proof of Proposition 5.3

A continuous function on a compact interval is uniformly continuous. Choose a partition

$$a=t_0<t_1<\cdots<t_m=b$$

fine enough that $|f(x)-f(y)|<\varepsilon$ whenever $x$ and $y$ lie in the same subinterval. Let $q$ be the polygonal interpolant through the points $(t_i,f(t_i))$. On $[t_{i-1},t_i]$, the value $q(x)$ is a convex combination of $f(t_{i-1})$ and $f(t_i)$. Both endpoint values differ from $f(x)$ by less than $\varepsilon$, so their convex combination does too. Hence $\|f-q\|_\infty<\varepsilon$.

Let

$$s_i = \frac{f(t_i)-f(t_{i-1})}{t_i-t_{i-1}}$$

be the slope of $q$ on the $i$th subinterval. For $x\in[a,b]$,

$$q(x) = f(t_0) + s_1\operatorname{ReLU}(x-t_0) + \sum_{j=1}^{m-1} (s_{j+1}-s_j)\operatorname{ReLU}(x-t_j).$$

The first ReLU supplies the initial slope; the unit at $t_j$ changes that slope by $s_{j+1}-s_j$. Thus a one-hidden-layer ReLU network realizes $q$ exactly. $\square$

For higher-dimensional compact sets there is no equally short polygonal formula. The following outline records the functional-analytic mechanism behind Theorem 5.1; the full projection argument is proved by Leshno, Lin, Pinkus, and Schocken (1993).

Proof outline of Theorem 5.1

A ridge function has the form $\mathbf{x}\mapsto\phi(\mathbf w^\top\mathbf{x}+b)$: it varies through one scalar projection of $\mathbf{x}$. Suppose the closed linear span of all such ridge functions were a proper subspace of $C(K)$. The Hahn—Banach theorem would give a nonzero continuous linear functional that vanishes on that span. By the Riesz representation theorem, this functional is integration against a nonzero finite signed measure $\mu$ on $K$. Consequently,

$$\int_K \phi(\mathbf w^\top\mathbf{x}+b)\,d\mu(\mathbf{x}) = 0$$

for every $\mathbf w\in\mathbb R^n$ and $b\in\mathbb R$.

Varying $\mathbf w$ and $b$ probes every one-dimensional projection of $\mu$. The projection argument in the cited paper shows that a continuous activation can admit such a nonzero annihilating measure only if it is a polynomial. Since $\phi$ is not a polynomial, $\mu$ must be zero, a contradiction. Therefore the ridge functions have dense linear span.

The non-polynomial condition is essentially necessary for shallow networks. If $\phi$ is a polynomial of degree $m$, every displayed one-hidden-layer sum is a polynomial of degree at most $m$. On a compact set with nonempty interior, that fixed finite-dimensional space is not dense in $C(K)$. $\square$

Background aside 5.5. Hahn—Banach separates a point from a closed linear subspace by a continuous linear functional. The Riesz representation theorem identifies continuous linear functionals on $C(K)$ with integration against finite signed measures. These standard results from functional analysis are not prerequisites for using the theorem.

Proof of Corollary 5.4

By Proposition 2.3, the centered logit map

$$z(\mathbf{x}) = \log p(\mathbf{x}) - \operatorname{mean}(\log p(\mathbf{x}))\,\mathbf1$$

is continuous. Approximate each coordinate $z_j$ uniformly by a shallow network, and combine all their hidden units into one network with $V$ outputs. Softmax is uniformly continuous on a compact neighborhood of $z(K)$, so sufficiently accurate logits give the required uniform probability approximation. $\square$

---

The preceding results concern representation. They do not settle optimization or generalization, which we separate next.

6. Approximation, optimization, and generalization

Three logically different questions are often compressed into the vague claim that a model “works.”

Distinguishing them requires one piece of vocabulary. The population is modeled as a probability distribution $\mathcal D$ on input–label pairs $(X,Y)$: an idealized, inexhaustible source of input–label pairs, of which the training set is one finite sample and future data are further samples. A loss $\ell(\theta;X,Y)$ penalizes the model’s prediction on a single pair; a risk is an average of losses over many pairs. Averaging over the whole distribution $\mathcal D$ gives the population risk (also called the population loss), while averaging over the training sample gives the empirical risk (the training loss). Both are defined precisely below.

question	mathematical form	what could go wrong
Approximation	Does some parameter in the model family have small population risk?	The architecture may be incapable of representing the target economically.
Optimization	Does the training algorithm find a parameter with small empirical risk?	The objective may be nonconvex, ill-conditioned, noisy, or approached only at infinite norm.
Generalization	Does performance on the training sample persist on new data?	The fitted model may exploit accidental features or memorize the sample.

To make the distinction precise, let $\mathcal D$ be the population distribution on input–label pairs $(X,Y)$, and let $\ell(\theta;X,Y)$ be a loss. The population risk is

$$R(\theta) = \mathbb E_{(X,Y)\sim\mathcal D} \left[\ell(\theta;X,Y)\right],$$

whereas the empirical risk on $N$ training observations is

$$\widehat R_N(\theta) = \frac1N\sum_{i=1}^N \ell(\theta;\mathbf{x}_i,y_i).$$

Let $R^\star$ denote the best risk attainable without restricting to the chosen model family. Then

$$\inf_\theta R(\theta)-R^\star$$

is an approximation error. For the output $\widehat\theta$ of a training algorithm,

$$\widehat R_N(\widehat\theta) - \inf_\theta\widehat R_N(\theta)$$

is an optimization error. The difference $R(\widehat\theta)-\widehat R_N(\widehat\theta)$ is one generalization gap.

Adding and subtracting these quantities gives the exact bookkeeping identity

$$\begin{aligned} R(\widehat\theta)-R^\star ={}& \underbrace{\inf_\theta R(\theta)-R^\star}_{\text{approximation}}\\ &+ \underbrace{\widehat R_N(\widehat\theta) -\inf_\theta\widehat R_N(\theta)}_{\text{optimization}}\\ &+ \underbrace{R(\widehat\theta)-\widehat R_N(\widehat\theta) +\inf_\theta\widehat R_N(\theta)-\inf_\theta R(\theta)} _{\text{estimation/generalization}}. \end{aligned}$$

The last line contains two sample-to-population discrepancies. Individual terms can have either sign, but the identity shows where distinct kinds of reasoning enter.

Remark 6.2 (a theoretical scorecard). Universal approximation and depth separation are rigorous approximation results. Logistic and softmax regression have unusually clean convex optimization theory. Deep-network losses are generally nonconvex. Optimization guarantees exist in special regimes, including some infinite-width neural-tangent-kernel limits, but those regimes do not capture every form of feature learning. Classical VC-dimension and Rademacher-complexity bounds control generalization through the size of a function class, yet can be loose or vacuous for heavily overparametrized networks. Why ordinary gradient methods so often find useful representations that generalize remains only partially understood.

---

7. The Project Step 2 diamond classifier

Example 4.6 defined the small network used in Project Step 2. We now collect its computation in one place. The point is not to train the network yet, but to see exactly how scalar units become matrix layers and how a hidden layer changes the geometry of a classifier.

Let $\mathbf{x}=(x_1,x_2)^\top$. The forward pass is

$$\mathbf{x}\in\mathbb R^2 \longmapsto a_1=W_1\mathbf{x}+b_1\in\mathbb R^4 \longmapsto h=\operatorname{ReLU}(a_1)\in\mathbb R^4 \longmapsto z=W_2h+b_2\in\mathbb R^2 \longmapsto p=\operatorname{softmax}(z)\in\Delta^1.$$

The matrices are

$$W_1= \begin{pmatrix} 1&0\\ -1&0\\ 0&1\\ 0&-1 \end{pmatrix}, \qquad b_1=0, \qquad W_2 = \gamma \begin{pmatrix} 1&1&1&1\\ -1&-1&-1&-1 \end{pmatrix}, \qquad b_2 = \gamma \begin{pmatrix} -1\\ 1 \end{pmatrix},$$

where $\gamma>0$. The output coordinates are ordered as outside, inside.

Proof

The first affine layer gives

$$a_1=(x_1,-x_1,x_2,-x_2)^\top,$$

so the hidden activation is

$$h= \bigl( \operatorname{ReLU}(x_1), \operatorname{ReLU}(-x_1), \operatorname{ReLU}(x_2), \operatorname{ReLU}(-x_2) \bigr)^\top.$$

Therefore

$$\mathbf1^\top h = |x_1|+|x_2| =:r.$$

The output logits are $z_{\mathrm{out}}=\gamma(r-1)$ and $z_{\mathrm{in}}=\gamma(1-r)$. Their difference is

$$z_{\mathrm{in}}-z_{\mathrm{out}} = 2\gamma(1-r).$$

The two-class softmax identity from Aside 3.1a now gives the probability formula. Since sigmoid is increasing and $\sigma(0)=1/2$, the three geometric claims follow. $\square$

For $\gamma=4$, a few forward passes are:

The parameter $\gamma$ changes confidence but not the decision boundary. As $\gamma\to0^+$, both logits approach zero and the output approaches the uniform distribution. As $\gamma$ grows, probabilities approach $1$ on the predicted class away from the boundary. This is the two-class version of changing softmax temperature.

---

Summary

• Binary logistic regression makes the log-odds an affine function of the features. Sigmoid converts the scalar logit to a Bernoulli probability, and binary cross-entropy is its negative log-likelihood.

• The logistic-regression gradient is prediction minus target, multiplied by the input. Its loss is convex, and suitably small-step full-batch gradient descent has a global convergence guarantee when a finite minimizer exists.

• Softmax converts $V$ logits into a strictly positive categorical distribution. It parametrizes the relative interior of the simplex with one redundant common-shift direction, and logit differences are pairwise log-odds.

• Multinomial logistic regression is the direct multiclass analogue of binary logistic regression. Its logit gradient is $p-e_y$, and its fixed-feature cross-entropy is convex.

• A perceptron has one affine boundary and cannot compute XOR. A width-two ReLU hidden layer computes XOR exactly. More generally, nonlinear activations prevent a stack of affine maps from collapsing to one affine map.

• A continuous non-polynomial activation gives one-hidden-layer universal approximation on compact sets. This establishes existence, not an efficient width, a successful optimization method, or generalization from finite data.

• Depth can express repeated composition exponentially more efficiently than a shallow network for particular function families. This is a representation theorem, not a claim about every task.

• Project Step 2 hand-wires a genuine hidden-layer network. Four ReLU units construct $|x_1|+|x_2|$; an affine output layer and softmax turn that feature into a diamond classifier. The parameter $\gamma$ changes confidence without changing the boundary.

---

Exercises (paired with Step 2)

A star marks a Project Step 2 task.

1. Odds and boundaries. For the delay model in Example 0.3, find the probability, odds, and predicted class at $\mathbf{x}=(-1,1)^\top$. Sketch its decision boundary and indicate which side is classified as a delay.

2. A gradient step by hand. Repeat Example 1.5 with learning rate $\eta=1/2$ and initial values $w^{(0)}=0$, $b^{(0)}=\log3$. Compute all logits, probabilities, gradients, updated parameters, and the loss before and after the step.

3. Prove directly from $\ell(z,y)=\log(1+e^z)-yz$ that binary logistic loss is convex in $z$. Determine when its second derivative is largest.

4. ★ Trace the diamond network. For each of $\mathbf{x}=(0,0)^\top$, $(0.5,0.25)^\top$, $(1,0)^\top$, and $(0.8,0.5)^\top$, compute $a_1$, $h$, $z$, and $p$ at $\gamma=4$. Verify Theorem 7.1 and explain the geometric role of each hidden unit.

5. ★ Move and stretch the diamond. Construct $W_1,b_1,W_2,b_2$ for a classifier whose boundary is

   $$\frac{|x_1-a|}{r_1}+\frac{|x_2-b|}{r_2}=1, \qquad r_1,r_2>0.$$

   State every matrix dimension. Check that your network reduces to the Project Step 2 network when $a=b=0$ and $r_1=r_2=1$.

6. Temperature. For $\tau>0$, analyze $\operatorname{softmax}(z/\tau)$. Prove that it tends to the uniform distribution as $\tau\to\infty$ and, as $\tau\to0^+$, to the uniform distribution on the set of maximizers of $z$. Include the unique maximum as a special case.

7. Perceptrons. Find perceptron weights and biases that compute AND and OR on $\{0,1\}^2$. Use either the inequalities in Theorem 4.2 or a geometric argument to reprove that XOR is impossible.

8. A second XOR network. Verify the weights in the XOR diagram by writing its two weight matrices and two bias vectors. Modify only the output affine map to compute XNOR on $\{0,1\}^2$.

9. Why activation matters. Derive explicit formulas for $W_{\mathrm{eff}}$ and $b_{\mathrm{eff}}$ when three affine layers are composed without activations. Identify the rank constraints imposed by bottleneck widths. Explain why widening those bottlenecks can remove the rank constraints but still cannot introduce nonlinearity.

10. Prove that a one-hidden-layer ReLU network of width $N$ on $\mathbb R$ has at most $N$ distinct breakpoints and at most $N+1$ affine pieces. When can it have fewer?

11. Triangle-map depth. Verify the three-ReLU formula for $\Delta$. Prove by induction that $\Delta^{\circ k}$ takes alternating values $0,1,0,1,\ldots,0$ at $j/2^k$ and has $2^k$ affine pieces.

12. Redundancy and regularization. Show that every rowwise softmax equivalence class contains a unique zero-mean logit vector, and that this vector has the smallest Euclidean norm in its class. Then explain why adding $\lambda\|W\|_F^2$ does more than choose representatives: it also changes the optimal probability distributions. Compare this shrinkage qualitatively with Laplace smoothing.

13. From logits to probabilities. Supply the uniform-continuity details in Corollary 5.4. Show also that for a continuous $p:K\to\operatorname{relint}\Delta^{V-1}$ with compact $K$, there is a single $\delta>0$ such that $p_b(\mathbf{x})\geq\delta$ for every $x$ and $b$.

14. Three different failures. Give one small classification example exhibiting each of the following: approximation error despite perfect optimization; optimization error despite a sufficiently expressive model; zero training loss but positive test loss. State clearly which model family, training procedure, and data distribution you use.

---

Pointers

Bengio, Ducharme, Vincent, and Jauvin, A Neural Probabilistic Language Model (2003); Cybenko (1989); Hornik (1991); Leshno—Lin—Pinkus—Schocken (1993); Pinkus, Approximation Theory of the MLP Model in Neural Networks (1999); Telgarsky, Representation Benefits of Deep Feedforward Networks (2015) and Benefits of Depth in Neural Networks (2016); Higham and Higham, Deep Learning: An Introduction for Applied Mathematicians. See the resources page, Project Step 2, and Project Step 3.