Math Building Blocks for ML Engineers

I feel the same way about Math as I do about squatting at the gym. It may not be as fun as bench or easy as arms, but if you don’t squat once a week you will get chicken legs. I try to cover the building blocks of math that are needed to approach any machine learning paper with confidence, with a strong mathematical foundation based in a visceral, intuitive understanding of the concepts. I focus on explaining in layman’s terms, with machine learning based examples, visuals to bring the ideas to life and spaced repetition cards for retention. I would not recommend learning the concepts from scratch via this blog, but instead using it as a refresher of the core intuition pumps.

Linear Algebra

Notation used in this section+

$\mathbb{R}^n$ — $n$-dimensional real vector space; $x \in \mathbb{R}^n$ is a vector with $n$ real entries

$A \in \mathbb{R}^{m \times k}$ — real matrix with $m$ rows and $k$ columns

$A^\top$ — transpose of $A$: rows and columns swapped, so $(A^\top)_{ij} = A_{ji}$

$\|x\|_p$ — the $p$-norm; subscript says which (1 = sum of absolutes, 2 = Euclidean, $\infty$ = max entry)

$\|A\|_F$ — Frobenius norm: flatten $A$ to a vector, take its L2 norm

$Av = \lambda v$ — eigenvalue equation; $v$ is an eigenvector, $\lambda$ its eigenvalue (a scalar)

$A = U\Sigma V^\top$ — SVD factorization; $U, V$ are rotation matrices, $\Sigma$ is diagonal with singular values $\sigma_i \geq 0$

Vectors

A vector can be described by how much of each basis vector it contains. A linear combination means multiplying a collection of vectors by scalars and adding the results:

$$v = a_1b_1 + a_2b_2 + \cdots + a_db_d$$

The basis vectors $b_1,\ldots,b_d$ provide the coordinate system; the scalars $a_1,\ldots,a_d$ are the parameters of $v$ in that basis. In the standard two-dimensional basis,

$$v=\begin{bmatrix}-1\\2\end{bmatrix} =-1\begin{bmatrix}1\\0\end{bmatrix} +2\begin{bmatrix}0\\1\end{bmatrix}.$$

linear combination

v = −1.0b₁ + 2.0b₂ = [−1.0, 2.0]

b₁ coefficient−1.0b₂ coefficient2.0

Vectors as semantic representations

In machine learning, a vector usually represents something rather than a literal arrow. Word2Vec was an early, vivid example: training arranged words so that semantic similarity and some relationships became geometric—nearby points and useful directions in a learned space. Modern embedding models apply the same broad idea to sentences, images, and document chunks storing them in vector databases.

Even more intriguingly, different models sometimes appear to learn similar relational geometry. The Platonic Representation Hypothesis reports that as vision and language models improve, their representations increasingly agree about which datapoints are similar—even though their raw parameters and architectures differ. This does not mean they learn identical vectors: one space may be rotated, reflected, or otherwise reparameterized relative to another. The stronger claim that all models converge to one universal representation remains debated; newer work finds clearer agreement in local neighborhoods than in the full global geometry.

Matrix Multiplication

Almost everything inside a neural net is a vector getting linearly transformed by a matrix: $Wx$, where $W$ is a weight matrix and $x$ is the incoming vector. Token embeddings and the residual stream are vectors; the attention projections $W_Q$, $W_K$, $W_V$, the MLP weights, and the unembedding head are matrices. This matrix-vector multiplication runs billions of times in a forward pass, so it is worth understanding what the matrix is actually doing to the vector.

There are several different ways to think about matrix multiplication. Mechanically, $AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$. Shapes contract along the shared dimension: $A \in \mathbb{R}^{m \times k}$ times $B \in \mathbb{R}^{k \times n}$ gives $AB \in \mathbb{R}^{m \times n}$.

A

1

2

3

4

5

6

×

B

1

0

0

1

1

1

=

AB

computing

Click Step to compute the first entry.

Step →▶ Auto-playReset

Geometrically $W$ is performing a linear transformation on $x$. You can think of a linear transformation as a function that takes a vector and can rotate, stretch, shear, reflect, or collapse it. The linear part means we never bend the vector or move the origin. What mathematicians like to do is compactly describe linear transformations with a matrix where the first column tells us where the basis vector $\left[\begin{smallmatrix}1\\0\end{smallmatrix}\right]$ lands, and the second column tells us where $\left[\begin{smallmatrix}0\\1\end{smallmatrix}\right]$ lands. $W$ can transform any vector $x$ so they think more about how it transforms the space and that is captured in how it transforms the basis vectors.

Lets take the vector, $x=\left[\begin{smallmatrix}-1\\2\end{smallmatrix}\right]$ which means:

$$x = -1\begin{bmatrix}1\\0\end{bmatrix} + 2\begin{bmatrix}0\\1\end{bmatrix}.$$

When the space moves via $W$, $x$ transforms with it. Matrix multiplication is the shortcut that carries out this recipe and tells us where any vector $x$ lands in the transformed space. For an incredible visualization please watch the goat of math education 3B1B.

W

32−21

before transformation

x =−1$\begin{bmatrix}1\\0\end{bmatrix}$+2$\begin{bmatrix}0\\1\end{bmatrix}$=$\begin{bmatrix}-1\\2\end{bmatrix}$

Transform the space →Reset

So if a matrix linearly transforms a vector how should we think about two matrices multiplying. Take two transformations $A$ and $B$. The product $AB$ applied to a vector $x$ means: apply $B$ first, then apply $A$ on top — short for $A(Bx)$. $AB$ is the single matrix that does both transformations in one step. This explains why we need non linearities in neural networks:

• A linear layer $Wx + b$ is a matrix function applied to its input.
• Stack three linear layers and you get $W_3 W_2 W_1 x$ a single effective matrix. You could just compact them into a single matrix.
• That’s why nonlinearities exist. Drop a ReLU between each pair and they stop collapsing into one matmul; only then does depth grow expressivity.

A

2001

×

B

0-110

=

AB

0-210

step 0 of 3

Two matrices, two transformations. Click Step to apply B to the plane.

Step →▶ Auto-playReset

Take a down-projection for example:

$$W_{\text{down}} = \begin{bmatrix} \vert & & \vert\\ q_1 & \cdots & q_k\\ \vert & & \vert \end{bmatrix} \in \mathbb{R}^{d \times k} \qquad xW_{\text{down}} = \begin{bmatrix} x \cdot q_1 & \cdots & x \cdot q_k \end{bmatrix}$$

Each column $q_j$ does its own dot product against the vector. We can view this as asking the vector a learned questions: how strongly does $x$ align with this column? The answer becomes coordinate $j$ of the new vector. A down-projection from $d$ dimensions to $k$ dimensions is therefore a bank of $k$ dot products.

input x

2-13102

1 × 6

×

q1q2q3

10101-11000-11-10001-1

6 × 3

=

answers z

???

1 × 3

three learned questions

Click Step to ask the input the first question.

Step →▶ Auto-playReset

Dot Product

The dot product measures direction alignment, scaled by magnitude. Large and positive when two vectors point the same way, zero when they’re perpendicular, negative when they point opposite ways.

Mechanically: $a \cdot b = \sum_i a_i b_i$.
Geometrically: $a \cdot b = |a||b|\cos\theta$, where $\theta$ is the angle between them.

Note $a\cdot b$ and $a^\top b$ are two notations for the same operation. Transposing $a$ turns it into a row, so the row-vector times column-vector multiplication produces the scalar $\sum_i a_i b_i$.

drag the tip of b · aligned

a · b = |a| · |b| · cos θ

0.00 =0.00 ·0.00 · cos(0°)

Reset

Just like matrix multiplication this operation shows up everywhere in ML:

• Matmul is dot product in bulk. Each entry of $AB$ is a dot product of a row of $A$ with a column of $B$.
• Attention scores. In $\text{softmax}(QK^\top / \sqrt{d})$, each entry of $QK^\top$ is a dot product between a query and a key “how aligned is this token’s question with that token’s offering?”
• Embedding similarity. Cosine similarity is the dot product after normalizing to unit length: $\cos\theta = \frac{a \cdot b}{|a||b|}$. The standard test for “are these two embeddings semantically close?”

Projection and Orthogonal Decomposition

A dot product measures alignment. Projection uses that measurement to extract the part of a vector that lies along a chosen direction. Suppose $x$ contains many directions mixed together, but we only care about one direction $u$. When $u$ is a unit vector, $\|u\|_2=1$, the dot product $u\cdot x$ is the signed amount of $x$ pointing along $u$. It is only a scalar. Multiplying that amount by $u$ turns it back into a vector:

$$\operatorname{proj}_u(x)=(u\cdot x)u$$

This is the projection of $x$ onto $u$: the part of $x$ that can be represented using only that direction. Projection splits $x$ into two pieces:

$$x = \underbrace{\operatorname{proj}_u(x)}_{\text{explained by }u} + \underbrace{\left(x-\operatorname{proj}_u(x)\right)}_{\text{unexplained residual}}.$$

The residual is orthogonal to $u$. We already extracted everything that points along $u$, so the remainder cannot contain any more of that direction. This split into an explained component and a perpendicular residual is an orthogonal decomposition.

start with the vector x

x = proj_u(x) + (x − proj_u(x))

Replay

Norms (L1, L2, Lp)

Ever wondered “how big is this vector?” Well machine learning engineers sure have: weight decay, gradient clipping, cosine similarity. There are multiple ways to measure, and the one you pick changes what your model does.

• L2 (Euclidean): $\|x\|_2 = \sqrt{\sum_i x_i^2}$. Straight-line distance. The default: weight decay, gradient clipping.
• L1 (Manhattan): $\|x\|_1 = \sum_i |x_i|$. Sum of absolute values. The norm you pick when you want sparsity.
• L∞ (max): $\|x\|_\infty = \max_i |x_i|$. The single largest coordinate. The standard for adversarial perturbations (“change every pixel by at most $\epsilon$”).

Why does adding L1 norm to the loss produces sparsity? I like to picture every norm as a unit ball. That’s the set of vectors with $\|x\| = 1$. Its shape is what makes different norms behave differently. Minimizing a loss subject to a norm budget pushes the solution toward the surface of the unit ball. L1’s diamond has corners pointing at the axes and a corner is sparse. In a 5 dimensions space a corner coordinate may be (1, 0, 0, 0, 0). L2’s circle has no corners, so its generically decreases the vector size. That geometric fact is the reason L1 regularization zeros weights out while L2 just shrinks them.

L₂ · circle

‖x‖_p = 1  ·  p = 2.00

L1L2L∞

Eigenvalues and eigenvectors

A matrix usually rotates and stretches vectors at the same time. But for certain special directions, it does only stretching no twist. Those directions are the matrix’s eigenvectors, and the scaling factors are its eigenvalues:

$$A v = \lambda v$$

• Eigenvector $v$ - vector that $A$ maps to a scaled copy of itself. The direction goes in, the same direction comes out.
• Eigenvalue $\lambda$ - the scalar that says how much $A$ stretches that direction.

For an $n \times n$ matrix you typically get $n$ such pairs: the natural axes of the transformation.

drag x · find the directions where Ax doesn't rotate ·searching

A = ((3, 1), (1, 3))  ·  angle(x, Ax) = 0.0°

Reset

Eigen Values are certainly more niche than dot product or norms. But when you pull out an eigen value reference in front of your machine learning friends at the white board they will be impressed. Here are some places you might use them:

• Principal component analysis (the standard technique for compressing high-dimensional data). The eigenvectors of the data covariance $\Sigma = X^\top X / n$ are the directions of maximum variance in your dataset; the eigenvalues tell you how much variance lies along each. Keep the top $k$ and you’ve projected your data onto its $k$ most informative directions.
• Eigenvalues of the Hessian tell you curvature at a critical point. All positive → minimum. All negative → maximum. Mixed signs → saddle (the failure mode that ate early deep learning).
• Condition number $\kappa = \lambda_{\max}/\lambda_{\min}$. Big $\kappa$ → gradients pull in wildly different scales across directions → optimization is slow.

Singular Value Decomposition (SVD)

It’s great that we can visualize matrices by how they transform space. The problem is most matrices are just a pile of numbers making it impossible to understand. But remember the Matrix Multiplication as Composition section? Matrices $AB$ are the same as applying transformation A then B. We can leverage this for matrix decomposition where we split any matrix into three separate matrices that rotate, scale then rotate.

$$A = U \Sigma V^\top$$

1. Rotate the input by $V^\top$.
2. Scale each axis by its singular value $\sigma_i$ — the diagonal entries of $\Sigma$, always real and non-negative.
3. Rotate the output by $U$.

The singular values are the matrix’s stretch factors: how much it lengthens or shrinks vectors along its preferred directions. Mathematicians design SVD such that the singular values tell you a lot about the matrix you are dealing with. A practical example would be in matrix approximation. The singular values magnitudes rank the importance of those transformation directions… keep the directions with large singular values and discard those with small ones. For a more in depth explanation check out this channel.

V^T =

0.000.000.000.00

Σ =

0.00000.00

U =

0.000.000.000.00

StartAfter V^TAfter ΣAfter U

Notation snag: $\Sigma$ here is just the capital Greek letter sigma, used as the name of a diagonal matrix not a summation. Also people usually make the leftmost column $\sigma_1$ the largest singular value, $\sigma_2$ the second largest and so forth. Since we can choose $V$ and $U$ we choose ones that work with this ordering. In three dimensions, it literally looks like this:

$$\Sigma = \begin{bmatrix} \sigma_1 & 0 & 0 \\ 0 & \sigma_2 & 0 \\ 0 & 0 & \sigma_3 \end{bmatrix}.$$

Another way to look at SVD is

• Columns of $V$ are the right singular vectors and each $v_i$ the input direction
• Diagonal entries of $\Sigma$ are the singular values and each $\sigma_i$ says how strongly $A$ transmits $v_i$
• Columns of $U$ are the left singular vectors and $u_i$ the the output direction $v_i$ becomes

If you feed the original matrix $A$ its $i$-th right singular vector $v_i$, it produces the corresponding left singular vector $u_i$, scaled by the singular value $\sigma_i$.

$$Av_i=\sigma_i u_i$$

From this view it makes a lot of sense why the rank of the transformation $A$ is the count of the non-zero singular values. Also note U and V the columns are of unit length 1, and each column is perpendicular to each other.

Rank, null space, column space

Three linear algebra jargon that describe what information a matrix preserves vs. destroys:

• Rank of $A$ — the number of dimensions in the output of $A$‘s linear transformation. For example if a matrix $A$ transforms all vectors via $Ax$ onto a plane we would say it has rank 2. Because math connects in bunch of ways you could also say the rank is the number of linearly independent columns (equivalently: rows) in $A$. In SVD terms, the count of non-zero singular values.
• Column space — the set of all possible outputs of $Ax$ as $x$ varies. Also the span of $A$‘s columns. A subspace of dimension equal to the rank.
• Null space — the set of inputs $x$ such that $Ax = 0$. The directions the matrix completely forgets.

In machine learning low-rank structure is everywhere. Trained neural network weight matrices are often empirically close to low rank… most of the useful signal lives in a small number of directions. This observation underwrites weight pruning, LoRA (forcing updates to be rank-$r$), and the broader claim that over parameterized models effectively use a far smaller subspace than their parameter count suggests.

Null space means the data cannot uniquely determine the parameters. If some features are redundant, different parameters can produce exactly the same predictions. The null space contains the parameter changes that the model’s predictions cannot “see” as the loss would be the exact same. L2 regularization resolves this ambiguity by preferring smaller parameters, preventing the parameters from drifting arbitrarily far while making the same predictions.

rank 2 — full plane preserved

V^T =

0.290.96-0.960.29

Σ =

3.26001.84

U =

0.47-0.880.880.47

σ₂

Review · spaced repetition

Linear Algebra — key ideas

What is a vector?

An ordered collection of scalars. Geometrically, it can represent a point or a direction in a vector space.

Show answerHide answer

What is a matrix?

A rectangular array of scalars. Geometrically a matrix is a linear transformation. A function that takes a vector and can rotate, stretch, shear, reflect, or collapse it.

Show answerHide answer

What makes a transformation $T$ linear?

It preserves weighted sums:

$$T(au+bv)=aT(u)+bT(v).$$

Show answerHide answer

What is a norm $\|x\|$?

A function that measures a vector's size or length. It is nonnegative, scales with $|\alpha|$, and obeys the triangle inequality.

Show answerHide answer

What is the span of a set of vectors?

The set of every vector obtainable as a linear combination of them.

Show answerHide answer

When are vectors $v_1,\ldots,v_k$ linearly independent?

When $a_1v_1+\cdots+a_kv_k=0$ implies $a_1=\cdots=a_k=0$. None of the vectors can be constructed from the others.

Show answerHide answer

What is a basis of a vector space?

A linearly independent set of vectors that spans the entire space.

Show answerHide answer

What is the column space of a matrix $A$?

The span of $A$'s columns, equivalently the set of every possible output $Ax$.

Show answerHide answer

What is the null space of a matrix $A$?

The set of inputs mapped to zero: $$\operatorname{Null}(A)=\{x:Ax=0\}.$$ These are the directions whose information $A$ discards.

Show answerHide answer

Why is linearity useful for feature representations in neural networks?

A linear layer transforms each feature direction consistently, and combines several features by adding their contributions: $W(au + bv) = aWu + bWv$. This lets learned feature directions be reused across many input combinations.

Show answerHide answer

What does column $j$ of a matrix $W$ represent?

It's where the $j$-th basis vector lands under the transformation $W$. The full matrix is just a list of these destinations — and $Wx$ is the weighted sum of the columns using $x$'s entries as weights.

Show answerHide answer

What does the matrix product $AB$ represent?

The single matrix that performs $B$ first, then $A$: $(AB)x = A(Bx)$. Stacked linear transformations collapse into one matrix-vector multiply — which is why stacking linear layers without nonlinearities buys you no expressive power, since $W_3 W_2 W_1 x$ is just one effective matrix.

Show answerHide answer

What does the dot product $a \cdot b = |a||b|\cos\theta$ measure?

Direction alignment, scaled by magnitude. Large positive when $a$ and $b$ point the same way, zero when perpendicular, large negative when opposite. Magnitudes amplify the result; direction is the signal.

Show answerHide answer

Why does L1 regularization produce sparse weights, while L2 just shrinks them?

The L1 unit ball is a diamond with corners on the axes. When optimization hits the constraint boundary, the corners — where most parameters are exactly zero — are the most likely contact points. The L2 ball is a circle with no corners, so its solutions are generic (no axes preferred).

Show answerHide answer

What is an eigenvector of a matrix $A$?

A nonzero vector $v$ that $A$ maps to a scaled copy of itself — same direction in, same direction out. The scaling factor is the eigenvalue $\lambda$, defined by:

$$Av = \lambda v$$

Show answerHide answer

What does SVD say about an arbitrary matrix $A$?

Every matrix factors as $A = U\Sigma V^\top$, where $U$ and $V$ are rotations and $\Sigma$ is diagonal with the singular values $\sigma_i \geq 0$. The transformation is rotate-scale-rotate, generalized to any matrix (the input rotation $V$ and the output rotation $U$ are allowed to be different).

Show answerHide answer

How do singular values $\sigma_i$ generalize eigenvalues, and what's their geometric meaning?

Two cleanups: always real, always non-negative. Geometrically, a matrix sends the unit sphere to an ellipsoid, and the singular values are the lengths of that ellipsoid's semi-axes.

Show answerHide answer

What is the rank of a matrix?

The number of linearly independent columns (equivalently, rows). In SVD terms: the count of non-zero singular values. Rank tells you how many output directions actually carry information — the rest are redundant or crushed to zero.

Show answerHide answer

Calculus & Optimization Math

Notation used in this section+

$f: \mathbb{R}^n \to \mathbb{R}^m$ — function taking $n$-dimensional input, returning $m$-dimensional output

$\frac{\partial f}{\partial x_i}$ — partial derivative: slope of $f$ along axis $i$, all other variables fixed

$\nabla f(\mathbf{x})$ — gradient at $\mathbf{x}=(x_1,\dots,x_n)$: the vector of all partials $\left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)$

$J_f \in \mathbb{R}^{m \times n}$ — Jacobian: the $m \times n$ matrix of all partials for a vector-valued function; row $i$ is the gradient of output $i$

$H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$ — Hessian entry; $H$ is the full $n \times n$ matrix of second partials

$\kappa = \lambda_{\max}/\lambda_{\min}$ — condition number; large $\kappa$ means the surface has directions of very different curvature

$\varepsilon$ — a small perturbation (a nudge), not machine epsilon

Derivatives and Partial Derivatives

Calculus enters ML through one question: which way should I nudge my weights to decrease the loss? If we think of the Neural Net as a giant function with many, many parameters the derivate answers this question by giving us the best constant approximation of the slope of the loss at the neural nets current parameters.

It’s helpful to think of $dx$ as a tiny nudge given to the input variable $x$ and $df$ as the resulting tiny change in the output of the function $f(x)$. The derivative $\frac{df}{dx}$ is the scaling factor that tells you how many times larger the output change is compared to your input nudge. In machine learning land you would think about $\frac{dL}{dw}$ or how nudging the weights will move the loss.

$$f'(t) = \lim_{dt \to 0} \frac{f(t+dt)-f(t)}{dt}$$

Derivative as a limit

f′(0.00)≈f(0.00 + 0.00) − f(0.00)0.00=0.00

as dt → 0, the secant slope approaches f′(0.00) = 0.00

f(x) = 1.4·sin(x)

secant slope = 0.00 →  tangent slope = 0.00 ·  |error| = 0.00

tdt

Partial derivatives add one rule: hold every other variable fixed.

$$\frac{\partial f}{\partial x_i}(\mathbf{x}) = \lim_{h \to 0} \frac{f(x_1, \dots, x_i + h, \dots, x_n) - f(x_1, \dots, x_n)}{h}$$

A partial is the slope of $f$ along one axis, with every other axis frozen. For $f(x_1, \dots, x_n)$ you get $n$ such slopes one per parameter and they’re the components of the gradient. In training, those inputs are weights and the function is the loss, so each partial answers “if I nudge only $w_i$, how does $L$ move”? The catch is that each partial is myopic: it assumes every other weight stays still. In practice you move them all at once, and the gradient is just the vector of these one-knob-at-a-time answers. Interactions between weights live at the second-derivative level, in the Hessian.

The gradient is the vector of all the partials. During gradient descent we compute the gradient (aka weight nudges) for each example in the batch and then average them.:

$$\nabla f(\mathbf{x}) = \left( \frac{\partial f}{\partial x_1}(\mathbf{x}),\ \dots,\ \frac{\partial f}{\partial x_n}(\mathbf{x}) \right)$$

Loss surfacef(x, y) = 0.5·x² + 1.5·y²

gradientconstant loss

Drag across the surface to move the point

The gradient is the steepest uphill direction; gradient descent follows its negative

(x, y) = (0.00, 0.00)∇f = (0.00, 0.00)‖∇f‖ = 0.00loss = 0.00

x

0.00

y

0.00

Note the jacobian is a fancy term for the matrix of all first-order partial derivatives of a vector-valued function $f: \mathbb{R}^n \to \mathbb{R}^m$. In our case $n$ is the number of parameters, $m$ is a scalar the loss and we get a $n \times 1$ column of partials aka the gradient aka the nudges to the weights.

$$J_f = \begin{pmatrix} \partial f_1/\partial x_1 & \cdots & \partial f_1/\partial x_n \\ \vdots & \ddots & \vdots \\ \partial f_m/\partial x_1 & \cdots & \partial f_m/\partial x_n \end{pmatrix}$$

Chain rule

Neural networks are a composition of many functions and to differentiate a composition $h(x) = f(g(x))$:

$$h'(x) = f'(g(x)) \cdot g'(x)$$

Multiply the local derivatives of each piece, but note where each one is evaluated: $g'$ at the original input $x$, $f'$ at the value $g(x)$ that $f$ actually received. Each derivative is evaluated at whatever its function saw on the forward pass, which is why every ML framework stores activations during the forward pass, backprop needs them to evaluate each local derivative at the right point.

g(x) = 0.6·x²

g(x)0.00g′(x)0.00

f(u) = sin(u)

f(g(x))0.00f′(g(x))0.00

h(x) = sin(0.6·x²)

h(x)0.00h′(x)0.00

chain rule in action

h′(x) =f′(g(x))·g′(x)=0.00×0.00=0.00

x =

0.00

Autodiff on a tiny neural network

Here is the chain rule in the form an autodiff library such as JAX actually encounters it. This network has one hidden neuron, a ReLU, and a squared-error loss:

import jax
import jax.numpy as jnp

def loss_fn(params, x, y):
    z = params["w1"] * x + params["b1"]
    h = jax.nn.relu(z)
    y_hat = params["w2"] * h + params["b2"]
    return 0.5 * (y_hat - y) ** 2

params = {
    "w1": 1.0,
    "b1": 0.0,
    "w2": 0.5,
    "b2": 0.0,
}

loss, grads = jax.value_and_grad(loss_fn)(params, 2.0, 0.5)

On the forward pass, JAX starts with the supplied example and runs the ordinary computation, retaining the intermediate values needed to differentiate it. The complete forward calculation is

$$\begin{aligned} x&=2,\qquad y=0.5,\\ z&=2,\qquad h=\operatorname{ReLU}(z)=2,\qquad \hat y=1,\qquad L=\frac12(\hat y-y)^2=0.125. \end{aligned}$$

Because the output is a scalar loss, JAX then applies reverse-mode autodiff and moves right to left through the recorded operations. Each operation receives the derivative accumulated from later in the graph, multiplies it by its own local derivative, and passes the result farther backward.

JAX reverse-mode traceloss, grads = jax.value_and_grad(loss_fn)(params, x, y)

forwardbackward

inputx = 2.00supplied example

→

affine₁z = 2.00w₁ = 1.00 · b₁ = 0.00∂L/∂w₁ = 0.50
∂L/∂b₁ = 0.25

→

ReLUh = 2.00ReLU′(z) = 1∂L/∂z = 0.25

→

affine₂ŷ = 1.00w₂ = 0.50 · b₂ = 0.00∂L/∂w₂ = 1.00
∂L/∂b₂ = 0.50

→

squared errorL = 0.125target y = 0.50∂L/∂ŷ = 0.50

gradient pytree{ w1: 0.50, b1: 0.25, w2: 1.00, b2: 0.50 }

ready

Click Step to run the same scalar loss function JAX transforms.

ReLU stateActiveDead

Step →▶ Auto-playResetstep 0 of 10

The result has the same pytree structure as params, so an optimizer can pair each parameter with its update directly:

grads = {
    "w1": 0.50,
    "b1": 0.25,
    "w2": 1.00,
    "b2": 0.50,
}

JAX does not need to materialize the full Jacobian of the network. Each primitive implements a local reverse rule (a vector-Jacobian product) and composing those rules performs the backward pass efficiently. For a scalar loss, one right-to-left sweep produces the gradient with respect to every parameter. This is the mechanism behind jax.grad and jax.value_and_grad.

Hessian

The gradient is the first derivative. The Hessian is the second, the matrix of all second partials of a scalar function $f: \mathbb{R}^n \to \mathbb{R}$:

$$H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$$

For a neural network with $n$ scalar parameters, the Hessian is an $n \times n$ matrix. Each entry measures how the gradient for parameter $\theta_i$ changes as parameter $\theta_j$ changes. Where the gradient gives the best linear approximation of $f$ near a point, the Hessian gives the best quadratic one: the curvature of the loss surface, how the slope itself changes as you move. The gradient tells you which way is downhill; the Hessian tells you whether you’re in a bowl, on a ridge, or somewhere in between.

Local approximationf(x) = 1.4·sin(x)

functionlinearquadratic

second-order approximation

f(a + ε) ≈ f(a) + f′(a)·ε + ½·f″(a)·ε²

curvature at a = 0.000.00linear error0.000quadratic error0.000

a

0.00

ε

0.00

For smooth $f$, because of some smart math proofs we know $H$ is symmetric, so we can apply the rotate scale rotate decomposition from the eigenvalue section: $H$‘s eigenvectors are the principal directions of curvature, its eigenvalues are the curvatures along them. At a critical point ($\nabla f = 0$), the signs classify what’s there:

• All positive → bowl. Local minimum.
• All negative → dome. Local maximum.
• Mixed signs → saddle. Surface goes up in some directions, down in others.

What the notation ∂²f / ∂xᵢ∂xⱼ actually means+

Read the denominator right-to-left: the variable closest to $f$ gets differentiated first.

$$\frac{\partial^2 f}{\partial x_i\, \partial x_j} \;=\; \frac{\partial}{\partial x_i}\!\left(\frac{\partial f}{\partial x_j}\right)$$

It’s two ordinary partial derivatives stacked: first take $\partial f/\partial x_j$ (holding everything except $x_j$ fixed), giving you a new function; then take $\partial/\partial x_i$ of that (holding everything except $x_i$ fixed). The “freeze everything else” rule applies at each step independently.

Worked example. Take $f(x, y) = x^2 y + 3xy^2$, and compute $\partial^2 f / \partial x\, \partial y$ — that’s $y$ first, then $x$:

$$\frac{\partial f}{\partial y} = x^2 + 6xy \quad\Longrightarrow\quad \frac{\partial}{\partial x}\!\left(x^2 + 6xy\right) = 2x + 6y$$

Flip the order:

$$\frac{\partial f}{\partial x} = 2xy + 3y^2 \quad\Longrightarrow\quad \frac{\partial}{\partial y}\!\left(2xy + 3y^2\right) = 2x + 6y$$

Same answer. That’s Schwarz’s theorem in action — and exactly why $H_{ij} = H_{ji}$, making the Hessian symmetric.

Curvature mapf(x, y) = ½·(λ₁·x² + λ₂·y²)

positivenegative

eigenvalue signs classify the critical point

HessianH = diag(0.00, 0.00)critical point at the originminimum

λ₁

0.00

λ₂

0.00

MinSaddleMaxDegenerate

Saddles dominate high-dimensional loss landscapes. In a million-dimensional loss, a true local minimum needs all million eigenvalues to come out positive, vanishingly unlikely if their signs are even somewhat random. Saddles, which need only one eigenvalue of either sign, are way more numerous. Pre 2014, deep learning was supposed to be impossible because the loss had too many bad local minima. The real obstacle was always saddles, and SGD’s noise turns out to escape them readily as long as the gradient picks up the negative curvature direction.

Conditioning and ill-conditioned optimization

Condition number = optimization difficulty. The ratio $\kappa = \lambda_{\max}/\lambda_{\min}$ of the Hessian’s eigenvalues at a minimum measures how stretched the bowl is. Small $\kappa$ → roughly circular, gradient descent walks straight in. Large $\kappa$ → long, narrow valley where gradient descent zig-zags.

Review · spaced repetition

Calculus & Optimization — key ideas

What is the Jacobian of $f:\mathbb{R}^n\to\mathbb{R}^m$?

The $m\times n$ matrix of first partial derivatives: $$J_{ij}=\frac{\partial f_i}{\partial x_j}.$$ It is the best local linear approximation of a vector-valued function.

Show answerHide answer

What is the directional derivative of $f$ at $x$ along a unit vector $u$?

The rate at which $f$ changes when moving from $x$ along $u$: $$D_uf(x)=\nabla f(x)^\top u.$$

Show answerHide answer

What is a critical point of a differentiable scalar function?

A point where the gradient is zero: $$\nabla f(x)=0.$$ It may be a local minimum, local maximum, or saddle.

Show answerHide answer

What is gradient descent?

An iterative optimization method that moves parameters opposite the gradient: $$\theta_{t+1}=\theta_t-\eta\nabla_\theta L(\theta_t).$$

Show answerHide answer

What is automatic differentiation?

Computing derivatives of a program by decomposing it into elementary operations and applying the chain rule through the resulting computation graph.

Show answerHide answer

What is the geometric meaning of the derivative $f'(t)$?

The slope of the best linear approximation of $f$ near $t$. For a small step $dt$:

$$f(t + dt) \approx f(t) + f'(t)\,dt$$

Zoom in close enough on any smooth curve and it looks like a straight line — $f'(t)$ is that line's slope. Gradient descent is this move once per step; backprop is this move chained through a computation graph.

Show answerHide answer

What does the partial derivative $\partial f / \partial x_i$ measure, and what's its key restriction?

The slope of $f$ along the $x_i$ axis, with every other variable frozen at its current value. So a partial is myopic — it answers "if I nudge only $x_i$, how does $f$ move?" assuming everything else stays still. Interactions between variables live at the second-derivative level, in the Hessian.

Show answerHide answer

Of all unit directions you could move from a point $a$, which one increases $f$ fastest, and how fast?

The gradient $\nabla f(a)$ — the vector of all partials. Its direction is the steepest-ascent direction; its magnitude $\|\nabla f(a)\|$ is the rate of increase along it. That's the entire basis of gradient descent: to decrease a loss, step along $-\nabla L$.

Show answerHide answer

State the chain rule for $h(x) = f(g(x))$. Where is each piece evaluated?

$$h'(x) = f'(g(x)) \cdot g'(x)$$

$g'$ is evaluated at the original input $x$; $f'$ is evaluated at $g(x)$ — the value $f$ actually received on the forward pass. This is why every ML framework caches activations during the forward pass — backprop needs them to evaluate each local derivative at the right point.

Show answerHide answer

For a function $\mathbb{R}^n \to \mathbb{R}$ with $n$ in the millions, why does autodiff chain Jacobians right-to-left (from the loss back to the parameters) rather than left-to-right?

Right-to-left keeps every intermediate quantity as a vector (Jacobian-vector products with a scalar on the right), costing about one forward pass total. Left-to-right would keep each intermediate as a full Jacobian matrix, costing roughly $n\times$ more. Reverse-mode is cheap precisely because there's only one output.

Show answerHide answer

What does the Hessian $H_{ij} = \partial^2 f / \partial x_i\, \partial x_j$ represent geometrically?

The curvature of $f$ — how the slope itself changes as you move. Where the gradient is the best linear approximation of $f$ near a point, the Hessian gives the best quadratic one:

$$f(a + \varepsilon) \approx f(a) + \nabla f(a)^\top \varepsilon + \tfrac{1}{2}\,\varepsilon^\top H(a)\, \varepsilon$$

It tells you whether you're in a bowl, on a ridge, or somewhere in between.

Show answerHide answer

At a critical point ($\nabla f = 0$), how do the signs of the Hessian's eigenvalues classify it?

All positive → bowl. Local minimum.
All negative → dome. Local maximum.
Mixed signs → saddle. Surface goes up in some directions, down in others.

In high-dimensional loss landscapes, saddles dominate: a true minimum needs all eigenvalues positive (exponentially unlikely), while a saddle only needs one of either sign.

Show answerHide answer

What part of the Hessian does Adam approximate, and what does it miss?

Adam approximates the diagonal of $H$: $H_{ii} = \partial^2 L / \partial w_i^2$ — the curvature of the loss with respect to a single weight $w_i$, with no coupling to any other weight. The estimate comes from an EMA of squared gradients ($\mathbb{E}[g_i^2]$ is the Fisher diagonal, $\approx H_{ii}$ for likelihood losses), and each parameter's step is scaled by $1/\sqrt{H_{ii}}$. Adam misses all off-diagonal entries $H_{ij}$ — how nudging $w_j$ shifts the slope along $w_i$ — so weight coupling is invisible to it.

Show answerHide answer

Probability

Notation used in this section+

$X$, $Y$ — random variables (capital letters); $x$, $y$ — specific values they take (lowercase)

$P(X = x)$ — probability mass: how often discrete variable $X$ equals $x$

$p(x)$ — probability density: not a probability itself; integrate over a range to get probability

$p(x, y)$ — joint distribution over two variables simultaneously

$p(y \mid x)$ — conditional distribution of $Y$ given $X = x$

$\mathbb{E}[X]$ — expectation (probability-weighted average). The subscript in $\mathbb{E}_p[f(X)]$ names the distribution: draw $X \sim p$, evaluate $f$, average — i.e. $\int f(x)\,p(x)\,dx$. The same $p$ appears as both the subscript label and the density inside the integral.

$\text{Var}(X)$ — variance; $\sigma = \sqrt{\text{Var}(X)}$ is the standard deviation

$\mathcal{N}(\mu, \sigma^2)$ — Gaussian with mean $\mu$ and variance $\sigma^2$; multivariate version uses covariance matrix $\Sigma$

$x \sim p$ — ”$x$ is sampled from distribution $p$”

$X \perp Y \mid Z$ — $X$ and $Y$ are conditionally independent given $Z$

Random variables, PMFs / PDFs

A random variable is a number you read off some random process - a die roll, a noisy sensor reading, the loss on a random minibatch. The random variable’s distribution maps possible outcomes to their probability of occuring represented as a number between 0 and 1.

A random variable has two flavours depending on if it takes discrete or continuous values:

• Probability Mass Function for discrete Random Variables. $P(X = x)$ is the probability of each specific value, and the values sum to 1.
• Probability Density Function for continuous Random Variables. $p(x)$ is a density its area under a range gives the probability of landing in that range, and the total area under $p$ is 1.

PMFs and PDFs are different mathemtical objects that you can’t really just exchange for one another. For a continuous random variable the probability that $x$ equals exactly 0.81485… is 0, you can always make the value more specific. It’s a bit confusing because in ML you constantly see continuous distributions $p(x)$ but this is a probability density not a probability itself. For PDFs you have to consider ranges $P(0.8 < x < 0.85)$. You think of the area under the curve not the height of the curve as the probability and to find the area under a curve you integrate. Once I internalized this distinction probability clicked a lot for me.

P(a ≤ X ≤ b)=∫_a^bp(x) dx=

0.683

A note on notation. Capital $X$ is the random variable itself, the unrealized abstraction. Lowercase $x$ is a specific value it might take. So $P(X = x)$ reads “the probability that the random variable $X$ comes out equal to the specific value $x$.”

Expectation and Variance

$\mathbb{E}[X]$ of a random variable is a tries to capture the center of the random variable’s distribution and can be interpreted as the long-run average of many independent samples from the given distribution:

$$\mathbb{E}[X] = \sum_x x \, P(X = x) \qquad \text{or} \qquad \mathbb{E}[X] = \int x \, p(x) \, dx$$

$\text{Var}(X)$ of a random variable quantifies the spread of that random variable’s distribution:

$$\text{Var}(X) = \mathbb{E}\!\left[(X - \mathbb{E}[X])^2\right]$$

Squaring does two things: deviations above and below the mean don’t cancel, and large deviations are penalized more than small ones. The standard deviation $\sigma = \sqrt{\text{Var}(X)}$ converts back to the original units.

Expectation = — · Standard deviation = — · Samples = 0 · Running average = —

Skew2.5Spread0.85

Draw samplesClear samples

Joint, marginal, conditional distributions

A joint distribution $p(x, y)$ tells you how often each combination of values shows up, not just how often $X = 3$ or $Y = 5$ separately, but how often the two happen together. It’s a 2D distribution or higher with more variables.

Marginal distribution: sum out the variable you don’t care about. You’re projecting the 2D joint down onto one axis. $p(x)$ is what the distribution of $X$ would look like if you’d never bothered tracking $Y$ at all.

$$p(x) = \sum_y p(x, y) \qquad \text{or} \qquad p(x) = \int p(x, y)\, dy$$

Conditional distribution: slice the joint at a specific value, then renormalize. The distribution of $Y$ now that you know $X$.

$$p(y \mid x) = \frac{p(x, y)}{p(x)}$$

xy

density p(y)

density p(x)

marginal

conditional

conditional distribution · 𝒩(mean, variance)

Y | X = 0.00 ~  𝒩(0.00, 0.00)

ρ0.60

Covariance and Correlation

Knowing the distribution of each variable separately does not tell us how they relate. Imagine a dataset containing each person’s height and weight. If we randomly shuffle the weights between people, the distribution of heights is unchanged and the distribution of weights is unchanged, but the relationship between height and weight has mostly disappeared. Covariance measures the part that the two marginal distributions miss: whether deviations in one variable systematically accompany deviations in the other. For each person, subtract the average height and average weight to get two signed deviations. Multiplying them gives a positive value when the person is above average in both variables or below average in both, and a negative value when the deviations point in opposite directions; covariance is the average of those products.

$$\begin{aligned} \operatorname{Cov}(X,Y) &= \mathbb{E}\left[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])\right] \\ &= \frac{1}{n}\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) \end{aligned}$$

The scale depends on the units of $X$ and $Y$, which makes raw covariance hard to interpret across different features. Correlation fixes that by measuring both deviations in standard deviations:

$$\rho(X,Y) = \frac{\operatorname{Cov}(X,Y)} {\sigma_X\sigma_Y} = \mathbb{E}\left[ \frac{X-\mu_X}{\sigma_X} \frac{Y-\mu_Y}{\sigma_Y} \right]$$

• This normalization constrains correlation to the interval $[-1,1]$
• A correlation of $1$ means the standardized variables move in perfect lockstep: whenever one is a certain number of standard deviations above or below its mean, so is the other
• A correlation of $-1$ means they move in exactly opposite directions
• A correlation near zero means there is little linear alignment between them

The word linear matters. Zero correlation does not imply that two variables are independent. If $Y=X^2$ and $X$ is symmetrically distributed around zero, knowing $X$ determines $Y$ completely, yet their covariance can be zero: positive and negative values of $X$ cancel when multiplied by $X^2$. Covariance and correlation detect straight-line structure, not every possible kind of dependence.

ρ = (1/n) Σᵢ zₓᵢzᵧᵢ

agreement +0.00+opposition −0.00=ρ +0.70

opposite−1+1together

The covariance matrix $\Sigma$ is something you will run across and generalizes this to $d$ random variables at once. Entry $\Sigma_{ij} = \text{Cov}(X_i, X_j)$; the diagonal is just the variances.

Common distributions

It’s good to know some of the common distributions that show up over and over again, especially the gaussian.

←

Bernoulli1 of 5

→

X ∼ Bernoulli(p)

probability p0.70

A distribution over one binary outcome: X = 1 with probability p, otherwise X = 0. A sigmoid converts a model logit into p; the Bernoulli negative log-likelihood is binary cross-entropy, used for binary and multi-label classification.

X ∼ Categorical(p₁, …, pₖ)

temperature T1.0

A distribution over one of K mutually exclusive classes, whose probabilities sum to one. Softmax turns model logits into categorical probabilities; its negative log-likelihood is multiclass cross-entropy, used for classification and next-token prediction.

X ∼ 𝒩(μ, σ²)

center μ0.0spread σ1.0

A continuous distribution centered at μ, with uncertainty controlled by σ. Assuming Gaussian prediction errors makes negative log-likelihood equivalent to mean squared error; Gaussians also provide initialization noise, VAE priors, and the noise used by diffusion models.

X ∼ Poisson(λ)

event rate λ4.0

A distribution over nonnegative counts when events occur independently at average rate λ. Poisson negative log-likelihood is used for count regression, event-frequency prediction, and as the building block of Poisson point-process models.

P ∼ Beta(α, β)

evidence for success α2.0evidence for failure β5.0

A distribution over an unknown probability P between zero and one; α and β encode evidence for success and failure. It is the conjugate prior for a Bernoulli probability, so Bayesian updating just adds observed successes and failures—useful for calibrated uncertainty and bandits.

Central Limit Theorem

The Central Limit Theorem states that the sample mean of a sufficiently large number of independent identically distributied random variables is approximately normally distributed. The larger the sample, the better the approximation. Its a two part operation:

1. Sample a batch and take their average
2. As you take more and more batch averages you see that the averages are normally distributed

Central Limit Theorem comes up constantly in ML and it is the reason why Gaussians also appear everywhere. Any quantity that’s the sum or average of many independent effects ends up Gaussian.

$$\frac{1}{N}\sum_{i=1}^N X_i \;\xrightarrow{N \to \infty}\; \mathcal{N}\!\left(\mu,\, \frac{\sigma^2}{N}\right)$$

Two things to notice: averaging $N$ samples leaves the mean unchanged, while the variance shrinks by $1/N$, so the standard deviation shrinks by $1/\sqrt{N}$. Four times as many samples therefore gives half the spread. Repeating the experiment across $D$ batches gives us more draws from this same sampling distribution; it does not narrow the distribution, whereas increasing $N$ does. Minibatch gradients are a classic example. A batch gradient averages $B$ per-example gradients, so the CLT suggests $\hat{g}_B \approx \mathcal{N}(g^*, \Sigma/B)$, where $g^*$ is the true gradient. Doubling $B$ halves the gradient-noise variance and reduces its standard deviation by $1/\sqrt{2}$; quadrupling $B$ halves the standard deviation. This cleaner gradient estimate is one reason larger batches can often support larger learning rates, although the exact scaling depends on the training regime.

each draw turns 4 values into one mean

100 means collected · latest x̄ = 0.286 · predicted σ_x̄ = 0.160 / √4= 0.080

sample size N4draws D100

Run new draws

Bayes’ rule

Bayes’ rule answers a specific question: you know how to compute $P(D \mid H)$ the probability of seeing your data given a hypothesis. But what you want is the reverse: given the data you actually observed, how probable is each hypothesis? That’s $P(H \mid D)$, and Bayes’ rule is how you flip the arrow:

$$P(H \mid D) = \frac{P(D \mid H)\, P(H)}{P(D)}$$

• $P(H)$ — prior: your belief about $H$ before seeing any data.
• $P(D \mid H)$ — likelihood: how probable is this data if $H$ were true.
• $P(H \mid D)$ — posterior: your updated belief after seeing the data. The prior reshaped by evidence.

In ML, $H$ is the model parameters $\theta$ and $D$ is the training data. The three terms translate directly to things you already know:

• Likelihood $P(D \mid \theta)$ — how well do these weights fit the training data? This is your loss function in disguise: low loss = high likelihood.
• Prior $P(\theta)$ — what weight values do you expect before seeing any data? A prior that prefers small weights is L2 regularization.
• Posterior $P(\theta \mid D)$ — the full distribution over weight settings after training.

$P(D)$ doesn’t depend on $\theta$ (a constant that gets ignored in the gradient) so to find the best $\theta$ you can ignore it.

Tak the log: maximizing $\log P(D \mid \theta) + \log P(\theta)$ is exactly loss + regularizer. When you train a network with weight decay, you are doing Bayesian MAP estimation — a Gaussian prior on the weights is L2, a Laplace prior is L1. Every regularizer is a prior in disguise.

Prior P(θ)Likelihood L(θ)Posterior P(θ|D)

Coin flips · true θ = 0.7

0 flips · 0 H · 0 T · posterior mode: —

FlipFlip ×10ResetPrior: UniformPrior: Skeptic (β(5,5))Prior: Tails (β(2,8))

Independence and conditional independence

A joint distribution over $n$ binary variables has $2^n$ cells. For a 28×28 image that’s $2^{784}$. No dataset fills it, and no model stores it. The whole history of generative modeling is the story of how to get around this.

Independence means knowing one variable tells you nothing about another — $P(A, B) = P(A)\,P(B)$. Conditional independence is subtler: $X$ and $Y$ may be correlated overall, but become independent once you fix $Z$, written $X \perp Y \mid Z$:

$$P(X, Y \mid Z) = P(X \mid Z)\, P(Y \mid Z)$$

All shared variation between $X$ and $Y$ flows through $Z$. Fix $Z$, nothing is left to share.

Early ML solved the tractability problem with explicit independence assumptions. Naive Bayes assumed each feature is independent given the class label, turning one enormous joint into a product of small marginals. N-gram language models assumed each word depends only on the previous $k$ words. Both assumptions are obviously wrong — but they made estimation possible from limited data, and the predictions were often good enough anyway.

Naive Bayes: worked example+

Take spam detection. You want to classify an email as spam or not. An email is a bag of $n$ words, each present or not — the full joint $p(w_1, \ldots, w_n \mid \text{spam})$ has $2^n$ entries. With a 10,000-word vocabulary that’s $2^{10000}$ cells. No dataset fills it.

Naive Bayes assumes each word is independent of every other word given the class:

$$p(w_1, \ldots, w_n \mid \text{spam}) = \prod_{i=1}^{n} p(w_i \mid \text{spam})$$

Now you just need one number per word: “how often does this word appear in spam?” Count frequencies in your training data and you’re done.

To classify a new email, you want $p(\text{spam} \mid \text{email})$. By Bayes’ rule, that’s proportional to $p(\text{email} \mid \text{spam})\,p(\text{spam})$ — so compare:

$$p(\text{spam}) \cdot \prod_i p(w_i \mid \text{spam}) \quad \text{vs} \quad p(\text{not spam}) \cdot \prod_i p(w_i \mid \text{not spam})$$

Pick the larger one. “Nigerian” and “prince” co-occur in spam far more than chance predicts, so the independence assumption is wrong — but the rank order between classes survives anyway. The genuinely likelier class still wins even with miscalibrated probabilities.

Modern generative models dropped the explicit assumptions. A transformer doesn’t assume any token is independent of any other — it attends to the full context. Instead it uses the chain rule, which is always valid:

$$p(x_1, \ldots, x_T) = \prod_t p(x_t \mid x_1, \ldots, x_{t-1})$$

Then approximates each conditional with a neural network. No hardcoded independence structure — just a flexible function trained on enough data to learn whatever conditional structure actually exists.

One catch: conditioning can create dependence, not just destroy it. Talent and luck are independent in the population. Filter to successful hires — conditioning on their shared effect — and they become negatively correlated. Knowing someone got lucky makes talent less necessary to explain their success. You manufactured a dependence that wasn’t there.

This is the collider trap: filter to high-quality training examples, evaluate only on accepted submissions, control for a mediator in a causal analysis — any time you condition on something downstream of two variables, you induce a spurious dependence between them. d-separation gives you the formal rules for when this happens.

Finally, the iid training assumption is also a conditional independence claim — each example contributes new information. Violate it (correlated sequences, duplicate data, patients from the same hospital) and your effective sample size is smaller than your dataset, and your generalization gap is larger than train loss suggests.

Monte Carlo Methods

Often in ML we don’t know a probability distribtuion be we can sample from it. The Monte Carlo methods are ways to learn about the unkown probability distribution through sampling.

Method	What access do we have?	What do we do?
Direct Monte Carlo	Can sample $x \sim p$	Average $f(x)$
Importance sampling	Can sample from $q$ and evaluate $p/q$	Draw from $q$, then reweight
MCMC	Can evaluate $p$ up to a constant, but cannot sample directly	Construct correlated draws whose long-run distribution is $p$

Note Bootstrapping is a related but different technique. In bootstrapping we have an observed dataset (we can’t just sample) and we typically create simulated datasets by sampling the original datasets with replacement. We then compute our statistic on each resampled dataset and see how much it varies. For example, repeatedly resample a test set, calculate model accuracy each time, and use the spread to estimate an accuracy confidence interval.

---

Direct Monte Carlo: estimate an expectation by taking a bunch of samples and averaging.

$$\mathbb{E}_p[f(X)] \approx \frac{1}{N} \sum_{i=1}^N f(x_i), \qquad x_i \sim p$$

• $p$ is the data distribution.
• $X$ is a randomly selected training example.
• $x_i$ is the $i$-th example in your minibatch.
• $f(x_i)$ is the model’s loss on that example.
• $\sigma_f$ measures how much the loss varies between examples. If some examples have tiny losses and others have huge losses, $\sigma_f$ is large, so you need a larger minibatch for a stable estimate.

Some interesting things to note

• This is a result of the Law of Large Numbers which says if you repeatedly sample from the same distribution, the sample average approaches the distribution’s true expected value
• The Central Limit Theorem quantifies how the number of samples determines how uncertain you answer is. The estimate has error $\sigma_f / \sqrt{N}$, where $\sigma_f$ is the standard deviation of $f(X)$ under $p$

The pain point with monte carlo methods is variance, when $\sigma_f$ is large, you need huge $N$. Policy gradient is a good example: rewards on rollouts are wildly variable, $\sigma_f$ is enormous, and a single REINFORCE update looks like pure noise. Baselines, GAE the whole variance-reduction wing of ML exists to shrink $\sigma_f$ so you can get away with fewer samples.

---

Importance sampling: reweight when you can’t sample from $p$.

Importance sampling is a Monte Carlo method used to approximate the properties or expected values of a target probability distribution p(x) by drawing samples from a different, more convenient proposal distribution q(x). Sample from $q$, multiply each by the importance weight $w(x) = p(x)/q(x)$, and you get an unbiased estimate of the expectation under $p$. Regions where $p \gg q$ get upweighted; regions where $p \ll q$ get downweighted. In math notation the reason this works is:

$$\mathbb{E}_p[f(X)] = \mathbb{E}_q\!\left[f(X)\cdot\frac{p(X)}{q(X)}\right]$$

μ_q0.00max weight6.0

SampleClear

PPO on an LLM is a canonical example where you have many samplers generating long rollouts. When you take a gradient step with your policy $\pi_\theta$ you don’t pause training to send the new weights to the samplers, and restart all the in progress rollouts. Instead you accept many of your rollouts are from $\pi_{\theta_\text{old}}$ and reweight using importance sampling.

The danger is when if an unlikely $q(x)=0.01$ is sampled in a $p(x)=0.8$ the importance weight can explode to 80 in this case. You generally want $q$ to fully cover and have strong support for all regions of $p$ and avoid $q$ having lighter tails than $p$. In PPO, this is exactly what goes wrong when $\pi_\theta$ drifts far from $\pi_{\theta_\text{old}}$: some $r_t$ blow up and training explodes. PPO’s fix is to clip $r_t$ to $[1-\varepsilon, 1+\varepsilon]$ introducing a small bias in exchange for dramatically lower variance.

---

Markov Chain Monte Carlo: when no tractable proposal $q$ works.

If we can’t sample from $p$ and there is no alternative $q$, but we can check the probability of $p(x)$ for whatever $x$ we can come up with then Markov Chain Monte Carlo can help. Image generators are a good example as the image space is so vast and the distribution so complex that no clever proposal $q$ from the previous section will ever cover it. MCMC’s answer is to instead start with an image $x$ of random noise and wander through the space step-by-step, biased toward higher-probability regions by assessing $p(x)$ the probability of generating that image. Over time the positions you visit look like samples from $p$.

The textbook algorithm is Metropolis-Hastings. From the current position $x$:

1. Take a small random step from $x$ e.g., add a Gaussian nudge to get a candidate $x'$.
2. Compare densities. If $p(x') \geq p(x)$ (the candidate is at least as likely), accept the move.
3. Otherwise accept with probability $p(x') / p(x)$ the smaller the drop, the more often you still take the step.
4. If you reject, stay at $x$. Repeat.

The third rule is key, as over time, MCMC visits each region in proportion to its probability under $p$. Once we obtain samples from MCMC we can utilize Direct Monte Carlo from earlier.

Metropolis-Hastings chain on a bimodal target

steps = 0 · accept rate = — · time in left mode = —

step size0.32mode gap1.60

Random walk Langevin (gradient-aware)

PlayReset

Review · spaced repetition

Probability — key ideas

What is a random variable?

A function that maps each outcome of a random process to a value, usually a number.

Show answerHide answer

What is a probability distribution?

A rule assigning probabilities to the possible values or events of a random variable.

Show answerHide answer

What is a joint distribution $p(x,y)$?

A distribution assigning probabilities or densities to combinations of values taken by multiple random variables.

Show answerHide answer

When are random variables $X$ and $Y$ independent?

When their joint distribution factorizes: $$p(x,y)=p(x)p(y).$$ Knowing one does not change the distribution of the other.

Show answerHide answer

What does $X\perp Y\mid Z$ mean?

Once $Z$ is known, $X$ and $Y$ are independent: $$p(x,y\mid z)=p(x\mid z)p(y\mid z).$$

Show answerHide answer

What is the standard deviation of a random variable $X$?

The square root of its variance: $$\sigma_X=\sqrt{\operatorname{Var}(X)}.$$ It measures spread in the same units as $X$.

Show answerHide answer

What is the covariance of $X$ and $Y$?

$$\operatorname{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}X)(Y-\mathbb{E}Y)].$$ It measures their unnormalized linear co-variation.

Show answerHide answer

What is a Bernoulli distribution?

A distribution over a binary outcome $X\in\{0,1\}$, determined by one parameter: $$P(X=1)=p,\qquad P(X=0)=1-p.$$

Show answerHide answer

What is a categorical distribution?

A distribution over one of $k$ discrete outcomes, specified by probabilities $p_1,\ldots,p_k$ that sum to $1$.

Show answerHide answer

What is the support of a probability distribution?

The set of values to which the distribution assigns nonzero probability or density.

Show answerHide answer

What is a Markov chain?

A sequence of random states in which the next state's distribution depends only on the current state: $$p(x_{t+1}\mid x_t,\ldots,x_0)=p(x_{t+1}\mid x_t).$$

Show answerHide answer

What's the difference between a PMF $P(X=x)$ and a PDF $p(x)$?

PMF (discrete): $P(X=x)$ is the actual probability of getting value $x$; values sum to 1.

PDF (continuous): $p(x)$ is a density, not a probability. Integrate over a range to get probability: $P(a \leq X \leq b) = \int_a^b p(x)\,dx$. Total area is 1, but $p(x)$ itself can exceed 1 at any point.

Show answerHide answer

What does it mean to normalize an unnormalized distribution $\tilde{p}(x)$, and what is the partition function $Z$?

Divide by the total so it integrates to 1: $p(x) = \tilde{p}(x) / Z$, where $Z = \int \tilde{p}(x)\,dx$.

$Z$ is the partition function (a.k.a. normalizing constant). Often $\tilde{p}$ is cheap but $Z$ is intractable — energy-based models with $\tilde{p}(x) = e^{-E(x)}$ are the canonical case. MCMC and self-normalized IS exist to sample without ever computing $Z$.

Show answerHide answer

What does it mean to marginalize out a variable from a joint distribution $p(x, y)$?

Sum (or integrate) over the variable you don't care about: $p(x) = \sum_y p(x, y)$ for discrete, or $p(x) = \int p(x, y)\,dy$ for continuous.

It collapses a 2D distribution into a 1D one along the axis you kept. Conceptually: ignore $y$, just tell me how often each $x$ happens.

Show answerHide answer

How is a conditional distribution $p(y \mid x)$ defined in terms of the joint?

$$p(y \mid x) = \frac{p(x, y)}{p(x)}$$Slice the joint at $X = x$ (a 1D strip), then renormalize so it sums to 1. The conditional is just the joint, restricted to one value of $X$ and rescaled.

Show answerHide answer

What does Bayes' rule look like in terms of weights $\theta$ and data $\mathcal{D}$, and how does each term map to ML?

$P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta)\, P(\theta)$.

Likelihood $P(\mathcal{D} \mid \theta)$ — how well these weights fit the data (= negative loss, up to a log).
Prior $P(\theta)$ — what weights you expect before training (e.g., a Gaussian prior = L2 regularization).
Posterior $P(\theta \mid \mathcal{D})$ — distribution over weights after training.

Show answerHide answer

Why is training a neural network with weight decay equivalent to Bayesian MAP estimation?

Taking the log of $P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta)\, P(\theta)$ and finding the peak gives $\hat{\theta} = \arg\max_\theta [\log P(\mathcal{D}|\theta) + \log P(\theta)]$ — that's negative loss + log-prior. A Gaussian prior on $\theta$ contributes a quadratic penalty, exactly L2 weight decay. A Laplace prior gives L1. Every regularizer is a prior in disguise.

Show answerHide answer

What is the linearity of expectation, and why is it surprisingly powerful?

$\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$ — regardless of whether $X$ and $Y$ are independent.

That's the surprising part. Variance only adds for independent variables: $\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$ requires independence. Linearity of expectation is why minibatch gradients are unbiased estimates of the true gradient — you can swap $\mathbb{E}$ and $\sum$ without caring about correlations between samples.

Show answerHide answer

Give two equivalent formulas for $\text{Var}(X)$, and explain why one is more useful in practice.

$\text{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - \mu^2$, where $\mu = \mathbb{E}[X]$.

The second form (mean of square minus square of mean) only requires running sums of $X$ and $X^2$, so you can compute variance in one pass over the data without first finding the mean. It's how numpy.var and online statistics estimators work.

Show answerHide answer

For a vector random variable $X \in \mathbb{R}^d$, what is the covariance matrix $\Sigma$ and what do its entries mean?

$\Sigma = \mathbb{E}[(X - \mu)(X - \mu)^\top]$, a $d \times d$ matrix. $\Sigma_{ii} = \text{Var}(X_i)$ — variance of each component (diagonal). $\Sigma_{ij} = \text{Cov}(X_i, X_j)$ — pairwise covariances (off-diagonal).

$\Sigma$ is symmetric and positive semi-definite. Its eigenvectors are the principal axes of the distribution; eigenvalues are the variances along those axes — exactly what PCA computes.

Show answerHide answer

State the Central Limit Theorem and one place it shows up in ML.

If $X_1, \dots, X_n$ are iid with mean $\mu$ and finite variance $\sigma^2$, then for large $n$: $\frac{1}{n}\sum_i X_i \approx \mathcal{N}(\mu, \sigma^2/n)$. Regardless of the original distribution. Standard error shrinks as $\sigma / \sqrt{n}$.

In ML: a minibatch gradient is a sample average of per-example gradients, so by the CLT it's approximately Gaussian noise around the true gradient — doubling the batch size halves the noise standard deviation.

Show answerHide answer

What is a Monte Carlo estimator for $\mathbb{E}_p[f(X)]$, and why doesn't its error depend on the dimension of $X$?

Draw $N$ i.i.d. samples $x_i \sim p$ and average: $\hat{\mu} = \frac{1}{N}\sum_i f(x_i)$. The estimator is unbiased with standard error $\sigma_f/\sqrt{N}$ — a rate that depends only on the variance of $f$ under $p$, not on the dimension. Averaging works the same whether $X$ is a scalar or a billion-parameter vector.

Show answerHide answer

What is an importance weight $w(x) = p(x)/q(x)$, and what does it correct for?

It corrects for sampling from the wrong distribution. If you draw $x_i \sim q$ instead of $p$, the IS estimator $\frac{1}{N}\sum_i w(x_i)\,f(x_i)$ is still unbiased for $\mathbb{E}_p[f]$. Regions where $p \gg q$ are upweighted (sampled too rarely); regions where $p \ll q$ are downweighted (sampled too often).

Show answerHide answer

When do importance sampling weights have catastrophic variance, and what does a good proposal $q$ look like?

When $q$ has lighter tails than $p$: rare high-$p$ regions get enormous weights, making the estimator spiky. In the extreme, the estimator's variance is infinite. A good $q$: (1) covers the full support of $p$, (2) has heavier tails than $p$, (3) concentrates mass where $|f(x)|\,p(x)$ is large. The ideal is $q^*(x) \propto |f(x)|\,p(x)$.

Show answerHide answer

How does PPO use importance sampling, and why does it clip the weights?

PPO estimates the policy gradient using data from an older policy $\pi_{\theta_{\text{old}}}$. The Importance Sampling weight is the probability ratio $r_t = \pi_\theta(a_t|s_t)/\pi_{\theta_{\text{old}}}(a_t|s_t)$. If $\pi_\theta$ drifts far from $\pi_{\theta_{\text{old}}}$, some $r_t$ blow up and variance explodes. PPO clips $r_t$ to $[1-\varepsilon, 1+\varepsilon]$ — a small bias for dramatically lower variance and stable training.

Show answerHide answer

You want samples from a complex distribution $p$ — say a generative model's distribution over images — but you can't sample directly. How does Metropolis-Hastings work?

Wander through the space, biased toward higher-probability regions. From the current point $x$:

1. Take a small random step (e.g., add a Gaussian nudge) to get a candidate $x'$.
2. If $p(x') \geq p(x)$, accept — uphill always.
3. Otherwise accept with probability $p(x')/p(x)$ — the smaller the drop, the more often you still take the step.
4. If you reject, stay at $x$. Repeat.

Only the ratio $p(x')/p(x)$ shows up — the normalizing constant cancels, so you can sample from a distribution whose total mass you can't compute.

Show answerHide answer

Information Theory

Information

Notation used in this section+

$x$ — a particular message or outcome; $X$ is the random variable that produces it

$p(x)$ — the true probability that outcome $x$ occurs

$q(x)$ — a model’s assigned probability for outcome $x$

$c(x)$ — the codeword assigned to message $x$: the actual bit string sent for that message

$\ell(x)$ — the code length of message $x$: the number of bits in its codeword

A codeword is the bit string assigned to one message; its code length is simply the number of bits in that string. With four equally likely movement commands, a straightforward fixed-length code is:

Message	Codeword	Code length
up	00	2 bits
down	01	2 bits
left	10	2 bits
right	11	2 bits

$I(x) = -\log_2 p(x)$ — information content (or surprisal) of one outcome, in bits

$H(p)$ — entropy: ideal average code length for data drawn from $p$

$H(p, q)$ — cross-entropy: average code length when data comes from $p$ but the code is based on $q$

$D_{\mathrm{KL}}(p \Vert q)$ — extra average code length from using $q$ instead of $p$

What does it mean for an observation to contain information? Not “is it interesting?” and not “how many characters did we write down?” In information theory, the question is sharper: how much uncertainty did seeing this particular outcome remove? “The sun rose this morning” may matter enormously, but if it was nearly certain, observing it communicates little new information.

Compression gives this information concept a physical meaning. If sender and receiver agree on a distribution over messages, they can compress the messages by giving likely messages short bit strings and unlikely messages long ones. The number of bits an outcome deserves is its information content.

Information of one event

Rarer events rule out more possibilities.

p(event)0.14→2.84bits

14%event

information

2.84 bits

event probability p

0.14

The core formula for information falls out when you realize that you cannot compress random noise. Imagine receiving an $n$-bit message whose bits are independent fair coin flips. There are $2^n$ possible strings of that length, all equally likely, so any particular one occurs with probability

$$\frac{1}{2^n} = 2^{-n}.$$

There is no regularity to exploit: knowing the first $n-1$ bits tells you nothing about the last one. A lossless compressor cannot make this uniform collection of messages shorter on average. In other words, an ideal compressor should look like random noise. It has removed every predictable pattern it can.

If a message has probability $p$ and an ideal code assigns it $n$ bits, then $p = 2^{-n}$. Solving for $n$ gives

$$I(x) = -\log_2 p(x) = \log_2\frac{1}{p(x)}$$

We call $I(x)$ the information content (or surprisal) of outcome $x$, measured in bits. For a sequence, the probability of the complete message is the product of the probabilities of its successive symbols, each conditioned on the preceding context:

$$p(x_{1:T}) = \prod_{t=1}^{T}p(x_t \mid x_{<t}).$$

Taking $-log_2$ turns that multiplication into addition. So the information in a whole message is its sequence of per-symbol information, added together:

$$I(x_{1:T}) = -\log_2 p(x_{1:T}) = \sum_{t=1}^{T}-\log_2 p(x_t \mid x_{<t}).$$

Information in a sequence

A message’s probability multiplies; its information adds.

revealed total0.00bits

I(message) = −log₂ P(message)=Σ_t −log₂ P(x_t | x_<t)

Ip = 0.0504.32 bNp = 0.2701.89 bFp = 0.0126.38 bOp = 0.4101.29 bRp = 0.9600.06 bMp = 0.9400.09 bAp = 0.7200.47 bTp = 0.9800.03 bIp = 0.9500.07 bOp = 0.9600.06 bNp = 0.9500.07 b p = 0.8900.17 bTp = 0.0564.16 bHp = 0.2002.32 bEp = 0.2502.00 bOp = 0.0175.88 bRp = 0.9200.12 bYp = 0.6700.58 b

4.3

1.9

6.4

1.3

0.1

0.1

0.5

0.0

0.1

0.1

0.1

0.2

4.2

2.3

2.0

5.9

0.1

0.6

Reveal next characterReset

Illustrative conditional probabilities from a toy character model—not universal letter frequencies. Predictable characters cost almost nothing; the surprising ones carry the message.

Entropy

Before a sample arrives, we do not know which outcome will occur; we only know the distribution $p(x)$. So the natural question is: how many bits will we need on average? Weight each outcome’s information by its chance of occurring:

$$H(X) = \mathbb{E}_{x \sim p}\left[-\log_2 p(x)\right] = -\sum_x p(x)\log_2 p(x).$$

This is Shannon entropy. It is the average surprisal of a draw from $p$, and more operationally the theoretical lower limit on the average number of bits per symbol needed to losslessly encode independent samples from that distribution.

In ML classification labels, token sequences, and latent variables are all draws from distributions. Entropy tells you the compression limit. For example, the compression limit of code is lower than books because of the boilerplate and repeatable patterns in code.

Entropy is average information

Each rectangle’s area is probability × surprisal.

H(X)2.00bits

four outcomescoin

H(X) = −Σ p(x) log₂ p(x)

probability of A

0.25

Move toward an even split to make the next observation harder to predict.

Cross-entropy

Entropy was the ideal case: your encoding matches reality. A message that occurs with probability $p(x)$ gets an ideal code length of $-\log_2 p(x)$ bits. Averaging over messages gives the irreducible compression limit aka entropy.

$$H(p) = \mathbb{E}_{x \sim p}[-\log_2 p(x)].$$

Cross-entropy changes one thing: your compressor is built using a model $q$, which may be wrong. “If the world draws messages from distribution $p$, but I compress them using beliefs $q$, how many bits per message will I spend on average?”

$$H(p,q) = \mathbb{E}_{x \sim p}[-\log_2 q(x)] = -\sum_x p(x)\log_2 q(x)$$

The price of a wrong prediction

Reality draws from p; your codebook is built from q.

latest sample—

True source p

A · 0.50

B · 0.25

C · 0.15

D · 0.10

samples
from →

Your model q

A · —

B · —

C · —

D · —

ideal code—

your code—

extra on this event—

H(p)—unavoidable bits

H(p, q)—bits used by q

KL(p || q)—extra bits on average

Match q to pUnderweight A

Sample a message

Try the two codebooks. When $q=p$, the average extra-bit bucket is empty; a mismatched $q$ makes it fill.

There is one practical wrinkle: ML libraries almost always use natural logs, so cross-entropy loss is reported in nats, not bits. Optimizing either gives the same model. Bits are better for intuition; nats are conventional because calculus with $e^x$ is convenient.

KL divergence

Cross-entropy measures the total average code length, and entropy the best possible code length. KL divergence is the difference between them: the extra bits per message from encoding $p$-data with $q$.

$$H(p,q) = H(p) + D_{\mathrm{KL}}(p \Vert q).$$

Entropy is the unavoidable uncertainty in the data; the KL term is the extra code length from a wrong model.

• $H(p)$: the best possible average code length
• $D_{\mathrm{KL}}(p \Vert q)$: extra bits paid because $q$ does not match $p$
• $H(p, q)$: total bits your particular model needs

KL divergence is not symmetric. $D_{\mathrm{KL}}(p \Vert q)$ means “data comes from $p$; I encode it using $q$.” Reversing the arguments changes both the data source and the codebook.

$$D_{\mathrm{KL}}(p\Vert q) = H(p,q) - H(p) = \sum_x p(x)\log_2\frac{p(x)}{q(x)}.$$

Where the extra bits come from

Each outcome can help or hurt; the total mismatch cost cannot be negative.

D_KL(p || q)—bits

True source p

A

—

B

—

C

—

D

—

Model q

A

—

B

—

C

—

D

—

Per-outcome contribution: p(x) log₂(p(x)/q(x))zero line

A

—

B

—

C

—

D

—

probability of A under p

0.50

probability of A under q

0.10

Swap p and qMatch q to p

Perplexity

Perplexity is just cross-entropy converted back out of logarithmic units. Cross-entropy tells you the model’s average surprise in bits (or nats in most ML libraries). Perplexity exponentiates that quantity.

$$\mathrm{PPL}(p,q) = 2^{H(p,q)}$$

where

$$H(p,q) = \mathbb{E}_{x \sim p}[-\log_2 q(x)].$$

The best intuition is that perplexity is the effective number of plausible choices. If the model is equally uncertain among four next tokens, it assigns each probability $1/4$. Its cross-entropy is $-\log_2(1/4) = 2$ bits and its perplexity is $2^2 = 4$. So the model is, in effect, choosing among four equally plausible continuations.

A few important caveats:

• Lower is better. It means the model assigned more probability to the tokens that actually appeared.
• Perplexity depends on tokenization. A character-level model and a BPE-token model cannot compare perplexities directly; one predicts many more, smaller units.
• It is not a general intelligence score. Perplexity measures predictive/compression performance on a particular distribution. A model can have excellent perplexity yet still be bad at reasoning tasks, tool use, or following instructions.
• It is especially natural for language modelling. It is less commonly reported for classification because “effective number of classes” is often less informative than accuracy, calibration, or log loss.

Review · spaced repetition

Information Theory — key ideas

In information theory, what does it mean for an observation to contain information?

Information is not “is it interesting?” and not “how many characters did we write down?” It is how much uncertainty seeing this particular outcome removed.

“The sun rose this morning” may matter enormously, but if it was nearly certain, observing it communicates little new information.

Show answerHide answer

If an outcome $x$ has probability $p(x)$, what is its ideal binary code length (its information)?

$$I(x) = -\log_2 p(x) = \log_2\frac{1}{p(x)}$$bits.

More probable outcomes get shorter codes; an outcome with probability $1/8$ costs $3$ bits.

Show answerHide answer

What does Shannon entropy $H(p)$ measure operationally?

$$H(p) = \mathbb{E}_{x \sim p}[-\log_2 p(x)].$$

It is the average information and the theoretical minimum average number of bits per independent draw needed for lossless compression when the encoder knows the true distribution $p$.

Show answerHide answer

In cross-entropy $H(p,q)$, what roles do $p$ and $q$ play?

$$H(p,q) = \mathbb{E}_{x \sim p}[-\log_2 q(x)].$$

$p$ is the distribution that produces the data (the internet); $q$ is the model used to assign probabilities (the llm).

So $p$ chooses which outcomes are averaged over, while $q$ determines the surprise/cost charged to each outcome.

Show answerHide answer

For one classification example with true class $y$, why is cross-entropy loss $-\log q(y)$?

The target distribution is one-hot: it puts probability $1$ on the observed class $y$ and $0$ on every other class.

Therefore:

$$H(p_{\text{one-hot}}, q) = -\sum_c p(c)\log q(c) = -\log q(y).$$

Training with cross-entropy therefore increases the probability assigned to the true label.

Show answerHide answer

How do entropy, cross-entropy, and KL divergence decompose in the compression view?

$$H(p,q) = H(p) + D_{\mathrm{KL}}(p \Vert q).$$

$H(p)$: the best possible average code length—the unavoidable uncertainty in the data.
$D_{\mathrm{KL}}(p \Vert q)$: extra bits paid because $q$ does not match $p$.
$H(p,q)$: total average code length using your particular model.

So KL divergence is the extra average code length from encoding $p$-data with $q$.

Show answerHide answer

How should you interpret perplexity in language modelling?

$$\mathrm{PPL} = 2^{H(p,q)}$$

when cross-entropy is measured in bits.

It is the model's effective number of equally plausible next-token choices. For example, $2$ bits of cross-entropy corresponds to perplexity $4$: roughly the uncertainty of choosing among four equally likely continuations.

Only compare perplexities computed with compatible tokenization and evaluation data.

Show answerHide answer

Statistics & Estimation

Notation used in this section+

$\mu$ — population mean: the true average of the whole population

$\sigma$ — population standard deviation: the true spread of individual values around $\mu$

$\bar{x}$ — sample mean: the average of the values we observed

$s$ — sample standard deviation: our estimate of $\sigma$ from the observed sample

$n$ or $N$ — number of observations in a sample or dataset

$\hat\mu$ — an estimate of $\mu$; the hat means “estimated from data”

$\operatorname{SE}(\hat\mu)$ — standard error: how much $\hat\mu$ would vary across repeated samples

$\mathcal D$ — an observed dataset

$p(x\mid\theta)$ — probability or density assigned to $x$ given parameters $\theta$

$\mathcal L(\theta)$ — likelihood: how well the parameter setting $\theta$ explains the observed data

$\mathbb E[X]$ — expected value: the probability-weighted average of $X$

$\operatorname{Var}(X)$ — variance: the average squared distance of $X$ from its mean

Expected loss and Empirical Estimates

Ideally, we would evaluate the model on every example it might encounter in the real world and calculate its average loss. For a language model, it is the average prediction error across all text the model might encounter not merely the text in its training set. Learning theory calls this quantity the population risk

$$R(\theta) = \mathbb{E}_{(x,y)\sim p_{\text{data}}} [\ell_\theta(x,y)].$$

We cannot compute this directly because we do not possess every possible example. Instead, we average the loss over a finite dataset. This dataset average is the empirical risk our measurable estimate of the expected loss on the real world.

$$\hat R_{\mathcal D}(\theta) = \frac{1}{N} \sum_{i=1}^{N} \ell_\theta(x_i,y_i).$$

During training, even that average is expensive, so each update uses a smaller random minibatch:

$$\boxed{ \text{Real world} \;\longrightarrow\; \text{Dataset} \;\longrightarrow\; \text{Minibatch} }$$

Likelihood and Maximum Likelihood Estimation (MLE)

In MLE, you choose a probability distribution such as a Gaussian or a model such as a neural network and then simply choose the parameters that best fit the samples you have. Thats the whole idea. Put another way MLE chooses the parameter setting under which the data we observed looks least surprising. Suppose we record the maximum squat of everyone at a gym and model those weights with a Gaussian distribution. The Gaussian has two parameters: $\mu$ controls the typical squat weight, and $\sigma$ controls how much lifters vary around it. For any choice of $\mu$ and $\sigma$, the Gaussian assigns a density to every observed squat. In stats likelihood has a specific definition that asks: under these parameters, how plausible is the dataset we actually observed?

$$\mathcal L(\mu,\sigma) = \prod_{i=1}^{N}p(x_i\mid\mu,\sigma).$$

Maximum Likelihood Estimation chooses the parameters that maximize this value:

$$(\hat\mu,\hat\sigma) = \arg\max_{\mu,\sigma}\mathcal L(\mu,\sigma).$$

How well does this Gaussian explain every observed squat?

μ = 235 lb·σ = 70 lb·likelihood = —

Center μ235 lbSpread σ70 lbFind the MLE

Maximum a Posteriori (MAP)

MLE chooses the parameters that best explain the observed data. MAP does the same thing, but also considers what parameter values were plausible before seeing the data. Suppose we observe only three people at a gym squatting 315, 335, and 350 lb. MLE would estimate the gym’s typical squat near their average but perhaps we happened to sample the powerlifting team. From experience with similar gyms, we may have a prior belief that the typical squat is closer to 225 lb.

Let $\mu$ represent the gym’s unknown true average maximum squat, the center of the Gaussian for all its lifters. We compare possible values of $\mu$. For simplicity, assume the spread $\sigma$ is already known and estimate only $\mu$. Bayes’ rule combines our prior with the evidence from the three observed lifters:

$$p(\mu\mid\mathcal D) \propto p(\mathcal D\mid\mu)\,p(\mu).$$

• $\mathcal D = \{315,335,350\}$ is the squat data we observed.
• $p(\mathcal D\mid\mu)$ is the likelihood: if the gym’s true average were $\mu$, how plausible would it be to observe these three squats?
• $p(\mu)$ is the prior: before sampling anyone, how plausible did we think each possible gym average was?
• $p(\mu\mid\mathcal D)$ is the posterior: after seeing the three squats, how plausible is each possible gym average now?

MAP chooses the peak of that posterior:

$$\hat\mu_{\text{MAP}} = \arg\max_\mu p(\mathcal D\mid\mu)p(\mu).$$

With little data, the prior pulls the estimate toward 225 lb. As we observe more lifters, the likelihood becomes sharper and the data overwhelms the prior. MAP approaches MLE.

posterior ∝ likelihood × prior

prior = 225 lb·MLE = 333 lb·MAP = 284 lb

Observed lifters3 / 40Add evidence

This connects directly to regularization in neural networks. A Gaussian prior that prefers small weights contributes an L2 penalty. The data says “fit the examples”; the prior says “prefer simpler, smaller parameters unless the data strongly demands otherwise.”

$$\text{MAP objective} = \text{negative log-likelihood} + \lambda\|\theta\|_2^2.$$

Bias, Variance, and Consistency of Estimators

An estimator is a rule that uses a sample to guess an unknown quantity. For example, we might estimate a gym’s true average maximum squat using the average of ten observed lifters. If we repeatedly sampled ten different lifters, we would get a slightly different estimate each time. The pattern across those repeated estimates tells us whether the estimator is good:

• Bias: Are its estimates centered on the true value, or systematically too high or low?
• Variance: How much do its estimates change from one sample to another?
• Consistency: As the sample grows, does the estimator converge to the true value?

Imagine the gym’s true average squat is 275 lb. Estimates clustered around 300 lb have high bias but low variance. Estimates scattered from 200 to 350 lb have low bias but high variance if their center is still 275 lb. Estimates tightly clustered around 275 lb have both low bias and low variance.

These two errors combine:

$$\mathbb E[(\hat\mu-\mu)^2] = \operatorname{Bias}(\hat\mu)^2 + \operatorname{Var}(\hat\mu).$$

An estimator can deliberately accept some bias to reduce variance. MAP does exactly this: its prior pulls estimates away from the sample mean, introducing bias, but can make them much more stable when data is scarce.

prior 225

true mean 275

Sample meandata only

center —· bias —· SD —

±1 SD

bias

MAP estimatedata + prior

center —· bias —· SD —

±1 SD

bias

200225250275300325350

estimated average squat (lb)

Lifters in each sample5

less datamore data

Replay estimates

Standard Error, Confidence Intervals, and The Bootstrap

A dataset gives you one estimate. A different dataset would give you a slightly different estimate. Standard error measures how much that estimate would wobble across repeated datasets. For the average squat,

$$\operatorname{SE}(\bar{x}) =\frac{s}{\sqrt{n}}, \qquad s=\sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2}.$$

Standard deviation and standard error describe the spread of two different things:

• Standard deviation: How much individual observations vary
• Standard error: How much an estimate would vary if you repeatedly collected new datasets

Here, $s$ is the sample standard deviation of the individual squat weights: $x_i$ is one lifter’s squat, $\bar{x}$ is their average, and $n$ is the number of lifters. More samples leaves the observations’ standard deviation roughly unchanged, but more samples reduces the mean’s standard error at roughly $\frac{1}{\sqrt{n}}$.

A confidence interval turns that uncertainty into a range. A rough 95% interval is

$$\text{estimate}\ \pm\ 2\,\operatorname{SE}.$$

“95% confidence” means that if we repeatedly collected data and built intervals this way, about 95% of them would contain the true value. But $\operatorname{SE}(\bar{x})=s/\sqrt n$ is only for the mean. Other ML statistics, such as F1, calibration error, or the performance difference between two models, do not have a simple standard-error formula, so we instead use the bootstrap to approximate repeated data collection using the dataset we already have:

1. Sample $n$ examples from the dataset with replacement
2. Compute the statistic
3. Repeat many times

The standard deviation of those bootstrap estimates gives the standard error; their central 95% gives a confidence interval.

Observed dataset12 lifters

215230245255265275280290300315330345

resample with replacement

Current bootstrap sampleduplicates are expected

————————————

0 dataset remixes → 0 estimates

current mean = —·standard error = —·95% interval = —

Draw one resampleRun 250Reset

Bayesian and Frequentist Framing

Statiticians have been locked in metal warfare for centuries on how to frame their field Bayesians vs Frequentists. This probably doesn’t matter much for an ML engineer, but it is good context. The cleanest difference is what is allowed to vary. Suppose $\mu$ is the gym’s unknown true average squat.

• Frequentist: $\mu$ is fixed; the dataset is random. Imagine repeatedly sampling lifters and ask how often your method finds the truth.
• Bayesian: the dataset is fixed; our uncertainty is over $\mu$. Combine the observed squats with a prior to obtain $p(\mu\mid\mathcal D)$.

This changes the meaning of a 95% interval:

• A 95% confidence interval comes from a procedure that would contain the fixed $\mu$ in 95% of repeated datasets.
• A 95% credible interval contains 95% of the posterior probability for $\mu$, given the observed data and model.

Frequentistparameter fixed · datasets vary

Bayesiandataset fixed · parameter uncertain

fixed data:255275285295305

Repeated-sampling guarantee20/20 intervals shown · 19 contain μ

Belief after seeing this dataset95% posterior probability: 244–304 lb

Replay repeated samples

The Bayesian statement is more direct: “given our assumptions and this data, there is 95% posterior probability that $\mu$ lies here.” But it depends on the prior and model being reasonable. The frequentist statement is less direct, but gives a repeated-experiment guarantee without assigning probabilities to the fixed parameter.

In ML, frequentist tools such as confidence intervals and the bootstrap ask whether a reported improvement would survive a new test sample. Bayesian methods are useful when we want a distribution over parameters or predictions, especially when prior knowledge matters or data is scarce. With enough data and compatible assumptions, the two often give similar answers.

Training Loss, Generalization, and Calibration

Training loss measures how well a model fits examples it has already seen. Generalization asks how well it performs on new examples from the same distribution. That is why we evaluate on held out data. The difference between held out loss and training loss is the generalization gap:

$$\text{generalization gap} = \text{held-out loss} - \text{training loss}$$

Calibration asks a different question: when the model says “80% confident,” is it correct about 80% of the time?

$$P(\text{correct}\mid\text{confidence}=p)\approx p.$$

A model can have good accuracy but poor calibration. If it gets 8 of 10 predictions right while claiming 95% confidence, its accuracy is still 80%, but it is overconfident.

Generalizationseen examples vs new examples

Calibrationconfidence vs observed accuracy

Epoch 20train loss —held-out loss —gap —

calibratedconfidence matches observed accuracy

Training epoch20

Model says 80%Model says 95%

Review · spaced repetition

Statistics & Estimation — definitions

What is the relationship between population risk and empirical risk?

Population risk is the model's expected loss across the real data distribution.

Empirical risk is the average loss on an observed dataset—our measurable estimate of population risk.

Show answerHide answer

What parameter setting does maximum likelihood estimation choose?

The parameters under which the observed dataset is most likely:

$$\hat\theta_{\mathrm{MLE}}=\arg\max_\theta p(\mathcal D\mid\theta).$$

Likelihood asks: “If these were the parameters, how well would they explain the data we observed?”

Show answerHide answer

How does MAP estimation differ from MLE?

MLE uses only the observed data:

$$\hat\theta_{\mathrm{MLE}}=\arg\max_\theta p(\mathcal D\mid\theta).$$

MAP also incorporates a prior:

$$\hat\theta_{\mathrm{MAP}}=\arg\max_\theta p(\mathcal D\mid\theta)p(\theta).$$

The prior matters most when data is scarce.

Show answerHide answer

Across repeated samples, what do estimator bias and variance measure?

Bias: whether the estimates are systematically centered above or below the truth.

Variance: how much the estimates change from one sample to another.

Show answerHide answer

What makes an estimator consistent?

As the sample size grows, the estimator converges to the true value:

$$\hat\theta_n \xrightarrow{p} \theta.$$

Consistency is an asymptotic guarantee; it does not guarantee a good estimate from a small sample.

Show answerHide answer

For an estimated mean, what is the difference between sample standard deviation and standard error?

The sample standard deviation measures variation among individual observations:

$$s=\sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2}.$$

The standard error measures how much the estimated mean would vary across repeated samples:

$$\operatorname{SE}(\bar{x})=\frac{s}{\sqrt n}.$$

Show answerHide answer

How does the bootstrap estimate uncertainty in a statistic?

Repeatedly sample $n$ observations with replacement from the observed dataset and recompute the statistic.

The standard deviation of the resulting estimates approximates the statistic's standard error.

Show answerHide answer

How do 95% confidence and credible intervals differ?

A 95% confidence interval comes from a procedure that would contain the fixed parameter in 95% of repeated datasets.

A 95% credible interval contains 95% of the posterior probability for the parameter, given the observed data, model, and prior.

Show answerHide answer

What is the generalization gap?

The difference between performance on new data and performance on the training data:

$$\text{generalization gap}=\text{held-out loss}-\text{training loss}.$$

A growing gap means the model is improving on seen examples without transferring that improvement to unseen examples.

Show answerHide answer

What does it mean for a model's probabilities to be calibrated?

Among predictions made with confidence $p$, approximately a fraction $p$ should be correct:

$$P(\text{correct}\mid\text{confidence}=p)\approx p.$$

For example, predictions labeled 80% confident should be correct about 80% of the time.

Show answerHide answer

Geometry & High-Dimensional Intuition

Curse of dimensionality

If each of $d$ features can take $m$ meaningfully different values, the number of possible combinations is $m^d$. Each added dimension multiplies the space by $m$. Unless the dataset grows just as quickly, training data is like stars in outerspace, small specs in a vast sea of nothing.

The counter of this idea that all data lives on a small sub manifold due to the physics of our universe. Take Tesla self driving which has used eight external cameras; modeling each as a $1280 \times 960$ RGB image gives roughly $29.5$ million values per instant and $256^{29{,}500{,}000}$ possible inputs. Yet most images it sees occupies only a thin, structured subset of that space.

The Square Root Scaling Rule

Imagine repeatedly flipping a coin and moving +1 if heads and -1 if tails. After 100 steps, you would usually not be 100 steps from the origin as the directions cancel. A typical final distance is around $\sqrt{100}=10$. This also works more generally even if the random variable is not centered. If the random variable has mean $\mu$, then

$$\sum_{i=1}^d X_i \approx d\mu \pm O(\sqrt d).$$

The sum grows like $d$, while the standard deviation grows only like $\sqrt d$. As we will see in the next section the relative difference, $d$ versus $\sqrt d$, is what causes concentration in high dimensions.

Attention Scaling

Square-root scaling shows up all over the place in ML. For example this is the reason why we divide by the $\sqrt{d_k}$ in attention. An attention score is a dot product:

$$q^\top k=\sum_{j=1}^{d_k}q_jk_j.$$

Since the query and key weights are roughly independent and centered their dot product is going to be mean 0 with a standard deviation $\sqrt{d_k}$ away from zero, just like our coin walk was. Therefore attention uses:

$$\frac{q^\top k}{\sqrt{d_k}}$$

to keep the logits at roughly the same scale as the head dimension grows. Without this scaling, wider query and key vectors produce increasingly large logits. Softmax then becomes extremely peaked, and its gradients can become very small.

Attention weights without scalingd_k = 4

typical magnitude|qᵀk| ≈ √d_k = 2

−160+16

bubble area = weight

keys ↓queries →

the

blue

creature

roamed

home

the

blue

creature

roamed

home

q: creature·k: blue·score —·weight —

Head dimension d_k

41664256

4

Neural Network Initialization

Neural network initialization also uses $1/\sqrt d$. A neuron computes

$$y=\sum_{j=1}^{d}W_jx_j.$$

If every weight had the same scale regardless of $d$, the output would grow like $\sqrt d$ as the layer became wider. To cancel that growth, initialize each weight with a scale proportional to $\frac{1}{\sqrt d}$. Xavier and He initialization are refined versions of this idea: choose weight variance so activations and gradients maintain a reasonable scale across layers.

Concentration of Measure

Concentration of measure is the phenomenon that in high dimensions, individual coordinates remain noisy, but aggregate quantities like a vector’s norm often become highly predictable, even though the space itself is enormous.

Square-root scaling describes the standard deviation when we add roughly independent terms. Concentration occurs when the standard deviation becomes small relative to the quantity’s overall scale. If the expected sum grows like $d$ while its standard deviation grows only like $\sqrt d$, then

$$\frac{\text{standard deviation}}{\text{expected sum}} \sim \frac{\sqrt d}{d} =\frac{1}{\sqrt d}\to 0.$$

In ML this shows up whenever we aggregate many roughly independent contributions. A high-dimensional vector has many coordinates, each of which can vary substantially. But quantities that combine all those coordinates, such as the vector’s norm, often vary surprisingly little.

Repeated samples · x ∼ N(0, I_d)

The coordinates stay noisy. The length becomes predictable.

typical radius√2 ≈ 1.41samples within ±10%

One sampled vectorEach coordinate remains unpredictable

160 independently sampled vectorsTheir normalized lengths gather around 1

dimensions d

2

Resample

The visual samples vectors $x\sim\mathcal N(0,I_d)$. Their individual coordinates keep jumping around as $d$ increases, but their lengths become steadily more predictable. That happens because the squared length is itself a sum:

$$\|x\|^2=x_1^2+\cdots+x_d^2\approx d,$$

so almost every vector has length $\|x\|\approx\sqrt d$. Geometrically, the samples collect in a thin shell at radius $\sqrt d$.

Here is the surprising part: the Gaussian density is highest at the origin, yet a high-dimensional Gaussian sample almost never lands near it. As we move away from the origin, each individual point becomes less likely, but the shell of available points becomes vastly larger. In high dimensions, the abundance of points wins: no particular point on the shell is especially likely, but a sample will almost certainly land somewhere on it.

This is the difference between the mode and the typical set. The mode is the single point with the highest density—the origin in this example. The typical set is the thin shell containing nearly all the samples. High dimensions separate “the most likely point” from “where samples are likely to be.”

For ML, the practical intuition is simple: high-dimensional randomness can be chaotic coordinate by coordinate while remaining highly structured in aggregate. Gaussian initialization and Gaussian noise produce unpredictable entries but a dependable overall scale. Norms and distances in many high-dimensional representations likewise cluster into narrow ranges.

Check out Sander’s excellent blog for a deeper dive.

Near-orthogonality in high dimensions

In high dimensions, two independent random directions are usually close to perpendicular.

To see why, imagine a simplified pair of random vectors whose parameters are either $+1$ or $-1$. In each coordinate, their signs either match, creating some alignment, or oppose each other, creating an equal amount of anti-alignment. Because the vectors are unrelated, both outcomes are equally likely.

Across many dimensions, roughly half the parameters match and half oppose, so their effects nearly cancel. For example, if $55$ of $100$ parameters match and $45$ oppose, $45$ pairs cancel completely, leaving only $10/100 = 0.1$ net alignment. They will not always balance perfectly, but the leftover imbalance becomes a smaller fraction of the whole as more dimensions are added. Real random vectors have parameters of varying sizes rather than just $\pm1$, but the same cancellation occurs and pushes their cosine similarity toward zero.

Coordinate-by-coordinate cancellation

Unrelated directions mostly undo their own alignment.

cosine similarity+0.100 means perpendicular

signs match: +1 signs oppose: −1

55 matchand45 oppose→45 pairs cancel, leaving 10 aligned

dimensions d

100

New random pairReplay cancellation

The leftover accidental alignment has a typical scale of $1/\sqrt d$, so in $10{,}000$ dimensions random cosine similarities are commonly around $\pm 0.01$. This gives embeddings room to encode many directions that interfere only weakly with one another. Learned embeddings are not truly random, but a cosine near zero is still a useful baseline for “no more aligned than chance.”

Why this matters for LLMs: the residual stream and superposition

At each token position, a transformer’s residual stream is one $d_{\text{model}}$ dimensional vector that acts like a shared workspace. Every attention head and MLP reads selected information from it through a linear projection, then writes its result back by addition. Information written by an early layer therefore remains available to later layers unless some later computation removes or overwrites it. This read/write view is developed in A Mathematical Framework for Transformer Circuits.

That one vector may need to simultaneously carry many facts about the current token: which entity is being discussed, its grammatical role, facts retrieved about it, the task being performed, and partial conclusions produced by earlier layers. The model can associate each feature with a direction and represent the current state as a sum:

$$r = a_1v_{\text{subject}} +a_2v_{\text{person}} +a_3v_{\text{sport}} +a_4v_{\text{basketball}} +\cdots$$

A later component can look for the basketball feature by projecting $r$ onto $v_{\text{basketball}}$. The basketball direction contributes strongly to that read, while unrelated feature directions contribute only their small accidental overlap. Near-orthogonality therefore lets the residual stream behave like many weakly interfering communication channels laid on top of one another. More complex information can occupy a multi-dimensional subspace rather than a single direction, allowing an attention head to read or write a structured bundle of features at once.

There is an important limit: a $d$-dimensional space can contain at most $d$ exactly orthogonal directions. But it can contain far more than $d$ approximately orthogonal directions. Models exploit this by assigning more features than dimensions and accepting some cross-talk between them, a compression strategy called superposition. This works because features are sparse: the model may know an enormous number of possible features, while only a small fraction are relevant to any particular token. When few are active, their interference remains manageable; if too many activate together, the noise accumulates and starts to obscure the signal. Toy Models of Superposition explores this trade-off directly.

This has several practical consequences:

• Features are not the same as neurons. A meaningful concept may be a direction spread across many parameters, while a single neuron can participate in many different features. Sparse autoencoders try to undo this compression by expanding a dense residual vector into a much larger space where only a few interpretable features are active.
• Wider models can form cleaner representations. More residual-stream dimensions give important features more room to separate, reducing interference. A narrower model may know many of the same features but represent them less reliably.
• Many computations can coexist. Different heads and MLPs can read and write different directions or subspaces of the same residual stream, allowing syntactic, semantic, factual, and task-related information to move through the model in parallel.

Review · spaced repetition

Geometry & High-Dimensional Intuition — key ideas

What is the core cause of the curse of dimensionality?

With $m$ distinguishable values along each of $d$ independent dimensions, the space contains $m^d$ possible regions.

Unless the dataset also grows exponentially, it becomes increasingly sparse relative to the space.

Show answerHide answer

What is concentration of measure?

The phenomenon where probability mass in a high-dimensional space concentrates in a narrow typical region.

As a result, aggregate quantities such as a vector's norm can vary surprisingly little.

Show answerHide answer

What is the difference between a distribution's mode and its typical set?

The mode is the single point with the highest density. The typical set is the region where samples usually land.

For a high-dimensional Gaussian, two factors compete as you move away from the origin: each individual point becomes less likely as density falls, but the amount of available space grows dramatically. The vast number of possible points in the larger shell outweighs their lower individual density, so samples typically land away from the mode.

Show answerHide answer

What does it mean for two vectors to be orthogonal?

They meet at a $90^\circ$ angle, equivalently:

$$x^\top y = 0.$$

“Orthogonal” is the vector-space version of “perpendicular.”

Show answerHide answer

What does near-orthogonality in high dimensions mean?

Independent random directions in a high-dimensional space usually have cosine similarity near $0$, so their angle is near $90^\circ$.

Show answerHide answer

Why do random high-dimensional vectors tend to be nearly orthogonal?

For independent random directions, roughly half their parameters align and half oppose, so their effects nearly cancel.

The leftover imbalance becomes a smaller fraction of the whole as dimension grows, pushing cosine similarity toward $0$.

Show answerHide answer

How do transformer components use the residual stream?

Attention heads and MLPs read selected directions or subspaces through linear projections, then write their results back by addition.

This makes the residual stream a shared communication channel across layers.

Show answerHide answer

How can a residual stream represent more features than it has dimensions?

It assigns features to many approximately orthogonal directions and stores the active features as a sum of those directions.

This is superposition: it supports more possible features than dimensions at the cost of some interference.

Show answerHide answer

Why is sparsity necessary for feature superposition?

Approximately orthogonal features still interfere slightly. If only a few features are active at once, that interference remains manageable; if many activate together, it accumulates and can overwhelm the signal.

Show answerHide answer

Deeper Connections and a Bag of Tricks

The Gradient Reveals the Update Rule

A loss tells us how a model is scored. Its gradient with respect to the parameters tells us what gradient descent will actually change. When a loss looks opaque, take one training example and ask: Which parameters move, in what direction, and when does the update vanish?.

Take softmax cross-entropy. A linear classifier stores one weight vector $w_j$ for each class. For an input representation $x$, it computes

$$z_j=w_j^\top x, \qquad p_j=\operatorname{softmax}(z)_j, \qquad L=-\log p_y.$$

The gradient with respect to each class’s weight vector is

$$\nabla_{w_j}L = \left(p_j-\mathbf 1[j=y]\right)x,$$

where $\mathbf 1[j=y]$ is $1$ for the correct class and $0$ otherwise. Gradient descent therefore updates

$$w_j \leftarrow w_j-\eta\left(p_j-\mathbf 1[j=y]\right)x.$$

The loss has become an update rule:

• The correct class weight moves toward $x$.
• Incorrect class weights move away from $x$; the class receiving the most mistaken probability gets pushed hardest.
• If the model is confidently correct, $p_y\approx1$ and the entire update is nearly zero.

Every weight moves along the example’s representation $x$. In a neural network, $x$ is the final hidden representation and these $w_j$ are the final-layer weights; backpropagation carries the same correction signal into earlier layers.

Stable probability computations

Logs and exponentials let us switch between multiplicative and additive views of the same calculation. Taking a log turns a product into a sum $\log(ab) = \log a + \log b$ so multiplying many probabilities becomes adding log-probabilities, which is both easier to work with and less likely to underflow. Exponentiation reverses it: softmax exponentiates logits into positive scores, while cross-entropy takes the negative log of the correct class’s probability so that confident mistakes receive a large penalty.

When ML engineers say they are working in log space, they mean storing and manipulating $\log p$ instead of $p$ itself. Multiplication becomes addition, division becomes subtraction, and the result is exponentiated only when an ordinary probability is actually needed. This keeps the probability of a long token sequence from being rounded down to zero.

Exponentials create the opposite danger: large logits can overflow. Softmax is unchanged if every logit is shifted by the same amount, so implementations subtract the largest logit $m=\max_j z_j$ before exponentiating:

$$\operatorname{softmax}(z)_i =\frac{e^{z_i-m}}{\sum_j e^{z_j-m}}.$$

The related log-sum-exp trick computes $\log\sum_j e^{z_j}$ as $m+\log\sum_j e^{z_j-m}$. In practice, stay in log space and use fused operations such as logsumexp and log_softmax rather than exponentiating and then taking a log again.

y = eˣ

additive input → multiplicative output

y = ln x

multiplicative input → additive output

y = −ln x

small inputs → large penalties

Loss Functions as Probabilistic Assumptions

A nueral network produces numbers (ex logits), but those numbers do not have probabilstic meaning by themselves. We give them meaning by choosing a likelihood family. Gaussian, Bernoulli, Categorical and so on and letting the network predict the distrution’s parameters:

$$x \xrightarrow{\text{neural network}} \text{distribution parameters} \xrightarrow{\text{likelihood family}} p_\theta(y\mid x).$$

Training then adjusts $\theta$ to make the observed targets likely. Each term asks: “How surprising is the observed target $y_i$ under the distribution predicted for $x_i$?”

$$\arg\min_\theta-\sum_i\log p_\theta(y_i\mid x_i).$$

For example, we can choose the probabilistic model to be Gaussian and decide that its output should mean “the mean of a Gaussian” and that the residual noise should be Gaussian. With fixed $\sigma$, minimizing the negative log-likelihood reduces to MSE.

$$p_\theta(y\mid x) = \mathcal N\!\left(y;\mu_\theta(x),\sigma^2\right).$$

The neural network computes

$$\mu_\theta(x)=f_\theta(x),$$

The same thing occurs in Large Language Models. The network determines the logits, while we chose to interpret their softmax as the parameters of a categorical distribution. This choice leads to cross-entropy.

$$z_\theta(x)=\text{neural-network logits},$$ $$p_\theta(y=k\mid x) = \operatorname{softmax}(z_\theta(x))_k.$$

So choosing a loss means deciding how to interpret the network’s outputs and which errors the model should consider plausible. MSE heavily penalizes large residuals because Gaussian noise treats them as exceptionally unlikely; MAE is more tolerant because the Laplace distribution assigns more probability to large deviations. The likelihood family supplies the assumption, the neural network predicts its parameters, and the resulting negative log-likelihood supplies the loss.

Equivalent Objectives and Disappearing Constants

Two formulas can look different but still lead training to exactly the same model parameters. An optimizer only cares about ranking the choices of $\theta$. Adding the same number to every score, or multiplying every score by the same positive number, cannot change which one wins:

$$\arg\min_\theta L(\theta) = \arg\min_\theta \left[aL(\theta)+c\right], \qquad a>0.$$

An strictly increasing function such as $\log$ also preserves the ordering. This is why maximizing a product of likelihoods is equivalent to maximizing their log-likelihood sum, and therefore to minimizing negative log-likelihood:

$$\arg\max_\theta \prod_i p_\theta(y_i\mid x_i) = \arg\min_\theta -\sum_i\log p_\theta(y_i\mid x_i).$$

Avoiding Optimizing Constraints

Some quantities are only valid inside a restriced range: a standard deviation must be positive, a probability must lie between zer and one, and a set of class probabilities must sum to one. Directly optimizing these constrained values is akward, a gradient step could make a standard deviation negative or push a probability above one.

The usual trick is instead of forcing the opimizer to obey a constraint, choose parameters in which every possible value is already valid. The transformation enforces the constraint automatically, although it also changes the geometry through which the gradient moves. For example, sigmoid turns an real number into a probabilities between 0 and 1, and softmax turns any vector of logits into nonnegative values that sum to one.

$$p=\operatorname{sigmoid}(z) \qquad\text{and}\qquad p_i=\operatorname{softmax}(z)_i.$$

Parameters are Not the Model

A neural network’s weights are one way of writing down a function, but they are not the function itself. Different parameter values can produce exactly the same output for every input. Adding the same constant to every logit leaves softmax unchanged. ReLU networks allow some weights to be scaled up if the following weights are scaled down.

This means that two independently trained networks can implement similar functions while having very different raw weights. It also explains why loss landscapes contain many equivalent solutions and why comparing neurons or parameters across models can be misleading.

Differentiating Through Randomness

When a model samples a random value during its forward pass, we do not differentiate the randomness itself. We need to differentiate how the model’s parameters influence the distribution of possible outcomes. For a Gaussian sample,

$$z\sim\mathcal N(\mu,\sigma^2),$$

sampling appears to interrupt the path from the loss back to $\mu$ and $\sigma$. The reparameterization trick rewrites the same sample as

$$\epsilon\sim\mathcal N(0,1), \qquad z=\mu+\sigma\epsilon.$$

The randomness now lives entirely in $\epsilon$. During backpropagation, $\epsilon$ is held fixed, making $z$ an ordinary differentiable function of $\mu$ and $\sigma$. The model can learn how shifting or stretching its distribution would have changed the loss. This is used in VAEs and other models with continuous latent variables.

Discrete samples, such as choosing a word or an action, cannot change smoothly. Methods such as REINFORCE instead use the result as feedback: increase the probability of choices that performed better than expected and decrease the probability of choices that performed worse. This works, but usually produces noisier gradients.

The general strategy is: separate the random noise from the learnable transformation whenever possible; when that is impossible, learn by adjusting the probability of sampled outcomes.

Reading ML Math Like Code

Dense math becomes much less intimidating when you stop trying to read it like prose and start reading it like code.

Shapes are types

It can be helpful to think of a Math variable’s type like a tensor’s shape in code. It tells you which operations are legal and often reveals what the equation means. A scalar such as a learning rate $\eta \in \mathbb{R}$ is one number. A vector $x \in \mathbb{R}^d$ is a list of $d$ numbers. A matrix $W \in \mathbb{R}^{m \times d}$ maps a $d$-dimensional vector to an $m$-dimensional one. Even in Math it can be helpful if we think of the axis names carry meaning and standard broadcasting rules still apply. Tensors can only be added when their shapes match or when a smaller tensor is deliberately broadcast across a missing axis.

The common machine learning operations

A few symbols account for a huge fraction of ML papers, and often you can think of them as control flow:

• $\sum_i f(x_i)$ means loop over $i$ and accumulate.
• $\mathbb{E}_{x\sim p}[f(x)]$ means draw $x$ from $p$, evaluate $f(x)$, and average in practice, usually a dataset or minibatch mean.
• $\arg\min_\theta L(\theta)$ returns the input $\theta$ that makes the loss smallest, not the minimum loss value itself.
• $p(x) \propto e^{-E(x)}$ means the two sides have the same shape but differ by a constant normalization factor.
• A subscript such as $p_\theta(x)$ says that the function’s behavior depends on the parameters $\theta$.
• $x \sim p$ means sample $x$ from the probability distribution $p$.
• $\nabla_\theta L$ means compute the gradient of loss $L$ with respect to parameters $\theta$ ie the vector of partial derivatives that backprop produces.
• $\theta \leftarrow \theta-\eta\nabla_\theta L$ means replace $\theta$ with its updated value. The arrow means assignment, not mathematical equality.
• $a\odot b$ means elementwise multiplication: multiply corresponding entries. This differs from the dot product $a^\top b$ and matrix multiplication $AB$.

Read from the inside out

Finally, execute nested notation from the inside out. For

$$\theta^* = \arg\min_\theta\; \mathbb{E}_{(x,y)\sim p_{\text{data}}} \left[-\log p_\theta(y\mid x)\right],$$

first compute the model probability of the observed target, then take its negative log, average that loss over data, and finally find the parameters that minimize the average. Math notation sure is compact.

Worked example: Attention

The transformer paper wrote the scaled dot-product attention as one nested expression:

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

B = examples in the batch
T = token positions in each example
d = features stored at each token position

Suppose $Q,K \in \mathbb{R}^{B\times T\times d_k}$ and $V \in \mathbb{R}^{B\times T\times d_v}$. Translate the equation from the inside out, assigning each intermediate result a name:

scores = Q @ K.transpose(-2, -1)  # [B, T, T]
scaled = scores / sqrt(d_k)       # [B, T, T]
weights = softmax(scaled, dim=-1) # [B, T, T]
output = weights @ V              # [B, T, d_v]

$QK^\top$ contracts the $d_k$ feature dimension, leaving two token dimensions: one query token against every key token. That is why scores has shape $B\times T\times T$. Softmax turns each query token’s row into weights over the key tokens. The final multiplication contracts that key-token dimension with $V$, producing one weighted mixture of value features for every query token.

Annoyingly, mathematical expressions often suppress implementation details, so translating them into code requires information from the surrounding text and conventions. Here, the scores have shape $B\times T_{\text{query}}\times T_{\text{key}}$. Attention requires each query token to produce a distribution over key tokens, so softmax must normalize the final, key-token dimension: dim=-1.