AI Math Focus · The Mathematics of Artificial Intelligence — From First Principles to Transformers

AI 数学焦点 · 英文版（English edition） — A progressive, intuition-first tour of the math that powers modern AI. No prior university math required: only high-school algebra and a willingness to follow one formula at a time.

由浅入深（From shallow to deep）: every part builds on the last, and every formula is followed by the answer to the question "where is this used in AI?"

---

How to Use This Focus

This focus is organised as seven ascending levels. Each level introduces one mathematical idea, explains why it exists, gives the formula, and then shows exactly where it appears in a real AI system.

Level 0  Why math is the language of AI        ← the big picture
   │
Level 1  Linear Algebra     representation    ← data & models as numbers
   │
Level 2  Calculus           learning          ← how to improve
   │
Level 3  Probability & Info uncertainty       ← how to measure & decide
   │
Level 4  Optimization       searching         ← how machines learn
   │
Level 5  Neural Networks    everything so far, assembled
   │
Level 6  The Transformer    modern AI's mathematical core

Three reading paths:

Path	What you read	Goal
Skim (30 min)	The "Intuition" and "Where it's used in AI" boxes only	Understand what each branch of math does for AI
Standard (3–4 h)	Every section, formulas included, skip derivations	Read the math in AI blog posts and papers
Deep (8–10 h)	Everything, with a pen; redo the worked examples	Build the mental models to design and debug models

A promise: by the end of Level 6, the core formulas of any modern AI paper — the loss, the gradient, the attention — should look like familiar friends, not foreign languages.

---

Level 0 — Why Math Is the Language of AI

0.1 One idea that connects everything

Every AI system, from a spam filter to ChatGPT, is built on the same abstraction:

A model is a function. It takes an input x, applies a mathematical transformation controlled by parameters θ (theta), and produces an output y:

y = f(x; θ)

• x — the input: a sentence, an image, a row of a spreadsheet, each converted into numbers.
• y — the output: "spam" or "not spam", the next word, a price prediction.
• θ — the parameters: billions of numbers that the model learns. They are the model's "knowledge".

0.2 Learning = searching for good parameters

A newborn model has random parameters, so it outputs garbage. Training is the process of adjusting θ so that f(x; θ) starts producing good answers. This single sentence — "find the parameters that make the function behave well" — is the seed of almost all the math in this focus:

• We need a way to represent x and θ as numbers → Linear Algebra (Level 1).
• We need a way to measure how wrong the output is → Probability (Level 3).
• We need a way to improve θ step by step → Calculus + Optimization (Levels 2 & 4).

0.3 The three pillars

Pillar	One-line job	Answer it gives AI
Linear algebra	Store and compute with data at scale	How is everything represented and calculated?
Calculus	Measure rates of change	Which way should each parameter move?
Probability & information	Handle uncertainty and measure error	How wrong are we, and how do we decide?
(Optimization = calculus + probability working together)	Search for the best parameters	How far should we move, and when do we stop?

Where it's used in AI: a neural network's forward pass is linear algebra (Level 1); its training loop is calculus-driven optimization (Levels 2 & 4); its loss functions are built from information theory (Level 3). There is no AI without all three.

---

Level 1 — Linear Algebra: The Language of AI

If you remember nothing else from this level, remember this: a neural network is a long chain of matrix multiplications, and an input is a vector.

1.1 Vectors: data as points in space

A scalar is a single number: 3.14, -7, 42.

A vector is an ordered list of numbers:

v = [ v₁, v₂, v₃ ]   e.g. v = [ 4, -2, 1 ]

Think of a vector as a point in space. A vector with 2 numbers is a point on a flat plane; with 3 numbers, a point in 3D space; with 784 numbers, a point in 784-dimensional space. The human brain can't picture 784 dimensions — but the math treats it exactly like the 2D case, just with more coordinates.

Why data is a vector. Before AI can see an image, hear a word, or read a sentence, everything is converted into a vector of numbers:

• A 28×28 pixel image = 784 numbers = one vector in 784-D space.
• A word like "king" = a list of ~300 numbers (an embedding — Level 6).
• A house for sale = its features [price, area, bedrooms, ...] as a vector.

Two basic operations:

Addition (combine):   u + v = [u₁+v₁, u₂+v₂, ...]
Scaling (stretch):    c·v   = [c·v₁, c·v₂, ...]

Where it's used in AI: every input to every model is a vector; adding and scaling vectors is how embeddings are combined and how images are normalised.

1.2 The dot product: the workhorse of AI

The single most used operation in all of machine learning is the dot product of two vectors:

a · b = a₁·b₁ + a₂·b₂ + ... + aₙ·bₙ      (multiply element-wise, then add)

Example: [1, 2, 3] · [4, 5, 6] = 1·4 + 2·5 + 3·6 = 4 + 10 + 18 = 32

Two ways to read the dot product:

1. A weighted sum. If b is a list of weights and a is a list of values, then a·b is the total — each value multiplied ("weighted") by its weight. This is exactly what a neuron does (Level 5).
2. A measure of similarity. The dot product is proportional to the angle between the vectors: a · b = |a|·|b|·cos θ
   • Same direction (θ = 0°): positive, maximal.
   • Perpendicular (θ = 90°): zero.
   • Opposite (θ = 180°): negative.

Where it's used in AI: a neuron computes w·x (weighted sum of inputs); search engines and recommender systems rank items by dot-product similarity; attention (Level 6) is built from dot products between words.

1.3 Matrices: the packing of many dot products

A matrix is a rectangular grid of numbers:

W = [ w₁₁  w₁₂  w₁₃ ]      (2 rows × 3 columns)
    [ w₂₁  w₂₂  w₂₃ ]

Matrix–vector multiplication y = Wx computes one dot product per row:

y₁ = w₁₁·x₁ + w₁₂·x₂ + w₁₃·x₃     ← dot product of row 1 with x
y₂ = w₂₁·x₁ + w₂₂·x₂ + w₂₃·x₃     ← dot product of row 2 with x

Dimensions rule: a matrix of size (m × n) times a vector of size n gives a vector of size m. The inner dimensions must match.

This is a linear layer. The most common line in AI code is:

z = W·x + b

where W (weights) is a matrix, x (input) is a vector, and b (bias) is a vector. This one line — repeated thousands of times — is a neural network.

Why matrices are so important:

1. They pack computation. One matrix multiply performs thousands of dot products at once, all independent — ideal for parallel hardware.
2. They compose. Applying layer 2 after layer 1 is W₂(W₁x) = (W₂W₁)x. Stacking layers = multiplying matrices. This is why GPUs (graphics cards) power AI: they are massively parallel matrix-multiplication engines.

Matrix × matrix (m×n) · (n×p) → (m×p): each output entry is the dot product of a row of the first matrix with a column of the second. Same rule, one dimension deeper.

Where it's used in AI: every neural network layer is a matrix; the entire forward pass of a transformer is a sequence of matrix multiplications; the "weights" of a trained model that you download are matrices of numbers.

1.4 Two small helpers

• Identity matrix I: the "do nothing" matrix (1s on the diagonal, 0s elsewhere); I·x = x. The neutral element, like 0 for addition.
• Transpose Aᵀ: flip rows and columns; used constantly to re-shape computations so dimensions match.

1.5 Eigenvalues and eigenvectors: the axes of action (intuition)

Some vectors are special for a given matrix A: applying A to them only stretches or shrinks them, never rotates them.

A·v = λ·v

• v is an eigenvector — a direction that A does not change.
• λ (lambda) is the eigenvalue — the factor by which A stretches along that direction.

λ	Effect of repeatedly applying A
λ > 1	grows (amplifies)
0 < λ < 1	shrinks (damps)
λ < 0	flips direction

Intuition: eigenvectors are the "axes of action" of a transformation — like the principal axes of a spinning object. Knowing them tells you what the matrix fundamentally does, ignoring coordinates.

Where it's used in AI: PCA (principal component analysis) — the workhorse of dimensionality reduction — finds the eigenvectors of a data covariance matrix to discover the directions of greatest variance; eigenvalue intuition underlies understanding of gradients, convergence, and why repeated multiplication (as in deep chains) can explode or vanish.

1.6 Level 1 recap — where linear algebra lives in AI

AI component	Linear-algebra object
Input (image, text, features)	a vector x
Model weights	matrices W
Forward pass	z = W·x + b, repeated layer by layer
Similarity of embeddings	dot product a·b
Dimensionality reduction (PCA)	eigenvectors & eigenvalues

---

Level 2 — Calculus: The Engine of Learning

Linear algebra tells AI how to compute. Calculus tells AI how to improve — and improvement is the entire point.

2.1 The core question

A model makes a prediction; we can measure how wrong it is (the loss, Level 2.5). Now we want to tweak the parameters θ to reduce the loss. But which way? The answer comes from the derivative: if I change a parameter by a tiny amount, how much does the loss change?

2.2 The derivative: slope, generalised

For a function f(x), the derivative f′(x) measures the rate of change of f at the point x — the slope of the tangent line:

f′(x) = lim (h → 0)  [ f(x+h) − f(x) ] / h

A few derivatives worth memorising:

f(x)	f′(x)
c (constant)	0
xⁿ	n·xⁿ⁻¹
eˣ	eˣ
ln(x)	1/x
sin(x)	cos(x)

Worked example. f(x) = x² → f′(x) = 2x. At x = 3, f′(3) = 6. Meaning: at x = 3, f is increasing at a rate of 6 units of output per unit of input. Increase x slightly → f increases by ≈ 6× that amount.

2.3 Partial derivatives and the gradient

A model's loss depends on millions of parameters, not one. So we take the derivative with respect to each parameter separately, holding the others fixed — a partial derivative ∂f/∂xᵢ.

The gradient is the vector of all partial derivatives:

∇f(θ) = [ ∂f/∂θ₁,  ∂f/∂θ₂,  ...,  ∂f/∂θₙ ]

Two facts make the gradient the most important object in AI:

1. ∇f points uphill — in the direction of steepest increase of f.
2. −∇f points downhill — the steepest way to decrease f.

So "which way should each weight move to reduce the error?" has a precise mathematical answer: move each weight opposite to its partial derivative.

Where it's used in AI: the whole of training. Every update to a neural network is a small step in the direction of −∇(loss).

2.4 The chain rule: how errors flow

Real models are compositions: layer 1's output feeds layer 2, which feeds layer 3, and so on. When we need the derivative of a composed function, we use the chain rule:

d/dx  f(g(x))  =  f′(g(x)) · g′(x)

"The rate of change of the whole equals the rate of change of the outer function, times the rate of change of the inner function." In words: changes multiply along a chain.

Worked example. Let g(x) = 2x and f(u) = u². Then f(g(x)) = (2x)² = 4x².

• f′(u) = 2u, so f′(g(x)) = 2·(2x) = 4x
• g′(x) = 2
• Chain rule: (4x)·(2) = 8x ✓ (matches the direct derivative of 4x²)

Why this is the backbone of AI: a deep network is a composition of thousands of functions. The chain rule tells us how to pass the error from the output back through every layer — this is exactly backpropagation (Level 5.4). Learning a neural network is the chain rule, repeated billions of times per training run.

2.5 Loss functions: measuring how wrong we are

To improve, we first need a single number that says "how bad was this prediction?" — the loss L(θ).

Mean Squared Error (MSE) — for regression (predicting numbers):

L = (1/n) · Σᵢ (yᵢ − ŷᵢ)²

• yᵢ = true value, ŷᵢ = predicted value, n = number of examples.
• Squaring punishes large errors disproportionately and keeps the loss positive.

Cross-entropy — for classification (predicting categories):

L = − Σᵢ yᵢ · log(ŷᵢ)

• We'll meet it properly in Level 3.6; for now: it rewards confident, correct predictions and punishes confident, wrong ones heavily.

The learning problem, stated once and for all:

Find the parameters θ that minimise the loss L(θ).

Where it's used in AI: every model is trained by minimising some loss — MSE for price/weather prediction, cross-entropy for spam/vision/language tasks. The choice of loss defines what "good" means.

2.6 Level 2 recap — calculus in one paragraph

The gradient ∇L tells us the direction of steepest increase of the loss; the chain rule lets us compute that gradient through arbitrarily deep chains of layers; the loss gives us a single number to minimise. Levels 4 and 5 will turn these three ideas into the actual training loop of a neural network.

---

Level 3 — Probability & Information: Handling Uncertainty

A model never sees the whole world — it sees noisy, incomplete data. Probability is the mathematics of that uncertainty, and information theory is the mathematics of measuring it. Together they produce the losses AI actually optimises.

3.1 Probability: the language of uncertainty

• P(A) is a number between 0 and 1: how likely event A is. The probabilities of all possible outcomes sum to 1.
• Conditional probability P(A | B) — "probability of A given that B happened":

P(A | B) = P(A and B) / P(B)

Example. P(flu | fever) is higher than P(flu): knowing you have a fever updates the probability of flu. This idea — evidence updates belief — is the heart of modern AI decision-making.

3.2 Bayes' theorem: updating beliefs

Bayes' theorem is just the conditional-probability formula rearranged:

P(A | B) = P(B | A) · P(A) / P(B)

Term	Name	Plain meaning
P(A)	prior	what we believed before the evidence
P(B | A)	likelihood	how likely the evidence is if A is true
P(A | B)	posterior	what we believe after the evidence

Intuition: posterior ∝ likelihood × prior. New evidence multiplies our prior belief; the denominator just renormalises.

Worked example. Suppose 1% of people have a disease; a test detects it with 99% accuracy and has a 5% false-positive rate. If you test positive, what is the probability you have the disease? Bayes' theorem gives ≈ 16.7% — far less than the 99% intuition suggests, because the disease is rare. This is why AI systems and doctors must think in probabilities, not certainties.

Where it's used in AI: spam filters and classifiers built on Naive Bayes; Bayesian optimisation for tuning models; and conceptually, every model that outputs a probability is doing "evidence → updated belief".

3.3 Random variables and distributions

A random variable X is a number whose value is uncertain. Its distribution describes how likely each value is.

Discrete (countable outcomes):

• Bernoulli — coin flip: P(X=1) = p, P(X=0) = 1−p.
• Categorical — a die, or a word choice from a vocabulary.

Continuous — the most famous is the Normal / Gaussian distribution N(μ, σ²), the bell curve:

• μ (mean) = centre of the bell.
• σ (standard deviation) = width/spread of the bell.

The bell curve appears everywhere because of the Central Limit Theorem: average many independent random things and the result tends toward a normal distribution — which is why "normalise your data" is such a powerful practical trick.

Where it's used in AI: models are initialised with small random (often normal) weights; data is standardised to a normal shape before training; dropout (Level 4.6) and noise-injection are all distributions at work.

3.4 Expectation and variance

• Expectation E[X] — the average value you'd see over many trials: E[X] = Σ x·P(x) (or an integral for continuous X). It is a weighted average.
• Variance Var(X) — how spread out the values are around the mean: Var(X) = E[(X − μ)²]. High variance = unstable, scattered results.
• Linearity of expectation: E[aX + b] = a·E[X] + b.

Where it's used in AI: every loss is (approximately) an expected error over the data distribution; the "bias–variance tradeoff" (Level 4.6) uses exactly these two words in their mathematical senses.

3.5 Maximum Likelihood Estimation (MLE): fitting to data

Suppose we have data and a family of possible models (parameterised by θ). Which θ is best? Maximum likelihood says: choose the θ that makes the observed data most probable.

L(θ) = P(data | θ)     ← the likelihood
θ̂    = argmax θ  L(θ)  ← the most likely parameters

Because products of probabilities get tiny and awkward, we almost always maximise the log-likelihood log L(θ) instead — the logarithm is monotonic, so it doesn't change which θ wins, but it turns products into sums.

Worked example. A coin lands heads 7 times out of 10 flips. The likelihood of the data for a head-probability p is p⁷(1−p)³. Maximising this over p gives θ̂ = 0.7 — which is exactly the intuition "estimate the probability by the observed frequency". MLE is the formalisation of common sense.

Why this matters enormously: minimising cross-entropy (the classification loss of Level 2.5) is mathematically equivalent to maximising likelihood. And minimising MSE is equivalent to maximum-likelihood under a normal error model. Every major training objective in AI is MLE in disguise — the losses of Levels 5 and 6 are not arbitrary inventions.

3.6 Information theory: measuring surprise

Information. An unlikely event carries more information: "it rained in the desert" tells you more than "it rained in the rainforest".

I(x) = −log₂ P(x)      (measured in bits)

Entropy — the expected surprise of a distribution — measures how uncertain it is:

H(p) = − Σᵢ pᵢ · log₂ pᵢ

• A fair coin (p = 0.5, 0.5): H = 1 bit. Maximum uncertainty for two outcomes.
• A coin that always lands heads (p = 1, 0): H = 0 bits. Zero surprise — boring.

Cross-entropy — the penalty for using distribution q when the true distribution is p:

H(p, q) = − Σᵢ pᵢ · log₂ qᵢ

• It is minimised exactly when q = p. The worse q approximates p, the larger the penalty.
• This is the classification loss: the model outputs q (its predicted probabilities), the true label is p (one-hot), and training pushes q toward p.

KL divergence — the "extra cost" of using q instead of p:

D_KL(p ‖ q) = Σᵢ pᵢ · log₂ (pᵢ / qᵢ) = H(p, q) − H(p)

• Always ≥ 0, and 0 only when p = q. It is not a true distance (asymmetric), but it's the standard "distance between distributions".

Where it's used in AI:

AI concept	Information-theory object
Classification loss	cross-entropy H(p, q)
LLM training	cross-entropy of next-token prediction
Model "perplexity"	e^(cross-entropy) — lower = better
Variational autoencoders (VAEs)	KL divergence in the loss
Decision trees (ID3)	information gain = entropy reduction

---

Level 4 — Optimization: How Machines Learn

We now have the three ingredients: a model (functions of Level 1), a gradient (Level 2), and a loss (Level 3). Optimization is the algorithm that ties them together — the actual search for good parameters.

4.1 The problem

We want to minimise L(θ) over possibly billions of parameters. Picture a hilly landscape: your current parameters θ are your position; the height of the ground is the loss L(θ); your goal is the lowest valley. You can't see the whole map — you can only feel the slope where you stand and take one step at a time. That is optimisation in one image.

4.2 Gradient descent: walk downhill

The gradient −∇L(θ) points in the steepest downhill direction. So the simplest learning rule is:

θ ← θ − η · ∇L(θ)

• η (eta), the learning rate, is the step size — the single most important hyperparameter in AI.
• Each pass over the whole formula is one step; thousands/millions of steps = training.

Worked example (1D). L(θ) = θ², start at θ = 10, η = 0.1.

Step	θ	∇L = 2θ	θ ← θ − 0.1·2θ
0	10.0	20	8.0
1	8.0	16	6.4
2	6.4	12.8	5.12
3	5.12	10.24	4.10
…	↓	↓	↓
∞	0	0	0 (the minimum)

Choosing η: too large → overshoot and diverge (θ bounces away); too small → crawl for eternity. In practice, η is tuned, scheduled (large → small over training), and often handled automatically by Adam (4.5).

4.3 Local minima and saddles

For the simple bowl above, gradient descent provably finds the minimum. Neural networks, however, are non-convex — the landscape has many valleys (local minima), ridges (saddle points), and plateaus.

Reassuring empirical facts (very roughly): in the extremely high-dimensional spaces where neural nets live, most local minima have similar quality, so landing in "a" valley is usually fine; the real obstacles are saddle points and plateaus, where the gradient is near zero but the loss is not minimal. Escaping these is why we add momentum (4.5) and why random initialisation matters.

4.4 Stochastic and mini-batch gradient descent

Computing the exact gradient requires the whole dataset — expensive and redundant. Instead:

• Full-batch GD: exact, but one step costs a full pass over all data.
• Stochastic GD (SGD): estimate the gradient from one random sample — noisy but very cheap per step. The noise can even help by jiggling the model out of poor valleys.
• Mini-batch GD (the standard): estimate from a random batch of 32–512 samples. A good compromise: stable enough to converge, fast enough to train.

Where it's used in AI: essentially all training. "Training a model" = mini-batch gradient descent (or Adam, below) repeated over many epochs (full passes through the data).

4.5 Momentum and Adam: smarter steps

Momentum keeps a running "velocity" of past gradients:

v ← β·v + ∇L(θ)        (β ≈ 0.9)
θ ← θ − η·v

The step builds up speed in consistent downhill directions and smooths out zig-zags — it rolls through plateaus instead of stalling.

Adam (Adaptive Moment Estimation) — the default optimizer of modern AI — combines momentum with a per-parameter adaptive step size: parameters with large gradients get smaller steps, small-gradient parameters get larger ones. One line of intuition: Adam gives every one of the billions of parameters its own learning rate.

Where it's used in AI: Adam (or a variant like AdamW) trains nearly every transformer, GPT-style model, and modern neural network.

4.6 Overfitting and regularisation

A model that memorises the training data instead of learning general rules is overfitting — great on training, terrible on new data. The mathematical framing is the bias–variance tradeoff:

	Bias (error from wrong assumptions)	Variance (error from sensitivity to data)
Too simple a model	high (underfits)	low
Too complex a model	low	high (overfits)

Regularisation = adding a penalty that nudges the model toward simplicity:

• L2 weight decay: add λ·‖θ‖² to the loss — keeps weights small, which keeps the function smooth. (‖θ‖² is the squared length of the parameter vector, from Level 1.)
• Dropout: randomly switch off neurons during training — the model can't rely on any single neuron, so it learns redundant, robust features.
• Early stopping: stop training when validation error stops improving.
• More data / data augmentation: the most reliable cure of all.

Where it's used in AI: every serious training run. The train/validation/test split — and all of deep-learning "hygiene" — exists because of the bias–variance problem.

---

Level 5 — Neural Networks: The Math in Action

This is the moment everything clicks: a neural network is just linear algebra (Level 1), composed into a function (Level 2), trained by gradient descent (Level 4) on an information-theoretic loss (Level 3).

5.1 From linear to nonlinear: the neuron

A single neuron does two things:

1.  z = w·x + b          ← weighted sum (dot product!) + bias   [linear]
2.  a = σ(z)             ← activation function                  [nonlinear]

Why the nonlinearity is non-negotiable. If we stacked only linear operations, layer 2 would compute W₂(W₁x + b₁) + b₂ — which is still just a linear function of x. Stacking linear layers is mathematically pointless: any number of them collapses into one. The activation function is what makes "deep" mean anything — without it, a 100-layer network is equivalent to a 1-layer network.

5.2 Activation functions: the shapes of nonlinearity

Function	Formula	Behaviour	Used for
Sigmoid	σ(x) = 1 / (1 + e⁻ˣ)	squeezes ℝ into (0, 1)	probability-like outputs; classical networks
tanh	tanh(x)	squeezes into (−1, 1), zero-centred	recurrent networks; earlier layers
ReLU	max(0, x)	cheap; 0 for negative, identity for positive	default for most modern nets
Softmax	softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ	converts scores into a probability distribution summing to 1	final layer of classifiers; attention

Softmax, closely: it takes raw "scores" and turns them into probabilities, with two nice properties — all outputs are positive and they sum to 1. Exponentiation makes the largest score stand out (winner takes most, but not all).

5.3 The forward pass: everything in one chain

A network with L layers is a chain of the neuron formula:

a⁰ = x                       (input)
z¹ = W¹·a⁰ + b¹              (layer 1 linear part)
a¹ = σ(z¹)                   (layer 1 activation)
z² = W²·a¹ + b²              (layer 2 linear part)
a² = σ(z²)                   (layer 2 activation)
   ...
ŷ  = softmax(zᴸ)             (output probabilities)

This entire chain is the function f(x; θ) of Level 0 — with all the W and b collected into θ. The output ŷ is a probability distribution over classes (or, for a language model, over the next word).

5.4 Backpropagation: the chain rule in action

The problem: we need ∂L/∂W for every layer's weight matrix. The loss depends on the last layer's output, which depends on the previous layer, and so on — a chain hundreds of links long.

The solution: the chain rule, applied from back to front. The backpropagation algorithm:

1. Forward pass: compute all activations and the loss.
2. Backward pass: compute the error signal δ at the output, then propagate it backwards: ∂L/∂Wˡ = δˡ · (activation of the previous layer)ᵀ where each δˡ is obtained from δˡ⁺¹ by the chain rule.

Worked micro-example (one neuron, no activation, MSE loss — the idea is identical for deep nets):

x = 1,  w = 2,  b = 0,  true y = 5
ŷ = w·x = 2
L = (ŷ − y)² = (−3)² = 9
∂L/∂ŷ = 2(ŷ − y) = −6
∂ŷ/∂w = x = 1
∂L/∂w = ∂L/∂ŷ · ∂ŷ/∂w = (−6)·(1) = −6
w ← w − η·(−6) = 2 + 6η        (w increases — correct: it should grow to reach y=5)

Key insight: each layer's error depends on the error of the layer after it — so the computation must travel backward, from the output to the input. This "backward pass" is why the field's most famous algorithm is called backpropagation and why the chain rule (Level 2.4) is the single most important formula in AI.

5.5 The training loop: one complete cycle

repeat until validation loss stops improving:
    1. sample a mini-batch of data
    2. forward pass:   compute ŷ and loss L
    3. backward pass:  compute ∂L/∂θ by backpropagation
    4. update:         θ ← θ − η·(Adam or SGD step)

That's it. Every neural network — image classifiers, speech recognisers, language models — is this loop, scaled to billions of parameters and trillions of training examples.

---

Level 6 — The Transformer: Modern AI's Mathematical Core

Large language models (ChatGPT and friends), modern translation, image models like Stable Diffusion — all rest on the transformer. Its mathematics is a beautiful reuse of everything above: embeddings (vectors), attention (dot products + softmax), training (cross-entropy + backprop + Adam).

6.1 Embeddings: words become vectors

Text must become numbers. The pipeline:

1. Tokenise: split text into tokens ("ChatGPT" → "Chat", "G", "PT" or similar).
2. Embed: look up each token in a learned embedding matrix E — a huge table where each row is a vector (say, 768 numbers) for one token.

The magic: training pushes semantically similar words to nearby vectors. In a classic example, embedding arithmetic works:

vector("king") − vector("man") + vector("woman") ≈ vector("queen")

Meaning has been arithmetised — exactly the dot-product similarity of Level 1.2.

6.2 Why attention: context is meaning

Static embeddings fail on context: "bank" in "river bank" and "money bank" is the same vector. Attention solves this by making each token's representation depend on the other tokens around it — a context-aware representation.

The core idea of attention in one sentence: each token looks at all the others, decides how relevant each one is, and mixes their information in proportion to that relevance.

6.3 Query, Key, Value: attention's mathematics

Each token produces three vectors by multiplying its embedding by three learned matrices:

• q (query): "what am I looking for?"
• k (key): "what do I offer?"
• v (value): "what information do I carry?"

Step 1 — Relevance. How relevant is token j to token i? By the dot product (Level 1.2):

score(i, j) = qᵢ · kⱼ

Step 2 — Softmax. Turn all scores for token i into a probability distribution (Level 5.2):

weights(i, j) = softmax_j( scores(i, ·) )

Step 3 — Weighted sum. Mix the values (a weighted combination, Level 1.2):

output(i) = Σⱼ weights(i, j) · vⱼ

All three steps at once — scaled dot-product attention:

Attention(Q, K, V) = softmax( Q·Kᵀ / √dₖ ) · V

• Q·Kᵀ: a matrix of all relevance scores.
• /√dₖ: scaling by the square root of the key dimension. Why? Dot products grow with dimension; without scaling, the softmax saturates into near-0/1 and gradients vanish. Dividing keeps the scores in a healthy range — a small but crucial piece of "why transformers train at all".
• softmax: relevance → weights.
• ·V: weighted mixing of information.

Self-attention: when Q, K, V all come from the same sequence, every token attends to every token. This is how the model builds context. The cost is O(n²) — every pair of positions is compared — which is why long documents are expensive and why "context windows" are a hot topic.

Where it's used in AI: attention is the heart of transformers — every LLM, translation system, and vision transformer is running this formula in every layer, for every token, for every position, in parallel.

6.4 Multi-head attention: many perspectives at once

One attention pass is one way of relating tokens. Multi-head attention runs h attention mechanisms in parallel, each with its own learned Q/K/V projections:

MultiHead(Q,K,V) = Concat(head₁, ..., head_h) · Wᴼ
headᵢ = Attention(Q·Wᵢ^Q, K·Wᵢ^K, V·Wᵢ^V)

Each head can specialise — one tracks syntax, another coreference ("the cat … it"), another position. Concatenating the heads and projecting back merges all perspectives. (Typical: 12–64 heads in modern models.)

6.5 Positional encoding: putting tokens in order

Attention is permutation-invariant: shuffle the input words and — with identical embeddings — the attention outputs are identical. Order matters for language ("cat bites dog" ≠ "dog bites cat"), so the model needs a sense of position.

Positional encoding adds a fixed pattern to each token's embedding based on its position:

PE(pos, 2i)   = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )

• pos = position in the sequence; i = dimension index; d = embedding dimension.
• Each dimension gets a sine/cosine wave of a different frequency. Together they form a unique "coordinate" for every position.
• Because the waves are regular, the difference between positions pos and pos+k depends only on k — so the model can learn relative distances ("the word 3 positions after this one").

Where it's used in AI: positional encodings (or their learned/rotary cousins like RoPE, used in most modern LLMs) are what give transformers a sense of order.

6.6 The transformer block: putting it together

One transformer block (repeated N times, e.g., 12–96):

input x
  → Multi-Head Attention → Add & LayerNorm → Feed-Forward (MLP) → Add & LayerNorm
  → output (input to next block)

Why the details matter:

• Residual connections (Add): each block computes x + f(x). The gradient (Level 2) can flow straight through the addition, bypassing deep chains — this is why transformers can be 100 layers deep without vanishing gradients.
• Layer normalisation: rescales activations to a stable range — keeps training stable, a normalisation trick in the spirit of Level 3.3.
• Feed-forward network: a small MLP (Level 5) applied to each position independently — the "thinking" after the "gathering".

The training objective — full circle. The transformer is trained to predict the next token, minimising the cross-entropy of the predicted distribution vs. the true next token (Level 3.6). That loss is propagated by backpropagation (Level 5.4) using Adam (Level 4.5). Every piece of this focus is in the loop at once.

6.7 Scaling laws: why bigger is (predictably) better

Empirically, the test loss of a transformer falls as a power law with more data, parameters, or compute:

L ≈ c · N^(−α)        (N = scale factor, α ≈ 0.05–0.1, c = constant)

Take the log of both sides and loss becomes a straight line in log-scale. This is why the industry races to scale: the mathematics says performance keeps improving in a predictable, extrapolatable way. It also explains emergent abilities — capabilities that appear almost abruptly once a model crosses a scale threshold.

---

Level 7 — Formula Cheat Sheet

Everything in one table. Bookmark this.

Concept	Formula	Level
Vector	x = [x₁, x₂, …, xₙ] — a point in n-D space	1
Dot product	a·b = Σᵢ aᵢbᵢ = |a||b|cos θ	1
Linear layer	z = W·x + b	1
Eigen equation	A·v = λ·v	1
Derivative	f′(x) = limₕ→₀ [f(x+h) − f(x)]/h; d/dx xⁿ = n·xⁿ⁻¹	2
Gradient	∇f = [∂f/∂θ₁, …, ∂f/∂θₙ] — points uphill; −∇f downhill	2
Chain rule	d/dx f(g(x)) = f′(g(x))·g′(x)	2
MSE loss	L = (1/n) Σᵢ (yᵢ − ŷᵢ)²	2
Conditional probability	P(A|B) = P(A∩B)/P(B)	3
Bayes' theorem	P(A|B) = P(B|A)·P(A)/P(B)	3
Expectation	E[X] = Σ x·P(x)	3
Entropy	H(p) = −Σ pᵢ log₂ pᵢ	3
Cross-entropy	H(p,q) = −Σ pᵢ log₂ qᵢ	3
KL divergence	D_KL(p‖q) = Σ pᵢ log₂(pᵢ/qᵢ) ≥ 0	3
Gradient descent	θ ← θ − η·∇L(θ)	4
Momentum	v ← βv + ∇L; θ ← θ − ηv	4
Sigmoid	σ(x) = 1/(1+e⁻ˣ)	5
ReLU	max(0, x)	5
Softmax	softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ	5
Neuron	a = σ(w·x + b)	5
Scaled dot-product attention	Attention(Q,K,V) = softmax(QKᵀ/√dₖ)·V	6
Positional encoding	PE(pos,2i)=sin(pos/10000^(2i/d)); PE(pos,2i+1)=cos(…)	6
Scaling law	L ≈ c·N^(−α)	6

---

Level 8 — Check Your Understanding

The IELTS-focus tradition: try each question before reading the answer.

Q1 (Level 1). An image is 32×32 pixels in grayscale. How many numbers are in its vector representation? What does that make the vector, geometrically?

Q2 (Level 1). Compute [1, 0, 2] · [3, 4, 1]. What are the two interpretations of this number in an AI context?

Q3 (Level 2). The loss is L(θ) = θ³ − 6θ. Compute the gradient and state which way θ should move from θ = 2 to reduce L.

Q4 (Level 2). Using the chain rule, find d/dx of f(x) = e^(x²). (Recall: d/dx e^u = e^u · du/dx.)

Q5 (Level 3). A test is 95% accurate for a disease present in 2% of the population, with a 4% false-positive rate. A person tests positive. Roughly what is the probability they have the disease? (Bayes: posterior ∝ likelihood × prior.)

Q6 (Level 3). A classifier outputs probabilities (0.7, 0.3) for a true label (1, 0). Compute the cross-entropy loss. Then compute it for (0.9, 0.1). Which is smaller, and why does that make sense?

Q7 (Level 4). Why is the learning rate η the most important hyperparameter, and what happens when it's too large or too small?

Q8 (Level 4). Explain in one sentence each: L2 weight decay, dropout, early stopping — and which single problem all three address.

Q9 (Level 5). Why does stacking linear layers without activation functions accomplish nothing?

Q10 (Level 5). In the backpropagation worked example, w moved up (from 2 toward 5). Explain, in terms of the chain rule, why the sign came out correct.

Q11 (Level 6). Write the scaled dot-product attention formula and identify which part is (a) the relevance scores, (b) the weights, (c) the information mixing.

Q12 (Level 6). Why is positional encoding necessary, and what property of attention makes it necessary?

A1. 32×32 = 1,024 numbers — a single point in 1,024-dimensional space.

A2. 1·3 + 0·4 + 2·1 = 5. Interpretations: a weighted sum (if one vector is weights) or a similarity score between two vectors (used for ranking, attention relevance, etc.).

A3. ∇L = 3θ² − 6. At θ = 2: 3·4 − 6 = 6 > 0, so the gradient is positive and L is increasing; move θ in the negative direction (−∇L), i.e. decrease θ, to reduce L.

A4. Let u = x². Then f′(x) = e^u · 2x = 2x·e^(x²).

A5. Prior P(disease) = 0.02. Likelihood P(+|disease) = 0.95. P(+) ≈ 0.95·0.02 + 0.04·0.98 ≈ 0.019 + 0.0392 = 0.0582. Posterior ≈ 0.019/0.0582 ≈ 0.33 (33%). The 95% accuracy is misleading because the disease is rare.

A6. H = −[1·log₂0.7 + 0] ≈ 0.515 bits; for (0.9, 0.1): −log₂0.9 ≈ 0.152 bits. The second is smaller because the model is more confident in the correct class — cross-entropy rewards confident correct predictions.

A7. η controls the step size of every update. Too large: overshoot, oscillation, divergence. Too small: slow convergence or stalling. It is the first thing to tune and the reason adaptive methods (Adam) were invented.

A8. L2 weight decay adds λ‖θ‖² to the loss to keep weights small; dropout randomly disables neurons during training to force redundancy; early stopping halts training when validation error rises. All three fight overfitting.

A9. A composition of linear functions is still a linear function: W₂(W₁x + b₁) + b₂ collapses into a single equivalent matrix. Without nonlinearity, "depth" buys nothing.

A10. ∂L/∂w = ∂L/∂ŷ · ∂ŷ/∂w = 2(ŷ−y)·x = −6·1 = −6, negative. Gradient descent takes w ← w − η·(∇), i.e. w increases when the gradient is negative — moving toward lower loss, exactly as the chain rule dictates.

A11. Attention(Q,K,V) = softmax(QKᵀ/√dₖ)·V. (a) QKᵀ = relevance scores; (b) softmax(·) = weights (a probability distribution); (c) ·V = weighted sum of values = information mixing.

A12. Attention is permutation-invariant: with identical embeddings, shuffling the tokens changes nothing, because every token attends to every token regardless of order. Positional encoding injects order information so the model can use word position (and relative distances).

---

Glossary — 30 terms in one line each

Term	One-line meaning
Activation function	Nonlinear function (e.g. ReLU) that lets deep networks represent more than linear functions
Adam	Default optimizer: momentum + per-parameter adaptive learning rates
Attention	Mechanism where each token mixes information from others, weighted by relevance
Backpropagation	Algorithm computing gradients by applying the chain rule backward through the network
Bayes' theorem	Rule for updating beliefs: posterior ∝ likelihood × prior
Bias	(a) learnable offset b in z = Wx + b; (b) error from over-simplification (bias–variance)
Chain rule	d/dx f(g(x)) = f′(g(x))·g′(x) — the backbone of backpropagation
Cross-entropy	Loss measuring how well predicted probabilities match true labels
Derivative	Rate of change of a function; slope of its tangent
Dot product	Weighted sum / similarity between two vectors
Dropout	Regularisation that randomly disables neurons during training
Eigenvalue / eigenvector	Direction v that a matrix only stretches by λ: A·v = λ·v
Embedding	Learned vector representation of a token or item
Entropy	Expected surprise / uncertainty of a distribution: −Σ p log p
Expectation	Probability-weighted average of a random variable
Gradient	Vector of all partial derivatives; points in the direction of steepest increase
Gradient descent	θ ← θ − η∇L: iterative downhill search for optimal parameters
KL divergence	Asymmetric "distance" between two distributions; ≥ 0, 0 iff equal
Learning rate (η)	Step size of each parameter update
Loss function	Single number measuring how wrong a model's predictions are
Matrix	Rectangular grid of numbers; each row = one dot-product computation
MLE	Choose parameters that make observed data most probable
Momentum	Optimisation trick: accumulate gradient velocity to smooth steps
Overfitting	Fitting training data too well, generalising badly
Positional encoding	Sine/cosine pattern added to embeddings to encode word order
Regularisation	Any penalty or technique that discourages overfitting
ReLU	Activation max(0, x); default choice in modern networks
Softmax	Converts scores into a probability distribution summing to 1
Stochastic gradient descent	Gradient descent using a small random batch per step
Transformer	Architecture of stacked attention + feed-forward blocks; basis of modern LLMs

---

This is the English edition (英文版) of the AI Math Focus. A Chinese edition (中文版) will follow, with the same structure and a shared EN↔ZH glossary.