Through L6 our V and Q lived in tables — one row per state. That breaks fast. Today we replace the table with a function we can train: a line, then a ReLU, then a small neural network. Same Bellman target — vastly more states.
Lectures 3–6 built a complete pipeline for solving any MDP — as long as it's small enough to write down and we know its rewards and slip probabilities. Today we relax both.
Tabular methods are perfect — for problems with thousands of states. The interesting problems have vastly more. We can't allocate one number per state, and even if we could, we'd never visit each state enough times to learn its value.
fits in a table easily.
more than atoms in Earth.
more than atoms in observable universe.
(84 × 84 × 4 frames, 256 grays)
And robotics, self-driving, language — all continuous states. Tables don't even apply.
Don't memorize one number per state. Learn a function \(V(s;\theta)\) — same input, same output, but only \(\theta\) (a handful of parameters) to store. Similar states get similar values for free — that's generalization.
V[s_1] = 4.2 V[s_2] = 3.7 V[s_3] = 5.1 V[s_4] = ? ⋮ V[s_N] = ? N may be huge
Each entry independent. Unseen states have no value. Doesn't fit in memory once N gets big.
V(s; θ) ≈ V*(s)
θ = (a few thousand
numbers, total)
Feed any \(s\) → get a value. Similar \(s\) → similar values. Storage independent of state count.
Same goal as before — make \(V(s;\theta)\) satisfy Bellman. The only question: what shape should the function be?
Pick a few features \(\phi(s)\) — numbers we extract from the state. The simplest model is a weighted sum of those features. Just two parameters: weights \(w\) and bias \(b\).
Same shape you saw in linear regression in stats class. We tune \(w\) and \(b\) to match Bellman targets.
For a 1-D feature, this is just y = wx + b. For an N-D feature, it's a hyperplane.
The features are how we tell the model "what matters about \(s\)." Pick well — your linear model has a chance. Pick badly — no number of weights will save you.
So… can a line fit any V? Let's try, on the next slide. (Spoiler: a stubborn one.)
Three stages. Drag sliders, watch MSE drop. Stage ① fails. Stage ② fits half. Stage ③ — stack ReLUs and the curve snaps. Hit Auto-Fit to watch gradient descent in 1.5s. Then click "What does this have to do with RL?"
What you just did with sliders has a name and a proof. A one-hidden-layer net with enough neurons (and a nonlinearity like ReLU or sigmoid) can approximate any continuous function on a closed region — to any precision you want.
Translated: "Stack enough ReLUs (or sigmoids), tune their weights, and you can match any reasonable function as closely as you like."
Some such network exists. The theorem doesn't tell us how to find it.
Width can grow with precision \(\varepsilon\). Smaller \(\varepsilon\) → more neurons.
In practice we use deeper nets — fewer parameters per accuracy. But the same idea.
The playground had inputs \(x\) and targets \(y\). RL's setup is identical in structure — just relabel the axes. Same neurons, same training; new application.
You drove this manually. Auto-Fit did gradient descent for you.
Same machinery. Replace MSE-against-truth with MSE-against-the-Bellman-target. That's it.
L6 ended on a question: what if we don't know the rewards and slip probabilities? Value iteration needs them — it averages over every next state. Without them, we just play, record each step \((s,a,r,s')\), and learn from what happened.
Exact — but only if someone hands us the model, and we can sweep every state.
No \(P\) at all. One step is noisy, but averaged over many steps it matches the backup on the left — that average is the \(\mathbb{E}\) in the loss.
Bellman tells us what \(Q^*\) should satisfy. We build a network \(Q(s,a;\theta)\), then push \(\theta\) so that the network's output matches the Bellman target. The gap, squared, is the loss.
What \(Q\) should equal: the immediate reward plus discounted best-next-Q. (Computed with frozen weights \(\theta^-\) for stability.)
What our network currently outputs for \((s,a)\). Just one forward pass.
Penalize big mismatches harder than small ones — and make the loss differentiable so gradients flow.
\(\mathcal{L}(\theta) = 0\) means the network satisfies Bellman's equation everywhere — i.e., it has found \(Q^*\).
You felt this when you clicked Auto-Fit: the sliders moved themselves. Under the hood, the computer asks "which way do I nudge each parameter to lower the loss?" and steps that direction. Repeat until convergence.
Run a batch of \((s, a, r, s')\) through the network → get \(Q(s,a;\theta)\).
Squared gap to Bellman target. L = (target − Q)².
Auto-diff (PyTorch / TF) returns \(\nabla_\theta \mathcal{L}\) — direction of steepest increase.
θ ← θ − α · ∇L. \(\alpha\) is the learning rate. Repeat from ①.
Same recipe trains image classifiers, language models, and Q-networks. Gradient descent is the universal solvent.
Mnih et al., DeepMind, 2015 (Nature). One network architecture. 49 Atari games. Beat humans on most. Same loss we just wrote — plus two stabilization tricks.
Convolutional layers extract features automatically. No hand-designed \(\phi(s)\).
~1.7M parameters. Outputs Q-value for every legal action at once — one forward pass per state.
Buffer breaks correlation in samples. Target net (\(\theta^-\)) keeps the moving target still long enough to chase.
Tap each card to reveal the answer.
Linear function approximators (\(V = w \cdot \phi(s) + b\)) are simple and trainable. What's wrong with stopping there?
Most real value functions are non-linear in raw state — see the U-shape in the playground. Stage 1 hits an MSE floor that no slope/intercept can reduce. ReLU adds bends; stacking them gives universal approximation.
When DQN "learns to play Atari," what specific quantity is gradient descent driving toward zero?
\(\mathcal{L}(\theta) = \mathbb{E}\big[(r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta))^2\big]\). When this hits zero, the network satisfies Bellman — i.e., it has found \(Q^*\) (under the data distribution).
The Universal Approximation Theorem says a wide-enough network can fit any continuous function. Does it tell us we'll find the right weights via gradient descent?
UAT is an existence result — a good network exists. Whether gradient descent (with finite data, finite compute) actually reaches it is a separate, much harder question. In practice it usually works — but we have no general guarantee. That's why training deep RL is engineering, not just math.
Function approximation is more powerful, but it's not always the right tool. Tabular still wins when the state space is small enough — and when you want a guaranteed-to-converge solution.
No grade. The point is to feel the equations, not to perform them.
Open the Universal Approximator playground in its own tab. Try: can you beat Auto-Fit? Get MSE under 0.05 with only 3 neurons. Then try with 2.
Pick a game you know — tic-tac-toe, Tetris, Minecraft. Roughly how many states does it have, and would a table fit? If you built a Q-network for it, what would you feed in as the state? Bring your answers to L8.
Optional. No grade. Bring questions to L8.
We replaced the table with a network and trained it on the squared Bellman error. Value iteration + samples + a neural net = DQN. Same target, vastly bigger problems.
Point the same tool at the policy: make π itself a network πθ(a|s) and train it by optimizing return directly. One objective, careful steps — the idea behind PPO and RLHF.