Policy iteration is exact — if you can touch every state. Go has more positions than atoms in the universe, so the table is impossible and the policy must become a function we fit from data. We'll follow one example — AlphaGo — from "why a table fails" all the way to the move it missed against Lee Sedol, and to how the same recipe trained ChatGPT.
Policy iteration, and why Go breaks it.
Store a function, not a table — the universal approximator.
It only works where data taught it (Game 4).
PMD · TRPO · PPO, in AlphaGo & ChatGPT.
References: Lan, Optimization Methods for RL, §2.4 · Cybenko (1989), Hornik (1991) · Schulman et al. (TRPO 2015, GAE 2016, PPO 2017) · Silver et al. (AlphaGo 2016).
Alternate two steps until the policy stops changing. Improvement is greedy: at each state, move to the action with the largest advantage. When no action can beat the current average anywhere, we've reached the optimal policy \(\pi^\star\).
Subtract the state's own average to get the advantage \(A^{\pi_k}=Q^{\pi_k}-V^{\pi_k}\): how much better than \(\pi_k\)'s average an action is. Averaged over \(\pi_k\)'s own actions it is \(0\).
All the mass goes on \(\arg\max_a A^{\pi_k}(s,a)\), the biggest-advantage action. (Same as greedy on \(Q\) — subtracting \(V(s)\) never moves the max.)
That largest advantage is the room left to improve at \(s\). Grab it and the value can never drop — \(V^{\pi_{k+1}}(s)\ge V^{\pi_k}(s)\), the improvement theorem. Once it hits \(0\) everywhere, no action beats the average and the policy is optimal.
Clean and exact — but the greedy max needs a stored advantage for every state, solved one at a time. Fine on a gridworld. Let's try it on Go.
The plan is exactly policy iteration: one row per board position, holding its value and its best move. Enumerate every position, evaluate it, improve it, repeat. So — how big is that table?
The table can't be stored and can't be filled. The only way out: something that reads any position — even unseen ones — and returns a move. We need to generalize.
Don't memorize an answer per position. Learn one function \(\pi_\theta\) that reads a position and outputs a move distribution. Then we keep only its parameters \(\theta\) — a fixed, modest list of numbers — instead of the whole table.
Feed any board — including positions never seen in training — and get a move distribution out. That's generalization.
Go's \(\sim 10^{172}\)-row table becomes \(\sim 10^{8}\) weights. The count no longer grows with the number of states.
Nearby positions share weights, so one update nudges many at once. The table treated every row as unrelated.
The old table is just the special case where \(f_\theta\) is a plain lookup (one-hot features, no sharing). Function approximation is the same object with the states tied together.
We store a function \(f_\theta\) instead of a table — but which one? First we pick a function class: a family of candidate functions that share one form and differ only by their parameters \(\theta\). Fitting then searches inside that class for the member closest to the data.
Fitting \(\theta\) pins the function down only where training data lives (the support). Off that support it must guess — and a flexible net can guess wildly.
In a table, Step B is an exact per-state \(\arg\max\): you hold the true \(A^{\pi_k}\) for every \((s,a)\) and solve each state on its own. A function has neither — no per-state knob, and only a finite-sample estimate \(\hat A\) from rollouts. So Step B becomes a single objective maximized over \(\theta\).
You hold the true \(A^{\pi_k}\) for every state — a closed-form \(\arg\max\), solved one state at a time, a full sweep each round.
Roll out \(\pi_k\), estimate the advantages \(\hat A\), then a gradient step on the shared \(\theta\). Same \(\langle \hat A,\pi_\theta\rangle\) as the left — averaged over sampled states, plus a \(-\mathrm{KL}\) leash to \(\pi_k\).
Function approximation is needed precisely because we can't enumerate every state and action. So \(\hat A\) (and \(Q\)) is reliable only around the states and actions the current policy visits often, and can be far off where the data never reached. A greedy jump can land the policy right in that untrusted region — betting on numbers that are wrong. So we move gradually: small step (the \(-\mathrm{KL}\) leash), collect new experience, re-estimate \(\hat A\), repeat.
Generating a response is an episode: choose tokens one at a time. That maps exactly onto the RL objects we have used all along.
The transition kernel is trivial — appending a token is deterministic. All the difficulty moves into the reward and the advantage.
The same evaluate → improve loop as policy iteration, now on language — with the small, careful steps from two slides ago.
For each prompt \(x\), sample responses \(y\sim\pi_\theta(\cdot\mid x)\) — the model answers.
Rate each finished response with a reward \(R(x,y)\); the advantage is how much better than the model's typical answer.
Take a small step raising the probability of above-average responses — then resample and repeat.
This is RLHF (and RL with verifiable rewards). Clean in principle — but it needs a reward \(R\) and an advantage \(A\), and getting each is genuinely hard.
The kernel was free and the policy is just the model. Nearly all the engineering in modern LLM RL goes into estimating \(R\) and \(A\) well enough to take a good step.
Pick a problem you care about and cast it into the objects we've used all lecture. Name these five and you can hand it to policy iteration.
Then run the loop you now know: evaluate \(V^\pi\), improve toward the max-advantage action, keep each step on a short leash — and repeat.