Policy iteration = score a policy, improve it, repeat. Today we nail the scoring step on our old friend the warrior tree — two ways: the vanilla way (roll up from the leaves) and the iterative way (guess, then repeat). Then we spend those values: improve the policy and loop — evaluate, improve, repeat — until it can't get any better.
Gridworld threw slip (randomness), Q-values, policy iteration, and value iteration at you in one lecture. No wonder it blurred together. Today we rewind to the tiny, deterministic warrior tree and change one thing at a time.
The tree is deterministic — a move goes exactly where you point.
Only 4 steps, so \(\gamma=1\): value = just add up the rewards along the way.
Given a policy, compute its value — two different ways.
Spend the values: pick a better move, then loop until it's optimal.
Four steps. At each node the warrior chooses L or R; the move lands on the next node and pays the reward shown inside it — a 🪙 coin or a 💀 boss. After the fourth move the game ends — nothing more to collect.
A POLICY \(\pi\)Its value \(V_\pi(s)\) = the total reward you'll collect from node \(s\) if you follow \(\pi\) to the end. That single number is the question of this whole lecture.
Start at the ends. An END node has no future, so its value is \(0\). Press Roll up to fill each parent from the child \(\pi_0\) leads to.
The whole roll-up hangs on one known fact: an END node is worth 0. That gives us a place to start. From there every value is just "my reward + a value I already know," so we sweep bottom-to-top and never guess.
Vanilla evaluation = backward induction. It needs a last step (the leaves) to anchor the recursion. Solve the ends first, then work backwards. One pass, exact answer.
So here's the trap for next slide: what if the game never ends? What if the warrior can wander in a loop, revisiting the same rooms forever — no leaves at all? Where would you even start rolling up from?
Every path ends in an END node. Start there (value 0) and roll up. Backward induction works.
Step around and you can return to a room you already left. There is no last step — so there is no leaf to start from. Backward induction has nowhere to begin.
We still want \(V_\pi\). But we can't start from the end, because there is no end. New idea needed.
If there's no leaf to anchor us, we just make one up: pretend every node is worth \(0\) to begin with. That guess is wrong — but the Bellman rule lets us make it less wrong, over and over.
Take your current guesses \(V_k\), apply the rule once to every node → better guesses \(V_{k+1}\). Repeat. When the numbers stop changing, they satisfy Bellman exactly — that's \(V_\pi\).
Each sweep, the truth "leaks in" one step further. Real values reach nodes one layer at a time — and (with discounting in looping worlds) the leftover error shrinks every pass until it vanishes.
| sweep k | V(🤺) |
|---|
Guess: all zeros. Press Sweep to apply the update to every node at once.
Each sweep changes the values by less than the last. Once a full sweep barely nudges anything — say the biggest change is under some tiny \(\varepsilon\) — the guesses have settled onto \(V_\pi\). On our depth-4 tree that took about 4 sweeps — one per layer; a looping grid takes more, but it always settles.
"The largest change across all nodes is tiny." In a discounted looping world each sweep shrinks the error by a factor \(\gamma<1\), so it can only get smaller — convergence is guaranteed.
This is the exact pseudocode from Lecture 5 — the one that looked like magic. Read it again now: it's just "guess zero, sweep the Bellman rule, stop when it settles."
On our tree the sum \(\sum_{s'}\) has one term (deterministic) and \(\gamma=1\), so it collapses to \(V(s)\leftarrow R+V(s')\) — exactly what we just did by hand. Gridworld only adds the average over slip and the discount.
The Bellman rule gives one equation per state. With \(n\) states that's \(n\) equations in \(n\) unknowns \(V(s)\) — and they're linear. So you don't have to roll up or sweep: solve the whole system at once, like the simultaneous equations from algebra class.
Substitute one into the other and solve — no leaves, no sweeps, exact in one shot. The loop that stumped backward induction is no trouble at all for algebra.
Stack every state's value into a vector. \(P_\pi\) is the table of move probabilities under \(\pi\), \(R_\pi\) the reward at each state. One matrix inverse returns every \(V_\pi(s)\) exactly.
Inverting an \(n\times n\) table costs about \(n^3\) work. For a few states it's instant — but Gridworld has thousands of states and a video game has billions. There the cheap iterative sweeps win. Direct for small, iterative for big.
Both gave \(V_{\pi_0}(🤺)=-11\) on our tree — and solving the equations directly lands on the very same number, because a linear system has exactly one solution. Vanilla is the shortcut you can use only when there's a bottom; iterative is the general tool that always works at scale. That's the whole point of iterating.
We scored \(\pi_0=\) "always Right" and got \(V(🤺)=-11\) — it marches down the right edge into boss after boss. The values now tell us, at every node, which move is really worth more: just compare reward + value of where it lands, and take the bigger one.
That's the greedy idea: wherever a move walks into a boss but the sibling grabs a coin, swap it. Three obvious wins are highlighted. Do it at every node and you get a new policy \(\pi_1\) — but is one pass enough? Next we write the rule, and hit a twist.
Read it aloud: at each node pick the move with the biggest "reward + value where it lands." Press Improve to switch the clear losers against \(V_{\pi_0}\) (shown on the spine; leaves \(=0\)).
Roll up from the leaves. Exact in one pass — but needs a bottom.
Guess 0, sweep, repeat until settled. Works on any looping world.
Values in hand → greedily pick the better move → loop to the best policy \(\pi^\star\).
Evaluation is the engine. Next we ask: what if we don't even know the rewards and slip probabilities — and the state is a raw image, not a tidy grid? L7: function approximation.