A new slippery game shows where randomness lives in an MDP: the transition kernel \(P(s'\!\mid\! s,a)\). That randomness puts an expectation in the Bellman backup — which gives policy evaluation, then policy iteration, and finally the one-line shortcut, value iteration.
Reading: Sutton & Barto, Ch 4.1–4.4
The robot 🤖 picks a direction each step, but the floor is slippery: it moves the way you chose 80% of the time and veers 90° to a side the other 20%. Try it — click the arrows (or W A S D) and steer to 💎 while dodging 🔥. Because you can slip, the answer we want is a policy (a move for every cell), not one fixed path.
One cell = one state. Actions ↑ ↓ ← →, but moves are noisy: 0.8 intended, 0.1 each 90° side. The black cell is a wall; wall and outer border share one rule — step (or slip) into either and you stay put.
−0.04 per step (a gentle nudge to be efficient). +1 for landing on 💎. −1 for landing on 🔥.
💎 and 🔥 end the episode. γ = 0.9 — far-off rewards count a little less than near ones.
Two questions drive the lecture: ① given a fixed policy π, what is \(V^\pi\) at every cell? ② what is \(\pi^*\), the best policy?
The slippery floor is the world's transition kernel \(P(s'\!\mid\! s,a)\) — the probability of landing in cell \(s'\) when you take action \(a\) in cell \(s\). A deterministic world puts all the mass on one cell; slip spreads it:
Aim into a wall or border and that 0.8 (or 0.1) mass stays on \(s\) — a self-loop. Same shape for ↓ ← →, just rotated. The numbers in each row sum to 1: it's a probability distribution over next cells.
That's the whole point of the kernel for us: where you land is random, so a cell's value can't be read off one move — it must be an average over where the floor sends you. That expectation is exactly what the Bellman backup carries. Part 2 makes it precise →
For a fixed policy π, last lecture defined two values: the value of a state, \(V^\pi\), and the value of a state–action pair, \(Q^\pi\). Each is an expectation — over the action π picks and over where the kernel \(P\) sends you.
\(V^\pi\) sits on both sides — a one-step consistency condition, not a formula you can read off directly. Part 2 turns it into an algorithm.
A policy π is a fixed rule for acting: in state \(s\) it picks action \(a\) with probability \(\pi(a\!\mid\! s)\). (A deterministic policy just puts all its weight on one action.) Fix π and ask: starting at \(s\) and following π forever, what total discounted reward do we expect?
The expected discounted return from \(s\) when every action is drawn from π and every transition from the kernel \(P\). It's "expected" because both the policy and the floor are random.
One number per cell — this is the answer to Question 1 from the game. Computing \(V^\pi\) for every cell is called policy evaluation. Next: the one-step equation it must satisfy.
\(V^\pi\) satisfies a one-step equation: the value of \(s\) is the immediate reward plus the discounted value of wherever you go next. Turn it into an update by plugging the current guess \(V_k\) on the right-hand side:
Two averages stacked. The outer \(\sum_a\pi(a\!\mid\! s)\) averages over the action π would pick; the inner \(\sum_{s'}P(s'\!\mid\! s,a)\) averages over where the floor sends you. That inner sum is the expectation — it exists only because the kernel is stochastic. Turn slip off and \(P\) keeps one term, collapsing it to \(r+\gamma V_k(\text{next})\).
Apply this to every cell = one sweep. Repeat sweeps and \(V_k\to V^\pi\). But why repeat at all — why not solve it in one pass like last lecture's tree?
Last lecture's tree backup worked because the leaves gave known values (\(V=0\), no future left) — so a single bottom-up sweep filled everything above. General MDPs aren't so kind: the process can run forever, and states loop — you can leave \(s\), wander, and come back.
With a loop, \(V(s)=\dots+\gamma V(s')\) and \(V(s')=\dots+\gamma V(s)\). Each needs the other first — a circular dependency. No bottom-up order exists, so no single sweep can solve it.
The fix: initialize \(V_0(s)=0\) everywhere, then repeatedly apply the Bellman backup. With \(\gamma<1\) each sweep shrinks the error, and \(V_k\to V^\pi\). That iteration is policy evaluation →
Input: a policy π, MDP (S, A, P, R, γ), small θ > 0
Output: V_π(s) for every s
Initialize V(s) = 0 for all s in S
(loop until values stop changing)
Loop:
Δ = 0
For each state s in S:
v_old = V(s)
V(s) = Σₐ π(a|s) [ R(s,a) + γ Σ_{s'} P(s'|s,a) · V(s') ]
Δ = max(Δ, |v_old − V(s)|)
Until Δ < θ
Return V
Δ is the largest change in any cell during the sweep. When Δ shrinks below θ (e.g. 0.001), V is "close enough" to \(V_\pi\) and we stop. Now we know how good π is. Can we do better?
When the states fit in a table, stack the values into a vector \(V^\pi\), the expected one-step rewards into a vector \(r_\pi\), and the policy-induced transitions into a matrix \(P_\pi\) (row \(s\) is the distribution over next states under π). The Bellman expectation equation becomes one matrix equation:
A linear fixed-point: \(V^\pi\) sits on both sides. With \(\gamma<1\) the matrix \(I-\gamma P_\pi\) is invertible, so a unique \(V^\pi\) exists.
Invert the matrix once — exact, but costs \(\mathcal{O}(n^3)\) in the number of states.
The repeated sweep from the previous slide — cheap per step, converges to the same \(V^\pi\).
Same equation, two solvers. Direct solve inverts once; Bellman iteration reaches the same answer by sweeps. Iteration wins when the state space is large or \(P_\pi\) isn't known as a clean matrix.
Once we have \(V^\pi\), define the action-value \(Q^\pi(s,a)\) — the value of taking \(a\) now, then following π:
The natural next step: at every state, switch to the action with the highest \(Q^\pi\).
At a state, look at all the actions' Q-values and pick the action with the biggest one.
That single action becomes the new policy there: πnew(s) = the argmax action. Done.
Repeat at every state in S. That's the entire update step.
Picking the single best action is the simplest choice. But step back: the update is really solving an optimization problem — at each state, find the policy that scores best under \(Q\).
Greedy is the answer when you optimize with no restrictions: put all the weight on the highest-\(Q\) action. Other methods solve the same problem with extra conditions — for example, taking a smaller, safer step. We'll meet those later.
The takeaway: greedy is the simplest solution to a general optimization problem. Many RL algorithms are just different ways of solving it.
Policy iteration is just evaluation and the greedy update, chained: \(\;\pi_k \;\to\; V^{\pi_k} \;\to\; Q^{\pi_k} \;\to\; \pi_{k+1}\).
Compute \(V^{\pi_k}\) by iterative policy evaluation (Part 2).
\(\pi_{k+1}(s) = \arg\max_a Q^{\pi_k}(s,a)\) at every state.
If \(\pi_{k+1} = \pi_k\) → done, π = π*. Else loop back to 1.
Each round: V values get more accurate, the policy gets better. Surprisingly, this converges in just 3–10 outer rounds for most small MDPs. (Each outer round runs Part 2's inner loop to convergence.)
Policy evaluation averages over a fixed π. Value iteration drops the fixed policy: at each cell it takes the max over actions instead of the policy-average. That single change turns the expectation backup into the Bellman optimality backup.
Vk+1(s) = Σₐ π(a|s) Σs' P(s'|s,a)[ r + γ Vk(s') ]
maxₐ replaces Σₐ π(a|s) — keep the single best action
Value iteration is an aggressive, collapsed form of generalized policy iteration: value propagation and greedy improvement are combined in one Bellman optimality backup — instead of evaluating π all the way to convergence before each improvement.
Input: MDP (S, A, P, R, γ), small θ > 0
Output: V*(s) for every s, and a greedy policy π*
Initialize V(s) = 0 for all s in S
(loop until values stop changing)
Loop:
Δ = 0
For each state s in S:
v_old = V(s)
V(s) = maxa [ R(s,a) + γ Σ_{s'} P(s'|s,a) · V(s') ]
Δ = max(Δ, |v_old − V(s)|)
Until Δ < θ
(extract π* from V*)
For each state s in S:
π*(s) = argmaxa [ R(s,a) + γ Σ_{s'} P(s'|s,a) · V(s') ]
Return V, π*
Spot the difference from Part 2's eval: Σₐ π(a|s) became maxₐ. Outer loop, convergence test, initialization — all unchanged. One greedy pass at the end reads off π*.
You can now solve any small, fully-known MDP. If the evaluate ↔ improve loop went by fast, L6 rebuilds it slowly on the warrior tree. Then the real catch: we used \(P(s'\!\mid\! s,a)\) and \(R(s,a)\) directly — L7 throws away the model and learns from experience.
For a standard finite MDP — a grid world included — the optimal value obeys the Bellman optimality equation, and the optimal policy is just the action that attains the max:
At each state \(s\) it selects one best action — a deterministic map, no probabilities. (The braced quantity is \(Q^*(s,a)\): the value of taking \(a\), then acting optimally.)
A weighted average of numbers is at most their maximum. So for any stochastic policy \(\pi(\cdot\!\mid\! s)\), spreading probability over actions can never beat putting all of it on the single best one.
You'd never choose to play randomly — but policy evaluation works on any π. Give cell \(s\) a stochastic policy: flip a fair coin, ½ go ↑ / ½ go →. Suppose we already know the neighbours' values. Then \(V^\pi(s)\) is a double average — over the coin and over the slip.
Each \(Q\) already hides one expectation — the slip kernel 0.8 / 0.1 / 0.1 over where that action lands.
Two averages stacked: slip inside each \(Q\), the coin across them. A deterministic policy that always took → would score 0.545 — beating the coin's 0.5135. Averaging two actions can't beat committing to the better one — that's why π* never needs a coin.