We have the MDP — states, actions, rewards. But what makes one policy better than another, and what is the best one? Build the value function, play the doors and warrior-tree games to see why greedy ≠ optimal, and write the Bellman equation that pins down \(\pi^*\).
Reading: Sutton & Barto, Ch 3.5–3.6
Everything today lives inside one object \(\mathcal{M}=(\mathcal{S},\mathcal{A},\mathbb{P},c,\gamma)\) — five pieces that fully describe the world.
A policy \(\pi(a\!\mid\! s)\) is the probability of choosing action \(a\) at state \(s\). Fix \(\pi\), and the world rolls forward:
pick an action · pay a cost · land somewhere new · repeat
γ was the fifth element of \(\mathcal{M}\) we just listed. It sets how much a cost one step later still counts: a cost \(t\) steps away is weighted \(\gamma^t\). Slide it and watch how far the agent "sees."
Small γ → the agent only cares about the next step or two. γ near 1 → distant costs matter almost as much as today's.
\(V^\pi(s)\) is the total cost you expect to pay if you start in state \(s\) and follow \(\pi\) forever — with costs further in the future discounted by \(\gamma\).
add up every future cost — but each step ahead counts a little less (γ < 1)
Now, solve the problem. Walk backward from the future: compute the value of every state by reusing values you've already computed one step ahead. That's dynamic programming.
Two doors at every step. Each has a peephole — peek to see the reward inside. Step through to claim it; two new doors appear. 4 picks total. Highest score wins.
The peephole shows only this step's reward — not what waits behind the next door. Easy to grab the big number now and lose later.
Take 4 steps. At each step, go left or right. Each cell pays a reward — a 🪙 coin (+5 to +10) or 💀 boss (−10 to −5). Goal: finish with the highest total. There are \(2^4 = 16\) possible paths.
Sum the rewards along the red path. What total does greedy end with?
What information would let you make the right choice at every step?
Greedy fails because it judges each move by this step's reward only. The fix in one sentence: at every step, pick the move whose whole future looks best. Today's choice is judged by where it eventually leads — all the way to the end.
"Take the biggest reward I can see now." Looks one step ahead. Walks straight into a boss two steps later.
"Take the move whose total future reward is highest." Looks all the way to the leaves before committing the first step.
A plan is a fixed 4-step sequence of L/R moves. There are \(2^4 = 16\) plans. For each one, sum the four rewards along its path; pick the plan with the highest total.
4 steps gave 16 plans. What if the warrior took 10 steps? 20? 50? How many plans must we sum?
\(2^{10}{=}1{,}024\) · \(2^{20}{\approx}10^6\) · \(2^{50}{\approx}10^{15}\). Exponential — brute force is hopeless almost immediately. We need a smarter way.
Greedy got trapped on the big tree because each move hides a long future. Strip the future to nothing — make the problem one step — and greedy snaps back to optimal. Three observations turn that into a plan.
If the tree is just one step deep, the best total reward starting from \(s\) is just the larger reward. Call this best total \(V(s)\) — the value of state \(s\). No future, no trap.
What if every next state \(s'\) came tagged with its value \(V(s')\) — the best total starting from there? Then any step is again a 1-step problem: pick the action with the biggest reward + value.
Evaluation answered "how good is this π?" Now ask "what's the best we can do?" Same recursion, but instead of averaging over what π would do, we pick the best action at every state — \(\sum_a\pi(a\mid s)\) becomes \(\max_a\):
the best value = best action's value; the best policy = the action that attains it
Roll up the same way, but each node now keeps the better of its two children instead of averaging. The greedy-looking root \(V^*=+17\) beats the random policy's \(-4.1\); reading \(\arg\max\) at each node traces the dashed optimal path \(\pi^*\).
Split the infinite sum into the cost paid now plus the discounted value of wherever you land next — averaged over the action you pick (\(\pi\)) and the move that follows (\(\mathbb{P}\)). Written one state at a time:
stack all \(|\mathcal{S}|\) rows → matrix form \(\;V^\pi = c_\pi + \gamma\,\mathbb{P}_\pi V^\pi\) (with \(c_\pi,\mathbb{P}_\pi\) the \(\pi\)-averaged cost and kernel)
Last lecture's tree, in rewards (here \(r=-c\), so maximize). Fix \(\pi=\) "go L with 0.6, R with 0.4." Start at the leaves (\(V=0\)), then fill each parent: its own reward plus the averaged value below. The root's \(V_\pi=-4.1\) is exactly the Bellman equation, solved bottom-up. The kernel here is deterministic, so the \(\sum_{s'}\) collapses to one term — all the randomness lives in \(\pi(a\mid s)\).
I was interested in planning… but "programming" was a respectable word. The Secretary of Defense had a pathological fear of the word "research." So I used "dynamic" — it's impossible to use in a pejorative sense. It was a name no one could object to.— Bellman, Eye of the Hurricane (paraphrased)
Bellman's one big idea: the rest of an optimal plan, from any state onward, must itself be optimal — so a long problem splits into best-first-move + optimal-tail:
If the best first step from 🤺 is L, reuse L's own value \(V^*(L)\) as-is — never recompute it.
Scoring every plan explodes: \(2^H\) plans in a depth-\(H\) tree, \(A^{|\mathcal S|}\) policies in an MDP. Bellman never enumerates them:
Each value is computed once and reused, so \(2^H\) collapses to \(|\mathcal S|\) lookups — the engine behind every RL algorithm here.