Last time we built the probability toolkit. Today we use it to name every part of the agent–environment loop — state, action, reward, transition, policy — bundle them as the MDP, meet two values V and Q, and write down the equation that ties them together: Bellman.
One running example — a fair 6-sided die — touches every concept we built. From outcomes all the way to belief.
Sample space \(\Omega = \{1,2,3,4,5,6\}\). Pick out an event \(A = \{2,4,6\}\) ("even").
Define a random variable \(X(\omega)=\omega\). Each face has the same chance — \(p_X(k)=\tfrac{1}{6}\).
You learn \(B = \{2,4,6\}\). Odd values drop to 0; even values rise to \(\tfrac{1}{3}\) each.
Same die, three lenses. Now we use these tools to formalize the agent–environment loop.
Almost every applied-math task — from physics to RL — follows the same pattern. Today's lecture walks all three steps, once, end to end.
Pick symbols and structure so a messy real-world problem becomes a precise math object you can reason about.
Given the math model, design an algorithm that produces the answer. Try it on small examples first.
The math is abstract — but that's the superpower. Proofs of correctness, convergence rates: guarantees, not hope.
Same recipe everywhere — physics: laws → solve → measure. Finance: model → strategy → backtest. RL is no different.
First, name every piece of the agent–environment loop. State, action, reward, transition, discount factor — bundled as the Markov Decision Process. This is how we write the problem down precisely.
Same picture as Lecture 1 — but now we name what lives inside each box. The agent carries a policy \(\pi(a \mid s)\); the environment carries a transition kernel \(P(s' \mid s, a)\). Both are conditional probabilities — exactly the kind we just built.
At time \(t\) the agent feeds \(S_t\) to its policy \(\pi\) and samples \(A_t\). The environment feeds \((S_t, A_t)\) to its kernel \(P\) and returns \(S_{t+1}, R_{t+1}\). Two conditional distributions, taking turns.
How big is the state space?
Small enough to list, or too big to even count.
Few enough you could write them all down.
Real-world tasks usually live here.
Crucially: the state should contain everything you need to decide what to do next. (More on this when we hit the Markov property.)
What does an action look like?
Pick from a finite list, or choose a real number.
A finite menu of choices.
A vector of one or more real numbers.
Note \(\mathcal{A}\) can depend on the state — a chess piece can move differently depending on where it is.
Does \(R\) always need all three?
Often the action drops out. Sometimes the next state does. The general form covers either case.
Where you land tells you everything — the action you chose to get there doesn't matter.
The action you choose has its own cost — regardless of where you end up.
Why is \(R(s, a)\) "enough" — but \(R(s')\) sometimes isn't?
Once \((s, a)\) is fixed, the kernel \(P(s' \mid s, a)\) determines \(s'\). So \(R(s, a) := \mathbb{E}_{s' \sim P}[\,R(s, a, s')\,]\) — averaging over \(s'\) loses nothing. But \(R(s')\) can't distinguish two actions that land in the same \(s'\): a robot that spins gently vs hard reaching the same cell gets the same reward, and action costs vanish.
So: \(R(s, a)\) is the canonical form. \(R(s')\) is a special case for goal-only tasks where "where you land" tells the whole story.
In our warrior tree the reward sits on each cell — that's \(R(s')\). Whatever the form, the agent's only goal is to maximize the sum over time.
DEFINITION · TRANSITION KERNEL
For each \((s, a)\), a distribution over next states — \(\sum_{s'} P(s' \mid s, a) = 1\).
ASSUMPTION · MARKOV
The kernel uses only the current \((S_t, A_t)\) — past states and actions don't enter.
Is the next state always determined?
Sometimes \((s, a)\) fixes \(s'\). Sometimes the world rolls dice.
\(P(s' \mid s, a) = 1\) for one \(s'\), 0 for the rest.
Probability spreads over many possible \(s'\).
A chess board state captures everything — you don't need the move history to decide your next move. The state is sufficient.
Same prize, sooner or later? Most people pick sooner. We bake that preference into the math by discounting each future step by a factor \(\gamma \in [0, 1]\) — the last knob we need before writing the MDP down.
Each step further out picks up one more factor of \(\gamma\). With \(\gamma < 1\) the powers shrink fast, so far-future rewards barely register.
Only the next reward counts. Everything past one step gets multiplied by 0 and disappears.
A reward 10 steps away is worth \(0.9^{10} \approx 0.35\) of one now — present-biased, but still long-sighted.
Every step weighs the same. Fine when the task ends (sum stops). Risky when it never ends — the total can blow up.
One number turns "future" into a dial — from "only now" to "all equal." That's all γ does.
A Markov Decision Process is just a 5-tuple. If you can write your problem this way, you can solve it with RL.
All possible situations
All possible moves
Rules of the world — \(P(s' \mid s, a)\)
How well you're doing
How much future matters
RL problems vary along a few axes. Where a problem sits on each one decides which methods apply — and how hard it is.
The MDP we just defined sits at full obs, unknown kernel, stochastic — the standard setting we'll work in for this course.
For each scenario, decide where it sits on the three axes — and name the five ingredients \(\mathcal{S}, \mathcal{A}, R, P, \gamma\). Tap to reveal.
You're playing a hand of Texas Hold'em.
You can never see opponents' hole cards — that's partial observability. The deck is fair, but opponents' strategies are the unknown part of the kernel.
You're playing Go on a 19×19 board.
The whole board is visible — that's full observability. Rules are deterministic, but the opponent's policy lives inside \(P\), and we don't know it.
A smart thermostat runs continuously, switching the heater on or off to keep the room near 70°F.
No terminal step — it just keeps running. That's exactly when \(\gamma < 1\) earns its keep: it makes the infinite sum of rewards finite.
An LLM produces an answer one token at a time, then a reward model scores the finished response.
The transition is just "stick the token on the end" — fully known. That's why post-training methods can plan against the LLM itself.
Now, solve the problem. Walk backward from the future: compute the value of every state by reusing values you've already computed one step ahead. That's dynamic programming.
Two doors at every step. Each has a peephole — peek to see the reward inside. Step through to claim it; two new doors appear. 4 picks total. Highest score wins.
The peephole shows only this step's reward — not what waits behind the next door. Easy to grab the big number now and lose later.
Take 4 steps. At each step, go left or right. Each cell pays a reward — a 🪙 coin (+5 to +10) or 💀 boss (−10 to −5). Goal: finish with the highest total. There are \(2^4 = 16\) possible paths.
Sum the rewards along the red path. What total does greedy end with?
What information would let you make the right choice at every step?
Greedy fails because it judges each move by this step's reward only. The fix in one sentence: at every step, pick the move whose whole future looks best. Today's choice is judged by where it eventually leads — all the way to the end.
"Take the biggest reward I can see now." Looks one step ahead. Walks straight into a boss two steps later.
"Take the move whose total future reward is highest." Looks all the way to the leaves before committing the first step.
A plan is a fixed 4-step sequence of L/R moves. There are \(2^4 = 16\) plans. For each one, sum the four rewards along its path; pick the plan with the highest total.
4 steps gave 16 plans. What if the warrior took 10 steps? 20? 50? How many plans must we sum?
\(2^{10}{=}1{,}024\) · \(2^{20}{\approx}10^6\) · \(2^{50}{\approx}10^{15}\). Exponential — brute force is hopeless almost immediately. We need a smarter way.
Greedy got trapped on the big tree because each move hides a long future. Strip the future to nothing — make the problem one step — and greedy snaps back to optimal. Three observations turn that into a plan.
If the tree is just one step deep, the best total reward starting from \(s\) is just the larger reward. Call this best total \(V(s)\) — the value of state \(s\). No future, no trap.
What if every next state \(s'\) came tagged with its value \(V(s')\) — the best total starting from there? Then any step is again a 1-step problem: pick the action with the biggest reward + value.
Same warrior tree (rewards untouched). Violet next to each node = V (state values). Red on each edge = Q (action values). The dashed path \(L\to LR\to LRL\to LRLR\) is the optimal route the math panel walks.
Same tree, same rewards. Suppose at every state π picks L with probability 0.6, R with 0.4. Max → expectation. Violet = \(V_\pi\). Red = \(Q_\pi\).
Finally, write it down as one equation: the value of a state equals the reward plus the value of the next state. Recursive, exact, and the foundation of every algorithm in the rest of the course.
Two values, one circular definition: V averages Q over actions; Q reaches forward through V. This recursion is the seed of Bellman's equation — the workhorse of RL we'll meet in L4.
The value of being at \(s\) is the average action value at \(s\), weighted by how often the policy picks each action.
The value of taking \(a\) from \(s\) is the reward you collect now plus the value of the next state.
So far each \((s, a)\) landed at one fixed \(s'\). In general the kernel \(P(s' \mid s, a)\) — from slide 9 — gives a distribution over next states. \(Q\) can't peek at one \(s'\); it has to average over them.
The warrior tree: each \((s, a)\) lands at exactly one \(s'\). \(V_\pi(s')\) is one number — no averaging needed.
Each \((s, a)\) gives a distribution over \(s'\). \(Q\) collects the reward, then takes the expected next-state value: \(\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[V_\pi(s')]\).
Slide 14 noticed something circular: \(V\) averages \(Q\) over actions; \(Q\) reaches forward through \(V\). Substitute one into the other and you get Bellman's equation — the most-used equation in all of RL.
One next state per \((s, a)\) — no averaging needed.
\((s, a)\) gives a distribution over \(s'\) — average \(V_\pi\) over them.
Setup for all three: state \(s\) has two actions — L: \(R = +5\), lands in \(s'\) with \(V_\cdot(s') = +10\). R: \(R = +12\), lands in \(s''\) with \(V_\cdot(s'') = -3\). Tap to reveal.
Use Bellman optimality. What is \(V^*(s)\)? Which action is optimal?
\(V^*(s) = \max\!\big(R_L + V^*(s'),\; R_R + V^*(s'')\big) = \max(5+10,\; 12-3) = \max(15, 9) = +15\). The bigger immediate reward (R = +12) loses to the better future (V = +10).
Now \(\pi\) picks L with prob 0.7, R with prob 0.3. What is \(V_\pi(s)\)?
\(V_\pi(s) = 0.7\,(5+10) + 0.3\,(12-3) = 0.7 \cdot 15 + 0.3 \cdot 9 = 10.5 + 2.7 = 13.2\). Average over actions, weighted by \(\pi\). Less than \(V^*\) because the random policy sometimes picks the worse action.
Same as Q1, but future value is now discounted by \(\gamma = 0.9\). Recompute \(V^*(s)\) and the optimal action.
\(V^*(s) = \max\!\big(5 + 0.9 \cdot 10,\; 12 + 0.9 \cdot (-3)\big) = \max(14,\; 9.3) = +14\). Discount shrinks both futures, but L still wins. (If \(\gamma\) were small enough — say \(\gamma = 0.3\) — the immediate +12 of R could overtake L.)
One equation, four jobs. Every algorithm we'll see for the rest of the course is some flavor of "satisfy Bellman."
\(V^*\) and \(Q^*\) are the unique fixed points of Bellman's optimality equation. The MDP's solution isn't a number — it's the function that satisfies this recursion. Solving Bellman = solving the MDP.
Treat Bellman as an update rule: \(V_{k+1}(s) \leftarrow \max_a [R(s,a) + \gamma \sum_{s'} P\, V_k(s')]\). Start anywhere; iterate; converge to \(V^*\). That's value iteration — coming next.
TD-learning, Q-learning, SARSA, DQN, AlphaGo's value net — all built on the recursion. The TD error \(\delta = R + \gamma V(s') - V(s)\) measures how far the current estimate strays from satisfying Bellman.
Gridworld, Atari, Go, robotic arms, chip-design RL, RLHF for LLMs — substrate changes, equation doesn't. If you remember one formula from this whole course, this is the one.
Every value-based RL algorithm is "satisfy Bellman." They differ on three knobs: do you know the model?, full expectations or single samples?, table or neural net?
When the MDP is fully known — the warrior tree, gridworld, inventory planning. L4 starts here.
When you can act in the world but don't know \(R\) or \(P\) — robotic learning, simple games. The bracket is the TD error: how far Q is from satisfying Bellman.
When the state space is huge — Atari from pixels (DeepMind, 2013), Go (AlphaGo). The neural net learns to satisfy Bellman.
Both halves are Bellman: eval uses expectation, improve uses optimality. Often fewer iterations than VI — control theory, dynamic programming.
Same equation, four solvers. Pick the one that matches what you have — model? compute budget? state size? — and the rest of RL fans out from there.
We named every part of the loop, met \(V\) and \(Q\), and wrote down Bellman's equation — the unique fixed point that is the MDP's solution.
Turn Bellman into an algorithm: iterate the equation until the values stop changing. That gives \(V_\pi\) for any policy, and \(V^*\) for the optimal one.