Last time we drew the agent–environment loop. Today we build the toolkit we'll use to formalize it next time — outcomes, random variables, expectation, conditional probability, ending with a tiny game that bridges from L1's bandit to next week's MDP.
Four ideas from last week — the picture, before we add the math today.
Many labeled examples, minimize the error. Powers most AI you use today — spam filters, photo tagging, translation.
No labels. The policy reads the current state to choose an action; that action plus the state drives where you land next, and the environment feeds back a reward.
Every learner — even you in the 3-box game — must balance trying new things against using what already works.
AlphaGo's Move 37 wasn't taught — it was discovered through self-play. RL invents strategies no teacher knows.
Before we can talk about random variables, we need to fix the playground. Every probability problem rests on three things: outcomes, events, and a probability rule.
The set of all possible outcomes of one trial. A single outcome is called \(\omega\). Outcomes need not be numbers — they can be abstract labels like heads or tails. (Some books write \(S\) instead of \(\Omega\) — same idea.)
A collection of outcomes from \(\Omega\) — anything you might ask about. (Braces matter: \(\{H\}\) is the event "got heads.")
A function of events. Rules: \(P(\Omega)=1\); and \(P(A\cup B)=P(A)+P(B)\) when \(A,B\) are disjoint. A handy consequence — the complement: \(P(A^c)=1-P(A)\).
† Math name: such a function of the events is called a (probability) measure.
Try this question: "In 100 coin flips, how many heads will I see?" Outcomes like H and T aren't numbers — you can't add them, average them, or plot them. Most real sample spaces are like this — weather conditions, cards drawn, songs played. So we build a tiny bridge: assign a number to every outcome, and the messy world becomes calculable.
One number per outcome — that's the whole job of \(X\). Here it encodes a yes/no question: "did we see a head?"
Over 5 flips let \(X_i = 1\) if flip \(i\) is heads, else \(0\). Turn each English event into a condition on the \(X_i\).
Event: exactly 2 heads in 5 flips.
\(\sum_{i=1}^{5} X_i = 2\)The head count is \(X_1+\cdots+X_5\); "exactly 2 heads" pins that sum to 2.
Event: at least one head in 5 flips. Write it.
"At least one" means the head count is 1 or more.
Event: more heads than tails in 5 flips. Write it.
With 5 flips, "more heads than tails" needs at least 3 heads.
Once events are conditions on numbers, we can ask how likely each value is — and what value to expect on average. Next slide.
Once \(X\) is a random variable, two questions follow: how often does it take each value, and what value should we expect on average?
For each possible value \(x\), the chance that \(X\) equals it. The whole list sums to 1.
A weighted average of the values — each value \(x\) weighted by its probability \(p_X(x)\). More-probable values pull the average toward them. There's no notion of time here — it's just a sum. (How this connects to real repeated trials → next slide.)
Why we care: RL agents maximize \(\mathbb{E}[G_t]\) — the expected total reward, not any single outcome. The symbol \(\mathbb{E}\) will show up in nearly every formula from here on.
We keep writing \(P(H)=\tfrac12\) — but what does that number mean? Flip a coin \(n\) times and track the frequency of heads. As \(n\) grows, that fraction stops wandering and settles onto the probability.
The empirical fraction — it changes with every flip.
The fixed value the frequency settles on — the law of large numbers. It's why a weighted average (expectation) predicts real performance.
† Law of large numbers — Jacob Bernoulli, Ars Conjectandi (1713); strong form: Borel (1909), Kolmogorov (1933).
We computed \(\mathbb{E}[X]=3.5\) for a fair die as a weighted average. Now roll it for real: the empirical mean of the numbers you roll converges to that very value.
The running average of the numbers you actually rolled — it jumps around at first.
The weighted average from the last slide, now confirmed by experiment. This is why RL can optimize \(\mathbb{E}\).
† Law of large numbers — Jacob Bernoulli, Ars Conjectandi (1713); strong form: Borel (1909), Kolmogorov (1933).
Probabilities update when you learn something. If I tell you it's cloudy outside, your guess that it'll rain rises. We write the updated probability as \(P(A \mid B)\) — read aloud "probability of \(A\) given \(B\)."
Knowing \(B\) jumps the chance: \(\tfrac{1}{13} \to \tfrac{1}{3}\).
Each is a whole conditional distribution — a probability for every next state \(s'\) (or action \(a\)), given what we condition on. Every | asks: "what's likely next, given what we know now?"
That's our probability toolkit. Now we use it to formalize every part of the agent–environment loop.
Q1 is a worked demo — the recipe is always: ① what is \(\Omega\)? ② model the event ③ compute \(P\). Tap Q2 and Q3 to check yourself.
Flip 3 fair coins. Find \(P(\text{exactly 2 heads})\).
\(P = 3/8\)
① Sample space. \(\Omega = \{HHH,HHT,HTH,HTT,THH,THT,TTH,TTT\}\); \(|\Omega| = 2^3 = 8\), all equally likely.
② Model the event. \(A = \{HHT, HTH, THH\}\), so \(|A| = 3\).
③ Probability. \(P(A) = \dfrac{|A|}{|\Omega|} = \dfrac{3}{8}.\)
Roll two fair dice. What is \(P(\text{sum} = 7)\)? Work the three steps: what is \(\Omega\), which outcomes form the event, then the probability.
① \(\Omega\). Ordered pairs \((d_1,d_2)\), \(d_i\in\{1,\ldots,6\}\); \(|\Omega| = 36\).
② Event. \(A = \{(1,6),(2,5),(3,4),(4,3),(5,2),(6,1)\}\), \(|A| = 6\).
③ Probability. \(P(A) = \dfrac{6}{36} = \dfrac{1}{6}.\)
Two chests, you can't tell them apart: 🎁 a Legendary (rare drop 80%) and 📦 a Standard (50%), each equally likely. You get 2 pulls. Pull #1 — RARE! 🎉 For pull #2: stick or switch?
You can't tell the chests apart, so each is Legendary or Standard with equal chance: \(P(\text{Leg}) = P(\text{Std}) = \tfrac12.\)
Average the two rare-rates over your prior (law of total probability):
\(P(\text{rare}) = \underbrace{\tfrac12(0.8)}_{\text{Leg branch}} + \underbrace{\tfrac12(0.5)}_{\text{Std branch}} = 0.65.\)
\(P(\text{Leg}\mid\text{rare}) = \dfrac{P(\text{rare}\mid\text{Leg})\,P(\text{Leg})}{P(\text{rare})} = \dfrac{0.8 \cdot 0.5}{0.65} = \tfrac{8}{13} \approx 62\%.\)
So \(P(\text{Std}\mid\text{rare}) = \tfrac{5}{13}\). The rare made Legendary more believable: \(50\% \to 62\%\).
Mix the rare-rates by your updated belief:
\(\underbrace{0.8}_{\text{Leg drops rare}}\cdot\underbrace{\tfrac{8}{13}}_{\text{belief it's Leg}} + \underbrace{0.5}_{\text{Std drops rare}}\cdot\underbrace{\tfrac{5}{13}}_{\text{belief it's Std}} = \tfrac{8.9}{13} \approx 68\%.\)
Each term reads: (this chest's rare-rate) × (your belief you're holding it). E.g. \(0.8 \cdot \tfrac{8}{13}\) = "I'm \(\tfrac{8}{13}\) sure it's Legendary, and a Legendary drops rare 80% of the time."
If this one is Legendary the other is Standard, and vice-versa — so the other chest is Legendary with prob \(\tfrac{5}{13}\):
\(0.8\cdot\tfrac{5}{13} + 0.5\cdot\tfrac{8}{13} = \tfrac{8}{13} \approx 62\%.\)
The chest problem looked a lot like the 3-box bandit you played in Lecture 1. Where do they actually diverge?
You're given the rates — 🎁 80% rare, 📦 50% rare. You only need to figure out which chest is which.
Bayesian update is exact: one observation directly maps to a posterior.
You don't know any box's drop rate. You learn the rates only by pulling — every action also produces data.
No closed-form answer. Explore-vs-exploit kicks in. ε-greedy lives here.
The next nine slides peek under the hood — σ-algebras, measures, almost-sure convergence, the geometry of random variables, Bayesian vs frequentist views, and even a tour of the Bayesian brain. Interestingly abstract (and honestly, hard).
If you don't follow everything — that's totally okay. None of this is a prerequisite for the rest of the course. Skim it, savor it, or skip ahead. You can always come back.
The aim isn't mastery — it's a guided look at how mathematicians think about probability.
The probability space is a three-piece object. We've used \(\Omega\) and \(P\) all along; the hidden third piece is \(\mathcal{F}\) — the family of allowable events. Together they form the probability triple.
Set of all possible outcomes \(\omega\). A single trial picks one.
A family of subsets of \(\Omega\) — the events you're allowed to ask about. Closed under complement, closed under countable union, and contains \(\Omega\).
Assigns a number to each event in \(\mathcal{F}\). Rules: \(P(\Omega) = 1\); for disjoint \(A_1, A_2, \ldots \in \mathcal{F}\), \(P\!\left(\bigcup_i A_i\right) = \sum_i P(A_i)\) (countable additivity).
Two ideas fit together. Events are sets — so logic on events is just set algebra. Probability is a "size function" on those sets — like length, area, or volume. The whole machinery has a name: measure theory.
A σ-algebra is closed under \(\cup, \cap, ^c\) — exactly so the events you can talk about stay in the family.
A measure on \((\Omega, \mathcal{F})\) is any function \(\mu : \mathcal{F} \to [0, \infty]\) with:
A probability measure adds one rule: \(P(\Omega) = 1\).
Total "mass" equals 1 — the only thing that makes probability special.
Studying all of these together is measure theory (Lebesgue, 1902) — one foundation under modern integrals, probability, and signal processing.
Remember slides 6–7? The coin frequency settled near \(\tfrac12\), the dice average settled near 3.5 — but neither stopped wobbling. How do you state "it converges" precisely, when the path keeps fluctuating? Two answers, two strengths — both made possible by Kolmogorov's measure theory (1933).
Pick a massive \(n\) (say \(10^6\)). The probability that the running average at that exact step is far from \(\tfrac12\) is nearly zero.
A statement about fixed-time probability. Says nothing about whole paths.
Run the experiment forever. The sequence of running averages converges to \(\tfrac12\) for every universe, except a collection of pathological universes whose total probability is \(0\).
A statement about whole paths. To even ask "what's the measure of the bad set?" you need \(\mathcal{F}\).
Pre-Kolmogorov, dice problems and continuous random variables (gas-particle velocities, daily temperatures, stock prices) needed different machinery — counting on one side, calculus on the other. After 1933, they're the same object: a probability measure on a measurable space. Different sets, identical axioms.
🪙 Coin flip · 🎲 Dice · 🎴 Cards · 📦 Bandit arms.
Sums. Counting measure underneath.
🌡 Temperature · 💹 Stock price · 🏃 Reaction time · 📡 Sensor noise.
Integrals. Lebesgue measure underneath.
One Lebesgue integral against the measure \(P\). Whether \(\Omega\) is discrete or continuous, the formula is the same — the measure does the work. LLN, CLT, Bayes all hold simultaneously in both worlds.
Sets become spaces; spaces inherit geometry. Once \(L^2\) — the space of square-integrable random variables — is the playground, abstract probability concepts turn into familiar vector geometry.
In \(L^2\), an inner product encodes covariance:
Uncorrelated ↔ orthogonal: \(\langle X, Y \rangle = 0\) iff \(\mathrm{Cov}(X,Y) = 0\) (after centering).
† Wait — aren't random variables functions?
Yes: \(X : \Omega \to \mathbb{R}\). And functions are vectors — they live inside an abstract, infinite-dimensional vector space (here, \(L^2\)). You can add them, scale them, take inner products. "Function" and "vector" aren't a contradiction; the second is a coordinate-free way to think about the first. (learn more in functional analysis)
\(\mathbb{E}[X \mid \mathcal{G}]\) is the orthogonal projection of \(X\) onto the subspace of \(\mathcal{G}\)-measurable random variables.
The "best guess of \(X\) using only info \(\mathcal{G}\)" is, geometrically, the closest point in that subspace — exactly like dropping a perpendicular.
A random variable \(X : \Omega \to \mathbb{R}\) doesn't just turn outcomes into numbers — it carries the probability with it. To ask "what's the chance \(X \in B\)?" we pull \(B\) back to a set of outcomes in \(\Omega\), where probability already lives.
\(\Omega = \{HH, HT, TH, TT\}\), each outcome \(\tfrac{1}{4}\). Define \(X(\omega)\) = number of heads.
The distribution we've been calling \(p_X\) is exactly \(P_X\).
Two random variables can have the same distribution on totally different sample spaces. Your dice game and a casino's roulette can share \(P_X\), even though their \(\Omega\) and \(P\) look nothing alike.
Expectation can be computed in either space:
For \(P_X\) to be a valid measure, \(X\) must be measurable: \(X^{-1}(B) \in \mathcal{F}\) for every Borel \(B\).
Same axioms (Kolmogorov), same equations — two interpretations of what \(P(A)\) actually means. The split runs through 250 years of statistics and shows up everywhere in modern AI.
Probability is a long-run frequency. Parameters are fixed but unknown; data is random. We use confidence intervals, p-values, and maximum likelihood. Slide 6's coin demo lives in this view.
Trouble: requires a "repeatable experiment." Hard to assign \(P(\text{rain tomorrow})\) — there's only one tomorrow.
Probability is a degree of belief. Parameters are uncertain (random); data is observed (fixed). We use priors, posteriors, credible intervals. Slide 9's chest puzzle lives in this view.
Trouble: where does the prior come from? Two analysts with different priors can read the same data and reach different posteriors.
The Bayesian brain hypothesis (Helmholtz, 1860s; revived by Knill, Pouget, and Friston in the 2000s) says perception and decisions are posterior inferences: prior expectations meet sensory likelihood, and we act on the result. Same machinery — beautiful when it works, buggy when it doesn't.
Outcomes, random variables, expectation, conditional probability, and Bayes — every | in RL is one of these. The bandit teaser shows where they meet RL.
State, action, reward, transition, policy — bundled into the MDP. Then we play a doors game and meet two values, V and Q.