RL-Book Ch3 - Finite MDPs

Chapter 3: Finite Markov Decision Processes

Overview

This chapter formalizes the problem of sequential decision making through the framework of Finite Markov Decision Processes (MDPs). Unlike bandit problems, MDPs involve an associative aspect—choosing different actions in different situations—and account for delayed rewards, where current actions influence future states and thus future rewards.

3.1 The Agent–Environment Interface

The Reinforcement Learning problem is framed as a continuous interaction between two entities:

• Agent: The learner and decision-maker.
• Environment: Everything outside the agent.

The Interaction Loop

At each discrete time step $t=0,1,2,\dots$:

1. The agent receives a representation of the environment’s state, $S_t\in \mathcal{S}$.
2. The agent selects an action, $A_t\in \mathcal{A}(s)$.
3. One time step later, the agent receives a numerical reward, $R_{t+1}\in \mathcal{R}\subset \mathbb{R}$, and finds itself in a new state $S_{t+1}$.

This creates a trajectory: $S_0,A_0,R_1,S_1,A_1,R_2,S_2,A_2,R_3,\dots$

Dynamics Function

In a Finite MDP, the sets $\mathcal{S},\mathcal{A},$ and $\mathcal{R}$ are finite. The environment’s dynamics are completely defined by the probability: $p(s^{\prime },r|s,a)\doteq \mathrm{Pr}\{S_t=s^{\prime },R_t=r\mid S_{t-1}=s,A_{t-1}=a\}$ for all $s^{\prime },s\in \mathcal{S}$, $r\in \mathcal{R}$, and $a\in \mathcal{A}(s)$.

Derived Dynamics Functions

• State-transition probabilities: $p(s^{\prime }|s,a)={\sum }_{r\in \mathcal{R}}p(s^{\prime },r|s,a)$
• Expected rewards for state-action pairs: $r(s,a)={\sum }_{r\in \mathcal{R}}r{\sum }_{s^{\prime }\in \mathcal{S}}p(s^{\prime },r|s,a)$
• Expected rewards for state-action-next-state triples: $r(s,a,s^{\prime })={\sum }_{r\in \mathcal{R}}r\frac{p(s^{\prime },r|s,a)}{p(s^{\prime }|s,a)}$

The Markov Property

The state $S_t$ must include information about all aspects of the past that matter for the future. If the transition probabilities depend only on the immediately preceding state and action, the state is said to have the Markov Property.

Recycling Robot

A robot collects empty cans. States: $\mathcal{S}=\{high,low\}$ battery levels. Actions: $\{search,wait,recharge\}$.

• Searching retrieves cans (reward) but drains battery.
• Waiting retrieves fewer cans but saves battery.
• Recharging is only possible when battery is low.
• Transition Graph: State nodes (large circles) and Action nodes (small solid circles) show the probabilities $p(s^{\prime }|s,a)$ and rewards $r$.

3.2 Goals and Rewards

The agent’s goal is to maximize the cumulative reward in the long run.

The Reward Hypothesis

All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward).

Key Insight: Rewards should communicate what you want the agent to achieve, not how you want it achieved. Sub-goals (e.g., in Chess) should not be rewarded directly if they might conflict with the ultimate goal (winning).

3.3 Returns and Episodes

The Return, $G_t$, is the specific function of the reward sequence the agent seeks to maximize.

Episodic Tasks

Tasks that break into finite subsequences ending in a terminal state. $G_t\doteq R_{t+1}+R_{t+2}+\cdots +R_T$

Continuing Tasks

Tasks that go on without limit ($T=\infty$). We must use a Discount Factor $\gamma \in [0,1]$ to keep the return finite. $G_t\doteq R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots ={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$

Recursive Relation of Returns

$G_t=R_{t+1}+\gamma G_{t+1}$

3.4 Unified Notation

We unify episodic and continuing tasks by treating episode termination as entering a special absorbing state that transitions only to itself and generates zero rewards. This allows using the discounted sum formula for both.

3.5 Policies and Value Functions

A Policy, $\pi$, is a mapping from states to probabilities of selecting each possible action: $\pi (a|s)$.

A Value Function estimates “how good” it is to be in a certain state or perform a certain action.

State-Value Function ( $v_{\pi }$)

The expected return when starting in state $s$ and following policy $\pi$: $v_{\pi }(s)\doteq {\mathbb{E}}_{\pi }[G_t\mid S_t=s]={\mathbb{E}}_{\pi }\left[{\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|S_t=s\right]$

Action-Value Function ( $q_{\pi }$)

The expected return starting from $s$, taking action $a$, and thereafter following policy $\pi$: $q_{\pi }(s,a)\doteq {\mathbb{E}}_{\pi }[G_t\mid S_t=s,A_t=a]$

The Bellman Equation for $v_{\pi }$

Values of states satisfy recursive relationships.

Bellman Equation

$v_{\pi }(s)={\sum }_a\pi (a|s){\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\pi }(s^{\prime })]$

Backup Diagrams

Backup diagrams show the flow of value. For $v_{\pi }$:

• Root node: State $s$ (open circle).
• Branches: Actions $a$ chosen via $\pi$.
• Leaves: Next states $s^{\prime }$ (open circles) and rewards $r$ determined by $p$. The value of $s$ is the average of the discounted values of successor states plus expected immediate rewards.

Gridworld

• Dynamics: 5x5 grid. Actions: N, S, E, W. Bumping into the edge results in -1 reward.
• Special States: Transitions from A give +10 and land in A’. Transitions from B give +5 and land in B’.
• Value Function: Under a random policy ($\pi =0.25$ for all actions), states near edges have lower/negative values, while states near A have high values.

3.6 Optimal Policies and Optimal Value Functions

An Optimal Policy, ${\pi }_{\ast }$, is better than or equal to all other policies (${\pi }_{\ast }\ge \pi ⟺v_{{\pi }_{\ast }}(s)\ge v_{\pi }(s)$ for all $s\in \mathcal{S}$).

Optimal Value Functions

• Optimal State-Value: $v_{\ast }(s)\doteq {\max }_{\pi }{v}_{\pi }(s)$
• Optimal Action-Value: $q_{\ast }(s,a)\doteq {\max }_{\pi }{q}_{\pi }(s,a)$
  • Note: $q_{\ast }(s,a)=\mathbb{E}[R_{t+1}+\gamma v_{\ast }(S_{t+1})\mid S_t=s,A_t=a]$

The Bellman Optimality Equation

Bellman Optimality Equation for $v_{\ast }$

$v_{\ast }(s)={\max }_a{\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\ast }(s^{\prime })]$

Bellman Optimality Equation for $q_{\ast }$

$q_{\ast }(s,a)={\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma {\max }_{a^{\prime }}{q}_{\ast }(s^{\prime },a^{\prime })]$

Greedy Policies

Once $v_{\ast }$ is known, the optimal policy is greedy with respect to it. Because $v_{\ast }$ already accounts for all future rewards, a local one-step search is sufficient for global optimality. With $q_{\ast }$, even the one-step search is unnecessary; the agent simply picks $\arg {\max }_a{q}_{\ast }(s,a)$.

3.7 Optimality and Approximation

Finding the exact solution to the Bellman Optimality Equation is often impossible due to:

1. Model Uncertainty: We don’t know the dynamics $p$.
2. Computational Constraints: The State Space is too large (e.g., Chess, Backgammon).
3. Memory Constraints: We cannot store a table for all states.

RL methods focus on approximating $v_{\ast }$ and ${\pi }_{\ast }$, often prioritizing frequently visited states to make the best use of limited resources.

Summary

• MDPs provide a mathematical framework for goal-directed learning.
• Value functions ($v$ and $q$) are central, representing expected future returns.
• Bellman equations provide the recursive structure needed to compute these values.
• Optimality is an ideal reached through greedy behavior relative to optimal value functions.
• RL agents must usually settle for approximations due to the curse of dimensionality and unknown dynamics.