RL-L02 - Dynamic Programming

RL Lecture 2: Dynamic Programming

Dynamic Programming (DP) refers to a collection of algorithms that can be used to compute optimal policies given a perfect model of the environment as a Markov Decision Process (MDP). While classical DP is limited by the requirement of a model and computational cost, it provides the theoretical foundation for almost all other RL methods, which can be viewed as approximations of DP.

Key Assumption

DP assumes the environment is a finite MDP with known dynamics $p(s^{\prime },r|s,a)$.

---

1. Unified Notation & Recap

To handle both episodic and continuing tasks simultaneously, we use a single notation for Returns:

• For episodic tasks, we consider termination as entering a special absorbing state that transitions only to itself with reward 0.
• This allows us to use the infinite sum notation with the possibility of $\gamma =1$ if all episodes eventually terminate.

Discounted Return

$G_t={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$ where:

• $G_t$ — the total discounted return from time $t$.
• $\gamma \in [0,1]$ — Discount Factor.
• $R_{t+k+1}$ — reward at step $t+k+1$.

---

2. Policy Evaluation (Prediction)

Policy Evaluation is the computation of the state-value function $v_{\pi }$ for an arbitrary policy $\pi$.

Bellman Equation for $v_{\pi }$

$v_{\pi }(s)={\sum }_a\pi (a|s){\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\pi }(s^{\prime })]$ Term-by-term explanation:

• $v_{\pi }(s)$: Value of state $s$ under policy $\pi$.
• ${\sum }_a\pi (a|s)$: Expectation over actions chosen by policy $\pi$.
• ${\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)$: Expectation over next states $s^{\prime }$ and rewards $r$ given action $a$ (dynamics).
• $[r+\gamma v_{\pi }(s^{\prime })]$: Immediate reward plus discounted value of the next state.

Iterative Policy Evaluation

If the dynamics are known, the Bellman equation forms a system of $|S|$ linear equations. We solve this iteratively.

Iterative Update Rule

$v_{k+1}(s)={\sum }_a\pi (a|s){\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_k(s^{\prime })]$ This is an expected update. The sequence $\{v_k\}$ converges to $v_{\pi }$ as $k\to \infty$ (under $\gamma <1$ or guaranteed termination).

Algorithm: Iterative Policy Evaluation

Algorithm: Iterative Policy Evaluation, for estimating V ≈ v_π
─────────────────────────────────────────────────────────────
Input: π (policy to be evaluated)
Parameter: θ > 0 (small threshold determining accuracy)
Initialize V(s) arbitrarily (e.g., 0), for all s ∈ S; V(terminal) = 0
 
Loop:
  Δ ← 0
  For each s ∈ S:
    v ← V(s)
    V(s) ← Σ_a π(a|s) Σ_{s',r} p(s', r|s, a) [r + γ V(s')]
    Δ ← max(Δ, |v - V(s)|)
until Δ < θ

Backup Diagram for $v_{\pi }$

       (s)          <-- State being updated
      / | \
     o  o  o        <-- Actions (a) chosen by policy π
    / \
  (s') (s')         <-- Possible next states (s') from dynamics p

The value flows back from the leaf nodes ($s^{\prime }$) through the actions to the root ($s$).

---

3. Policy Improvement

Once we have $v_{\pi }$, we want to find a better policy ${\pi }^{\prime }$.

Policy Improvement Theorem

Let $\pi$ and ${\pi }^{\prime }$ be any pair of deterministic policies such that, for all $s\in S$: $q_{\pi }(s,{\pi }^{\prime }(s))\ge v_{\pi }(s)$ Then ${\pi }^{\prime }$ must be as good as, or better than, $\pi$: $v_{{\pi }^{\prime }}(s)\ge v_{\pi }(s)\forall s\in S$

Proof Sketch

We start with $v_{\pi }(s)\le q_{\pi }(s,{\pi }^{\prime }(s))$ and repeatedly expand the right side:

1. $v_{\pi }(s)\le {\mathbb{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma v_{\pi }(S_{t+1})|S_t=s]$
2. Apply the inequality again: $v_{\pi }(s)\le {\mathbb{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma q_{\pi }(S_{t+1},{\pi }^{\prime }(S_{t+1}))|S_t=s]$
3. Continue expanding the expectation of rewards.
4. Eventually, the sum reaches $v_{{\pi }^{\prime }}(s)$.

Greedy Policy Improvement

A natural candidate for ${\pi }^{\prime }$ is the greedy policy with respect to $v_{\pi }$: ${\pi }^{\prime }(s)={\text{arg max}}_a{q}_{\pi }(s,a)={\text{arg max}}_a{\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\pi }(s^{\prime })]$

---

4. Policy Iteration

Policy Iteration is the alternating sequence of Evaluation and Improvement.

The PI Loop

${\pi }_0\stackrel{E}{\to }{v}_{{\pi }_0}\stackrel{I}{\to }{\pi }_1\stackrel{E}{\to }{v}_{{\pi }_1}\stackrel{I}{\to }{\pi }_2\stackrel{E}{\to }\cdots \stackrel{I}{\to }{\pi }_{\ast }\stackrel{E}{\to }{v}_{\ast }$ Because a finite MDP has only a finite number of deterministic policies, this process must converge to an optimal policy in finite steps.

Algorithm: Policy Iteration

Algorithm: Policy Iteration (for finding π ≈ π*)
─────────────────────────────────────────────────────────────
1. Initialization
   V(s) ∈ ℝ and π(s) ∈ A(s) arbitrarily for all s ∈ S
 
2. Policy Evaluation
   Loop:
     Δ ← 0
     For each s ∈ S:
       v ← V(s)
       V(s) ← Σ_{s',r} p(s', r|s, π(s)) [r + γ V(s')]
       Δ ← max(Δ, |v - V(s)|)
   until Δ < θ
 
3. Policy Improvement
   policy-stable ← true
   For each s ∈ S:
     old-action ← π(s)
     π(s) ← arg max_a Σ_{s',r} p(s', r|s, a) [r + γ V(s')]
     If old-action ≠ π(s), then policy-stable ← false
   If policy-stable, then stop; else go to step 2

---

5. Value Iteration

One drawback of Policy Iteration is that each evaluation step itself requires an iterative process. Value Iteration combines evaluation and improvement into a single sweep.

Value Iteration Update Rule

$v_{k+1}(s)={\max }_a{\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_k(s^{\prime })]$ This turns the Bellman Optimality Equation into an update rule.

Algorithm: Value Iteration

Algorithm: Value Iteration, for estimating π ≈ π*
─────────────────────────────────────────────────────────────
Initialize V arbitrarily (e.g., V(s) = 0), V(terminal) = 0
 
Loop:
  Δ ← 0
  For each s ∈ S:
    v ← V(s)
    V(s) ← max_a Σ_{s',r} p(s', r|s, a) [r + γ V(s')]
    Δ ← max(Δ, |v - V(s)|)
until Δ < θ
 
Output a deterministic policy π, such that
π(s) = arg max_a Σ_{s',r} p(s', r|s, a) [r + γ V(s')]

Relation to Policy Iteration

Value Iteration is equivalent to Policy Iteration with only one sweep of policy evaluation between improvement steps.

---

6. Generalized Policy Iteration (GPI)

GPI is the general framework describing any interaction of policy evaluation and policy improvement.

• Evaluation: Makes the value function consistent with the current policy.
• Improvement: Makes the policy greedy with respect to the current value function.

The GPI Diagram

Imagine two lines (or manifolds) in the value-policy space:

1. $v=v_{\pi }$: where values are consistent with the policy.
2. $\pi =\text{greedy}(v)$: where the policy is optimal for the values. Evaluation pulls us toward line 1; Improvement pulls us toward line 2. They compete and eventually intersect at $(v_{\ast },{\pi }_{\ast })$.

graph TD
    E[Policy Evaluation] --> |Estimate V| I[Policy Improvement]
    I --> |Greedy Policy| E
    style E fill:#f9f,stroke:#333
    style I fill:#ccf,stroke:#333

---

7. Asynchronous Dynamic Programming

A major drawback of classical DP is that it involves “sweeps” over the entire state set (updates all states). Asynchronous DP updates states in any order.

• Key idea: Use whatever values are available.
• Benefits: Can focus computation on “important” or frequently visited states. No need to wait for a full sweep to finish.
• Requirement: Must continue to update all states (none can be ignored forever) for convergence to $v_{\ast }$.

---

8. Summary of Bellman Equations

Name	Formula
Bellman Expectation ($v_{\pi }$)	$v_{\pi }(s)={\sum }_a\pi (a\Vert s){\sum }_{s^{\prime },r}p(s^{\prime },r\Vert s,a)[r+\gamma v_{\pi }(s^{\prime })]$
Bellman Expectation ($q_{\pi }$)	$q_{\pi }(s,a)={\sum }_{s^{\prime },r}p(s^{\prime },r\Vert s,a)[r+\gamma {\sum }_{a^{\prime }}\pi (a^{\prime }\Vert s^{\prime })q_{\pi }(s^{\prime },a^{\prime })]$
Bellman Optimality ($v_{\ast }$)	$v_{\ast }(s)={\max }_a{\sum }_{s^{\prime },r}p(s^{\prime },r\Vert s,a)[r+\gamma v_{\ast }(s^{\prime })]$
Bellman Optimality ($q_{\ast }$)	$q_{\ast }(s,a)={\sum }_{s^{\prime },r}p(s^{\prime },r\Vert s,a)[r+\gamma {\max }_{a^{\prime }}{q}_{\ast }(s^{\prime },a^{\prime })]$

---

9. Examples from Lecture

9.1 Gridworld (Iterative Policy Evaluation)

A 4x4 grid where transitions have reward -1 until termination.

• Under a random policy, the values represent the negative expected number of steps to the exit.
• After a single policy improvement step, the greedy policy already becomes optimal for this simple grid.

9.2 Transition Graph Stochasticity

Recycling Robot

Arcs are labeled with $(p,r)$: probability $p$ and reward $r$.

• If low battery, action search might transition to low (prob $\beta$) or result in “rescue” (transition to high, prob $1-\beta$, reward -3).
• This complexity is handled inherently by the expectation over $p(s^{\prime },r|s,a)$.

Big Picture of Policy Learning

1. Model-based: Learn dynamics $p(s^{\prime },r|s,a)$ then plan (DP).
2. Model-free Value-based: Learn $V(s)$ or $Q(s,a)$ directly (TD, Q-learning).
3. Model-free Policy-based: Directly optimize $\pi (a|s)$ (Policy Gradient).

---

Related Concepts:

• Markov Decision Process
• Bellman Equation
• Returns
• Policy Iteration
• Value Iteration
• Generalized Policy Iteration
• Optimality and Approximation

Book Reference: Sutton & Barto Ch 3.4-3.8, Ch 4.