RL-L07 - Off-Policy RL with Approximation

RL Lecture 7: Off-Policy RL with Approximation

Introduction

The extension of reinforcement learning to the off-policy case with function approximation is significantly harder than the on-policy case. While tabular off-policy methods extend readily to semi-gradient algorithms, these algorithms do not converge as robustly and can often diverge. This note explores the “Deadly Triad” of instability, Baird’s counterexample, and advanced algorithms designed to provide stability.

---

1. Episodic Semi-Gradient Control (Ch 10.1)

To extend function approximation to control, we use semi-gradient descent to update the action-value function $q(s,a,\mathbf{w})$.

Semi-Gradient Sarsa update

The update rule for the weights $\mathbf{w}$ in semi-gradient Sarsa is:

Formula

${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\alpha [R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},{\mathbf{w}}_t)-\hat{q}(S_t,A_t,{\mathbf{w}}_t)]\nabla \hat{q}(S_t,A_t,{\mathbf{w}}_t)$ Where:

• ${\delta }_t=R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},{\mathbf{w}}_t)-\hat{q}(S_t,A_t,{\mathbf{w}}_t)$ is the TD error.
• $\nabla \hat{q}(S_t,A_t,{\mathbf{w}}_t)$ is the gradient with respect to $\mathbf{w}$.

Pseudocode: Episodic Semi-gradient Sarsa

Definition

Input: a differentiable action-value function parameterization $\hat{q}(s,a,\mathbf{w})$ Algorithm parameters: step size $\alpha >0$, small $ϵ>0$ Initialize: weight vector $\mathbf{w}\in {\mathbb{R}}^d$ arbitrarily

Loop for each episode:   Initialize $S$   Choose $A\sim \pi (\cdot |S)$ (e.g., $ϵ$-greedy)   Loop for each step of episode:     Take action $A$, observe $R,S^{\prime }$     If $S^{\prime }$ is terminal:       $\mathbf{w}\leftarrow \mathbf{w}+\alpha [R-\hat{q}(S,A,\mathbf{w})]\nabla \hat{q}(S,A,\mathbf{w})$       Go to next episode     Choose $A^{\prime }\sim \pi (\cdot |S^{\prime })$     $\mathbf{w}\leftarrow \mathbf{w}+\alpha [R+\gamma \hat{q}(S^{\prime },A^{\prime },\mathbf{w})-\hat{q}(S,A,\mathbf{w})]\nabla \hat{q}(S,A,\mathbf{w})$     $S\leftarrow S^{\prime }$, $A\leftarrow A^{\prime }$

---

2. Semi-Gradient Off-Policy Methods (Ch 11.1)

Off-policy methods learn a target policy $\pi$ from experience generated by a behavior policy $b$.

Importance Sampling Ratio

To account for the discrepancy between $\pi$ and $b$, we use the per-step importance sampling ratio:

Formula

${\rho }_t\doteq \frac{\pi (A_t|S_t)}{b(A_t|S_t)}$

Off-Policy Semi-Gradient TD(0) update

Formula

${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\alpha {\rho }_t{\delta }_t\nabla \hat{v}(S_t,{\mathbf{w}}_t)$ Where ${\delta }_t=R_{t+1}+\gamma \hat{v}(S_{t+1},{\mathbf{w}}_t)-\hat{v}(S_t,{\mathbf{w}}_t)$.

---

3. Examples of Off-Policy Divergence (Ch 11.2)

Instability in off-policy learning can occur even in simple MDPs with linear function approximation.

Simple 1D Example

Consider a two-state MDP with values $w$ and $2w$. A transition from the first state to the second with reward 0 gives TD error $\delta =2w-w=w$. Updating $w$ with $\alpha =0.1$ gives $w\leftarrow 1.1w$. Since both states’ values depend on $w$, the error in the next step will be larger, leading to divergence if $\gamma >0.5$.

Baird’s Counterexample

This is a famous 7-state episodic MDP designed to show divergence.

The Setup

• States: 6 “upper” states and 1 “lower” state (state 7).
• Actions:
  • Dashed action: Transitions to one of the 6 upper states with equal probability.
  • Solid action: Transitions to the 7th state.
• Policies:
  • Behavior policy $b$: selects dashed ($6/7$) and solid ($1/7$) actions such that the next-state distribution is uniform.
  • Target policy $\pi$: always takes the solid action.
• Features: Linear function approximation where $v(s_i)=2w_i+w_8$ for $i=\mathrm{1..6}$ and $v(s_7)=w_7+2w_8$.
• Result: Semi-gradient TD(0) and even DP updates diverge to infinity for any positive $\alpha$, as $w_8$ grows unboundedly.

---

4. The Deadly Triad (Ch 11.3)

Instability arises when three specific elements are combined. Giving up any one removes the danger of divergence.

The Deadly Triad

1. Function Approximation: Using non-tabular representations (linear or ANNs).
2. Bootstrapping: Updating estimates based on other estimates (TD, DP).
3. Off-policy Training: Training on a distribution different from the target policy’s.

• Any two together are fine (e.g., On-policy learning with bootstrapping and FA is generally stable).
• All three together cause potential divergence in even the simplest cases.

---

5. Linear Value-Function Geometry (Ch 11.4)

Understanding the error surfaces in high-dimensional value function space.

Projection Operator ($\Pi$)

The operator that maps an arbitrary value function $v$ to the representable subspace that is closest in the weighted norm $\Vert \cdot {\Vert }_{\mu }$: $\Pi v=\mathbf{X}({\mathbf{X}}^{\top }\mathbf{DX}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{D}v$

Error Measures

• Value Error (VE): Distance to the true value function $v_{\pi }$. $VE(\mathbf{w})=\Vert v_{\mathbf{w}}-v_{\pi }{\Vert }_{\mu }^2$
• Bellman Error (BE): The expectation of the TD error. ${\bar{\delta }}_{\mathbf{w}}=B_{\pi }{v}_{\mathbf{w}}-v_{\mathbf{w}}$. $BE(\mathbf{w})=\Vert {\bar{\delta }}_{\mathbf{w}}{\Vert }_{\mu }^2$
• Projected Bellman Error (PBE): The Bellman error projected back into the representable subspace. $PBE(\mathbf{w})=\Vert \Pi {\bar{\delta }}_{\mathbf{w}}{\Vert }_{\mu }^2$

Intuition

The TD fixed point ${\mathbf{w}}_{TD}$ is the point where the PBE is exactly zero.

---

6. Gradient Descent in Bellman Error (Ch 11.5)

Naive minimization of the Bellman Error doesn’t work well because it is not easily learnable from samples without independent “double sampling” of the next state.

Residual-Gradient Algorithm

A true SGD algorithm for minimizing the MSBE.

Formula

${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\alpha [{\rho }_t(R_{t+1}+\gamma \hat{v}(S_{t+1},\mathbf{w})-\hat{v}(S_t,\mathbf{w}))][{\rho }_t(\nabla \hat{v}(S_t,\mathbf{w})-\gamma \nabla \hat{v}(S_{t+1},\mathbf{w}))]$

Problem: Requires two independent samples of $S_{t+1}$ from $S_t$ to be unbiased. “Naive” residual gradient (using the same sample twice) leads to poor convergence (e.g., the A-split example).

---

7. Bellman Error is Not Learnable (Ch 11.6)

The Bellman Error (BE) cannot be estimated from state-to-state transition data alone; it requires knowledge of the underlying MDP dynamics.

Learnability

A quantity is learnable if it can be determined from the observed distribution of experience (states, actions, rewards).

• VE is not learnable, but its minimizer is (it’s the same as the Return Error minimizer).
• BE is not learnable, and its minimizer is also not learnable from experience alone.

---

8. Gradient-TD Methods (Ch 11.7)

These methods minimize the Projected Bellman Error (PBE) and are true SGD methods of $O(d)$ complexity. They use a second weight vector $\mathbf{v}$ to estimate a part of the gradient.

GTD2

Formula

${\mathbf{v}}_{t+1}={\mathbf{v}}_t+\beta [{\rho }_t{\delta }_t-{\mathbf{v}}_t^{\top }{\mathbf{x}}_t]{\mathbf{x}}_t$ ${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\alpha {\rho }_t({\mathbf{x}}_t-\gamma {\mathbf{x}}_{t+1})({\mathbf{x}}_t^{\top }{\mathbf{v}}_t)$

TDC (Temporal-Difference with Correction)

Also known as GTD(0).

Formula

${\mathbf{v}}_{t+1}={\mathbf{v}}_t+\beta [{\rho }_t{\delta }_t-{\mathbf{v}}_t^{\top }{\mathbf{x}}_t]{\mathbf{x}}_t$ ${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\alpha {\rho }_t[{\delta }_t{\mathbf{x}}_t-\gamma {\mathbf{x}}_{t+1}({\mathbf{x}}_t^{\top }{\mathbf{v}}_t)]$

Key Properties

• Convergence: Guaranteed to the TD fixed point for linear FA even off-policy.
• Two Time-Scale: $\beta$ (the “fast” scale for $\mathbf{v}$) should be larger than $\alpha$ (the “slow” scale for $\mathbf{w}$).
• Complexity: $O(d)$, same as standard TD.

---

Created for the RL course vault.