Advantage Function

Definition

The Advantage Function measures how much better taking a particular action is compared to the value of the state:

$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s)$

It answers: “How much does this action improve upon the average (state value)?”

---

Intuition

• Positive advantage: The action is better than the state baseline
• Zero advantage: The action is average (equal to state value)
• Negative advantage: The action is worse than baseline

Why It Matters

The advantage captures the relative quality of an action, not the absolute value. In policy gradient methods:

$\nabla J(\theta )\propto \mathbb{E}[\nabla \log \pi (a|s)Q(s,a)]$

Using advantage instead: $\nabla J(\theta )\propto \mathbb{E}[\nabla \log \pi (a|s)A(s,a)]$

Since $Q(s,a)=V(s)+A(s,a)$, we can rewrite: $\mathbb{E}[\nabla \log \pi (a|s)Q(s,a)]=\mathbb{E}[\nabla \log \pi (a|s)V(s)]+\mathbb{E}[\nabla \log \pi (a|s)A(s,a)]$

The first term is zero (policy’s gradient of state value is zero), leaving us with the advantage term.

---

Empirical Estimation

In practice, we don’t have access to $Q$ and $V$ directly. Common estimators:

1. Single-Step TD Advantage

${\hat{A}}_t=r_t+\gamma \hat{V}(s_{t+1})-\hat{V}(s_t)$

Also called the temporal difference error or TD residual.

Bias: Only one step into the future (underestimates long-term effects)
Variance: Low (short horizon)

2. Multi-Step Advantage

${\hat{A}}_t^{(n)}={\sum }_{l=0}^{n-1}{\gamma }^l{r}_{t+l}+{\gamma }^n\hat{V}(s_{t+n})-\hat{V}(s_t)$

Bias: Decreases with more steps
Variance: Increases with more steps

3. Monte Carlo Advantage

${\hat{A}}_t^{(\infty )}=G_t-\hat{V}(s_t)$

where $G_t={\sum }_{l=0}^{\infty }{\gamma }^l{r}_{t+l}$ (full episode return)

Bias: Unbiased
Variance: High (full trajectory variance)

---

Advantage in Actor-Critic

The Actor-Critic Update

${\theta }_{t+1}={\theta }_t+\alpha {\hat{A}}_t{\nabla }_{\theta }\log \pi (a_t|s_t,{\theta }_t)$

where ${\hat{A}}_t$ is the advantage estimate.

With Baseline

The baseline (typically the value function) is subtracted:

${\hat{A}}_t=R_{t+1}+\gamma \hat{V}(s_{t+1})-\hat{V}(s_t)$

Effect of baseline:

• ✓ Reduces variance (centering the signal)
• ✓ Remains unbiased (if value function is perfect)
• ✗ Introduces bias if value function is inaccurate

---

Generalized Advantage Estimation (GAE)

Rather than choosing one specific advantage estimator, GAE interpolates between all of them:

${\hat{A}}_t^{\text{GAE}(\gamma ,\lambda )}=(1-\lambda ){\sum }_{l=1}^{\infty }{\lambda }^{l-1}{\hat{A}}_t^{(l)}$

or equivalently:

${\hat{A}}_t^{\text{GAE}}={\sum }_{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}$

where ${\delta }_t=r_t+\gamma \hat{V}(s_{t+1})-\hat{V}(s_t)$ is the TD error.

Hyperparameter $\lambda$

• $\lambda =0$: Single-step TD advantage (low bias, low variance)
• $\lambda =1$: Monte Carlo advantage (unbiased, high variance)
• $\lambda \in (0,1)$: Interpolation (tunable bias-variance tradeoff)

Common choice: $\lambda =0.95$ or $\lambda =0.99$

---

Properties

Unbiased Property

If the critic is perfect ($\hat{V}=V^{\pi }$), then: $\mathbb{E}[{\hat{A}}_t]=Q^{\pi }(s_t,a_t)-V^{\pi }(s_t)=A^{\pi }(s_t,a_t)$

The advantage estimate is unbiased.

Variance Reduction

Subtracting the value function baseline reduces variance without introducing bias (with a perfect critic):

$\text{Var}[{\hat{A}}_t]<\text{Var}[{\hat{Q}}_t]$

Causality in TD

The TD advantage only uses future values starting from $s_{t+1}$:

${\hat{A}}_t=r_t+\gamma \hat{V}(s_{t+1})-\hat{V}(s_t)$

This respects causality: action at $t$ cannot affect rewards before time $t$.

---

Implementations

PyTorch Example

def compute_advantages(rewards, values, gamma=0.99, lambda=0.95):
    """Compute GAE advantages."""
    advantages = []
    gae = 0
    
    # Backward pass through trajectory
    for t in reversed(range(len(rewards))):
        if t == len(rewards) - 1:
            next_value = 0  # Terminal state
        else:
            next_value = values[t + 1]
        
        # TD error
        delta = rewards[t] + gamma * next_value - values[t]
        
        # Accumulate GAE
        gae = delta + gamma * lambda * gae
        advantages.insert(0, gae)
    
    return torch.tensor(advantages)

---

Comparison: Advantage Estimators

Estimator	Formula	Bias	Variance
1-step TD	$r_t+\gamma V(s_{t+1})-V(s_t)$	High	Low
n-step	${\sum }_{l=0}^{n-1}{\gamma }^l{r}_{t+l}+{\gamma }^{n}V(s_{t+n})-V(s_t)$	Medium	Medium
MC	$G_t-V(s_t)$	None	High
GAE	${\sum }_{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}$	Tunable	Tunable

---

Connections

• Used in: Actor-Critic, Policy Gradient Methods, PPO
• Related to: Value Function, Temporal Difference Learning, Generalized Advantage Estimation
• Appears in: A3C, A2C, SAC, TRPO

---

Key References

1. Schulman, G., et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR.
2. Mnih, V., et al. (2016). Asynchronous Methods for Deep Reinforcement Learning (A3C). ICML.
3. Schulman, G., et al. (2017). Proximal Policy Optimization Algorithms. Arxiv.