Reparameterization Trick

Reparameterization Trick

A technique for computing gradients through stochastic sampling operations. Instead of sampling $a\sim {\pi }_{\theta }(\cdot |s)$ (which blocks gradient flow), express the sample as a deterministic, differentiable function of the parameters and independent noise: $a=f_{\theta }(ϵ;s)$ where $ϵ\sim \mathcal{N}(0,I)$.

Intuition

Making Randomness Differentiable

Normally, you can’t backpropagate through a random sampling step. The reparameterization trick moves the randomness into an input variable $ϵ$ that doesn’t depend on the parameters $\theta$. The actual sample becomes a deterministic transformation of $ϵ$, so gradients can flow through $f_{\theta }$.

For Gaussian Policies

If ${\pi }_{\theta }(a|s)=\mathcal{N}({\mu }_{\theta }(s),{\sigma }_{\theta }(s))$, then:

$a={\mu }_{\theta }(s)+{\sigma }_{\theta }(s)\odot ϵ,ϵ\sim \mathcal{N}(0,I)$

Now ${\nabla }_{\theta }a$ is well-defined: ${\nabla }_{\theta }a={\nabla }_{\theta }{\mu }_{\theta }(s)+{\nabla }_{\theta }{\sigma }_{\theta }(s)\odot ϵ$

Application in SAC

In Soft Actor-Critic (SAC), the reparameterization trick enables computing the policy gradient as:

${\nabla }_{\theta }J(\pi )={\nabla }_{\theta }{\mathbb{E}}_{ϵ}\left[Q(s,f_{\theta }(ϵ;s))-\alpha \log {\pi }_{\theta }(f_{\theta }(ϵ;s)|s)\right]$

The expectation over $ϵ$ doesn’t depend on $\theta$, so the gradient moves inside the expectation.

Key Properties

• Enables lower-variance gradient estimates compared to the log-derivative trick (REINFORCE)
• Works for continuous distributions that can be expressed as transformations of base distributions
• Standard technique in variational autoencoders (VAEs) and modern deep RL
• Requires the sampling distribution to be reparameterizable (Gaussian, etc.)

Connections

• Used in Soft Actor-Critic (SAC) for policy optimization
• Related to Policy Gradient Methods — an alternative way to compute policy gradients
• Also used in Variational Autoencoders (VAEs)
• Provides lower variance than the REINFORCE log-derivative estimator

Appears In

• RL-L11 - SAC, Decision Transformer & Diffuser