Gaussian Policy

Definition

A Gaussian policy is a stochastic policy for continuous action spaces that models the action distribution as a multivariate Gaussian (normal) distribution:

$a\sim \mathcal{N}({\mu }_{\theta }(s),{\Sigma }_{\theta }(s))$

where:

• ${\mu }_{\theta }(s)\in {\mathbb{R}}^{|A|}$ is the mean (mean action), parameterized by $\theta$
• ${\Sigma }_{\theta }(s)$ is the covariance matrix (controls exploration magnitude)
• $a\in {\mathbb{R}}^{|A|}$ is the continuous action

Intuition

For continuous control (e.g., robot joint angles, continuous force), we need a policy that:

• Learns a preferred action (the mean)
• Maintains uncertainty/exploration around that mean
• Adjusts both mean and variance based on state

Gaussian policy naturally provides this: it’s differentiable, supported on ${\mathbb{R}}^{|A|}$, and captures both exploitation (mean) and exploration (variance).

Mathematical Formulation

Probability Density

For a diagonal Gaussian (common simplification):

${\pi }_{\theta }(a|s)={\prod }_{i=1}^{|A|}\frac{1}{\sqrt{2\pi {\sigma }_i^2}}\exp \left(-\frac{(a_i-{\mu }_i(s){)}^2}{2{\sigma }_i^2}\right)$

Log-Policy (for gradient computation)

$\log {\pi }_{\theta }(a|s)=-{\sum }_{i=1}^{|A|}\left[\frac{(a_i-{\mu }_i(s){)}^2}{2{\sigma }_i^2}+\log {\sigma }_i\right]+\text{const}$

Gradient w.r.t. Mean

${\nabla }_{\theta }\log {\pi }_{\theta }(a|s)=\frac{1}{{\sigma }^2}(a-{\mu }_{\theta }(s)){\nabla }_{\theta }{\mu }_{\theta }(s)$

Interpretation: Update mean in direction of the action error, scaled by inverse variance.

Gradient w.r.t. Variance

$\frac{\partial }{\partial \sigma }\log {\pi }_{\theta }(a|s)=\frac{(a-\mu (s){)}^2}{{\sigma }^3}-\frac{1}{\sigma }$

This shows variance should increase when actions are far from mean, decrease when close.

Key Properties/Variants

Mean Parameterization

Common choices:

1. Linear: ${\mu }_{\theta }(s)={\theta }^T\varphi (s)$

   • Simple, interpretable
   • Good for linear relationships

2. Neural network: ${\mu }_{\theta }(s)={\text{NN}}_{\theta }(s)$

   • Highly expressive
   • Standard for deep RL

Variance Parameterization

1. Fixed variance: $\sigma$ is a hyperparameter, not learned

   • Simpler, faster
   • May require careful tuning

2. Learned scalar variance: One $\sigma$ per dimension

   • Adapts exploration per action dimension
   • Common in practice

3. State-dependent variance: ${\sigma }_{\theta }(s)$ also learned

   • Maximum flexibility
   • Needs careful initialization

4. Log-variance: Often parameterize $\log \sigma$ to ensure positivity

Diagonal vs Full Covariance

• Diagonal (most common): $\Sigma =\text{diag}({\sigma }_1^2,\dots ,{\sigma }_d^2)$
  • Simpler gradient computation
  • Assumes action dimensions are independent
• Full covariance: Allows correlation between actions
  • More expressive, more expensive
  • Rarely needed

Connections

• Related to: Normal distribution, Continuous control
• Basis for: Policy Gradient Methods for continuous actions
• Alternative to: Softmax Policy (which is for discrete actions)
• Enables: Smooth, differentiable action sampling

Appears In

• Policy Gradient Methods — Standard for continuous action spaces
• REINFORCE — Continuous control variant
• Actor-Critic — Continuous action actor
• PPO — Continuous benchmark tasks
• Deep Deterministic Policy Gradient — Alternative to Gaussian (deterministic policy)
• Soft Actor-Critic (SAC) — Uses Gaussian policies with entropy regularization