Bellman Optimality Equation

See Bellman Optimality Equations for the full treatment.

For $v_{\ast }$

$v_{\ast }(s)={\max }_a{\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma v_{\ast }(s^{\prime })]$

For $q_{\ast }$

$q_{\ast }(s,a)={\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)\left[r+\gamma {\max }_{a^{\prime }}{q}_{\ast }(s^{\prime },a^{\prime })\right]$

Solved by: Value Iteration, Policy Iteration, Q-Learning (sample-based)

Appears In

• RL-L01 - Intro, MDPs & Bandits, RL-L02 - Dynamic Programming