In reinforcement learning, value-function methods such as Q-learning and SARSA(0) with $\epsilon$-greedy exploration are among the state of the art, and their tabular (exact) forms converge to the optimal Q-function under reasonable conditions.