Expectation propagation

{{Short description|Method to approximate a probability distribution}}

Expectation propagation (EP) is a technique in Bayesian machine learning.{{Cite book|title=Pattern Recognition and Machine Learning|last=Bishop|first=Christopher|publisher=Springer-Verlag New York Inc.|year=2007|isbn=978-0387310732|location=New York}}

EP finds approximations to a probability distribution. It uses an iterative approach that uses the factorization structure of the target distribution. It differs from other Bayesian approximation approaches such as variational Bayesian methods.

More specifically, suppose we wish to approximate an intractable probability distribution $p(\backslash mathbf\{x\})$ with a tractable distribution $q(\backslash mathbf\{x\})$. Expectation propagation achieves this approximation by minimizing the Kullback-Leibler divergence $\backslash mathrm\{KL\}(p||q)$. Variational Bayesian methods minimize $\backslash mathrm\{KL\}(q||p)$ instead.

If $q(\backslash mathbf\{x\})$ is a Gaussian $\backslash mathcal\{N\}(\backslash mathbf\{x\}|\backslash mu,\; \backslash Sigma)$, then $\backslash mathrm\{KL\}(p||q)$ is minimized with $\backslash mu$ and $\backslash Sigma$ being equal to the mean of $p(\backslash mathbf\{x\})$ and the covariance of $p(\backslash mathbf\{x\})$, respectively; this is called moment matching.