Evidence lower bound

Let $X$ and $Z$ be random variables, jointly distributed with distribution $p\_\backslash theta$. For example, $p\_\backslash theta(\; X)$ is the marginal distribution of $X$, and $p\_\backslash theta(\; Z\; \backslash mid\; X)$ is the conditional distribution of $Z$ given $X$. Then, for a sample $x\backslash sim\; p\_\backslash text\{data\}$, and any distribution $$

q_\phi

, the ELBO is defined as$$L(\backslash phi,\; \backslash theta;\; x)\; :=\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot\; |\; x)\}\; \backslash left[\; \backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z|x)\}\; \backslash right]\; .$$

The ELBO can equivalently be written as{{Cite book |last1=Goodfellow |first1=Ian |title=Deep learning |last2=Bengio |first2=Yoshua |last3=Courville |first3=Aaron |date=2016 |publisher=The MIT press |isbn=978-0-262-03561-3 |series=Adaptive computation and machine learning |location=Cambridge, Mass |chapter=Chapter 19}}

$$\backslash begin\{align\}$$

L(\phi, \theta; x) = & \mathbb E_{z\sim q_\phi(\cdot | x)}\left[ \ln{} p_\theta(x, z) \right] + H[ q_\phi(z|x) ] \\

= & \mathbb \ln{} \,p_\theta(x) - D_{KL}( q_\phi(z|x) || p_\theta(z|x) ) . \\

\end{align}

In the first line, $H[\; q\_\backslash phi(z|x)\; ]$ is the entropy of $q\_\backslash phi$, which relates the ELBO to the Helmholtz free energy.{{Cite journal |last1=Hinton |first1=Geoffrey E |last2=Zemel |first2=Richard |date=1993 |title=Autoencoders, Minimum Description Length and Helmholtz Free Energy |url=https://proceedings.neurips.cc/paper/1993/hash/9e3cfc48eccf81a0d57663e129aef3cb-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Morgan-Kaufmann |volume=6}} In the second line, $\backslash ln\; p\_\backslash theta(x)$ is called the evidence for $x$, and $D\_\{KL\}(\; q\_\backslash phi(z|x)\; ||\; p\_\backslash theta(z|x)\; )$ is the Kullback-Leibler divergence between $q\_\backslash phi$ and $p\_\backslash theta$. Since the Kullback-Leibler divergence is non-negative, $L(\backslash phi,\; \backslash theta;\; x)$ forms a lower bound on the evidence (ELBO inequality)$$\backslash ln\; p\_\backslash theta(x)\; \backslash ge\; \backslash mathbb\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\; \backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z\backslash vert\; x)\}\; \backslash right].$$

{{further|Variational Bayesian methods}}

Suppose we have an observable random variable $X$, and we want to find its true distribution $p^*$. This would allow us to generate data by sampling, and estimate probabilities of future events. In general, it is impossible to find $p^*$ exactly, forcing us to search for a good approximation.

That is, we define a sufficiently large parametric family $\backslash \{p\_\backslash theta\backslash \}\_\{\backslash theta\backslash in\backslash Theta\}$ of distributions, then solve for $\backslash min\_\backslash theta\; L(p\_\backslash theta,\; p^*)$ for some loss function $L$. One possible way to solve this is by considering small variation from $p\_\backslash theta$ to $p\_\{\backslash theta\; +\; \backslash delta\; \backslash theta\}$, and solve for $L(p\_\backslash theta,\; p^*)\; -\; L(p\_\{\backslash theta+\backslash delta\; \backslash theta\},\; p^*)\; =0$. This is a problem in the calculus of variations, thus it is called the variational method.

Since there are not many explicitly parametrized distribution families (all the classical distribution families, such as the normal distribution, the Gumbel distribution, etc, are far too simplistic to model the true distribution), we consider implicitly parametrized probability distributions:

• First, define a simple distribution $p(z)$ over a latent random variable $Z$. Usually a normal distribution or a uniform distribution suffices.
• Next, define a family of complicated functions $f\_\backslash theta$ (such as a deep neural network) parametrized by $\backslash theta$.
• Finally, define a way to convert any $f\_\backslash theta(z)$ into a distribution (in general simple too, but unrelated to $p(z)$) over the observable random variable $X$. For example, let $f\_\backslash theta(z)\; =\; (f\_1(z),\; f\_2(z))$ have two outputs, then we can define the corresponding distribution over $X$ to be the normal distribution $\backslash mathcal\; N(f\_1(z),\; e^\{f\_2(z)\})$.

This defines a family of joint distributions $p\_\backslash theta$ over $(X,\; Z)$. It is very easy to sample $(x,\; z)\; \backslash sim\; p\_\backslash theta$: simply sample $z\backslash sim\; p$, then compute $f\_\backslash theta(z)$, and finally sample $x\; \backslash sim\; p\_\backslash theta(\backslash cdot\; |\; z)$ using $f\_\backslash theta(z)$.

In other words, we have a generative model for both the observable and the latent.

Now, we consider a distribution $p\_\backslash theta$ good, if it is a close approximation of $p^*$:$$p\_\backslash theta(X)\; \backslash approx\; p^*(X)$$since the distribution on the right side is over $X$ only, the distribution on the left side must marginalize the latent variable $Z$ away.

In general, it's impossible to perform the integral $p\_\backslash theta(x)\; =\; \backslash int\; p\_\backslash theta(x|z)p(z)dz$, forcing us to perform another approximation.

Since $p\_\backslash theta(x)\; =\; \backslash frac\{p\_\backslash theta(x|z)p(z)\}\{p\_\backslash theta(z|x)\}$ (Bayes' Rule), it suffices to find a good approximation of $p\_\backslash theta(z|x)$. So define another distribution family $q\_\backslash phi(z|x)$ and use it to approximate $p\_\backslash theta(z|x)$. This is a discriminative model for the latent.

The entire situation is summarized in the following table:

class="wikitable" |+ !$X$: observable !$X,\; Z$ !$Z$: latent

$p^*(x)\backslash approx\; p\_\backslash theta(x)\; \backslash approx\; \backslash frac\{p\_\backslash theta(x|z)p(z)\}\{q\_\backslash phi(z|x)\}$ approximable | |$p(z)$, easy

|$p\_\backslash theta(x|z)p(z)$, easy |

$p\_\backslash theta(z|x)\; \backslash approx\; q\_\backslash phi(z|x)$ approximable | |$p\_\backslash theta(x|z)$, easy

In Bayesian language, $X$ is the observed evidence, and $Z$ is the latent/unobserved. The distribution $p$ over $Z$ is the prior distribution over $Z$, $p\_\backslash theta(x|z)$ is the likelihood function, and $p\_\backslash theta(z|x)$ is the posterior distribution over $Z$.

Given an observation $x$, we can infer what $z$ likely gave rise to $x$ by computing $p\_\backslash theta(z|x)$. The usual Bayesian method is to estimate the integral $p\_\backslash theta(x)\; =\; \backslash int\; p\_\backslash theta(x|z)p(z)dz$, then compute by Bayes' rule $p\_\backslash theta(z|x)\; =\; \backslash frac\{p\_\backslash theta(x|z)p(z)\}\{p\_\backslash theta(x)\}$. This is expensive to perform in general, but if we can simply find a good approximation $q\_\backslash phi(z|x)\; \backslash approx\; p\_\backslash theta(z|x)$ for most $x,\; z$, then we can infer $z$ from $x$ cheaply. Thus, the search for a good $q\_\backslash phi$ is also called amortized inference.

All in all, we have found a problem of variational Bayesian inference.

A basic result in variational inference is that minimizing the Kullback–Leibler divergence (KL-divergence) is equivalent to maximizing the log-likelihood:$$\backslash mathbb\{E\}\_\{x\backslash sim\; p^*(x)\}[\backslash ln\; p\_\backslash theta\; (x)]\; =\; -H(p^*)\; -\; D\_\{\backslash mathit\{KL\}\}(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$where $H(p^*)\; =\; -\backslash mathbb\; \backslash mathbb\; E\_\{x\backslash sim\; p^*\}[\backslash ln\; p^*(x)]$ is the entropy of the true distribution. So if we can maximize $\backslash mathbb\{E\}\_\{x\backslash sim\; p^*(x)\}[\backslash ln\; p\_\backslash theta\; (x)]$, we can minimize $D\_\{\backslash mathit\{KL\}\}(p^*(x)\; \backslash |\; p\_\backslash theta(x))$, and consequently find an accurate approximation $p\_\backslash theta\; \backslash approx\; p^*$.

To maximize $\backslash mathbb\{E\}\_\{x\backslash sim\; p^*(x)\}[\backslash ln\; p\_\backslash theta\; (x)]$, we simply sample many $x\_i\backslash sim\; p^*(x)$, i.e. use importance sampling$$N\backslash max\_\backslash theta\; \backslash mathbb\{E\}\_\{x\backslash sim\; p^*(x)\}[\backslash ln\; p\_\backslash theta\; (x)]\backslash approx\; \backslash max\_\backslash theta\; \backslash sum\_i\; \backslash ln\; p\_\backslash theta\; (x\_i)$$where $N$ is the number of samples drawn from the true distribution. This approximation can be seen as overfitting.{{NoteTag|note=In fact, by Jensen's inequality,

$\backslash mathbb\; E\_\{x\backslash sim\; p^\{*\}(x)\}\backslash left[\backslash max\; \_\{\backslash theta\; \}\backslash sum\; \_\{i\}\backslash ln\; p\_\{\backslash theta\; \}(x\_\{i\})\backslash right]\backslash geq\; \backslash max\; \_\{\backslash theta\; \}\backslash mathbb\; E\_\{x\backslash sim\; p^\{*\}(x)\}\backslash left[\backslash sum\; \_\{i\}\backslash ln\; p\_\{\backslash theta\; \}(x\_\{i\})\backslash right]=N\backslash max\; \_\{\backslash theta\; \}\backslash mathbb\; \{E\}\; \_\{x\backslash sim\; p^\{*\}(x)\}[\backslash ln\; p\_\{\backslash theta\; \}(x)]$

The estimator is biased upwards. This can be seen as overfitting: for some finite set of sampled data

$x\_\{i\}$

, there is usually some

$\backslash theta$

that fits them better than the entire

$p^\{*\}$

distribution.|name=in fact}}

In order to maximize $\backslash sum\_i\; \backslash ln\; p\_\backslash theta\; (x\_i)$, it's necessary to find $\backslash ln\; p\_\backslash theta(x)$:$$\backslash ln\; p\_\backslash theta(x)\; =\; \backslash ln\; \backslash int\; p\_\backslash theta(x|z)\; p(z)dz$$This usually has no closed form and must be estimated. The usual way to estimate integrals is Monte Carlo integration with importance sampling:$$\backslash int\; p\_\backslash theta(x|z)\; p(z)dz\; =\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]$$where $q\_\backslash phi(z|x)$ is a sampling distribution over $z$ that we use to perform the Monte Carlo integration.

So we see that if we sample $z\backslash sim\; q\_\backslash phi(\backslash cdot|x)$, then $\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}$ is an unbiased estimator of $p\_\backslash theta(x)$. Unfortunately, this does not give us an unbiased estimator of $\backslash ln\; p\_\backslash theta(x)$, because $\backslash ln$ is nonlinear. Indeed, we have by Jensen's inequality, $$\backslash ln\; p\_\backslash theta(x)=\; \backslash ln\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; \backslash geq\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\backslash ln\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]$$In fact, all the obvious estimators of $\backslash ln\; p\_\backslash theta(x)$ are biased downwards, because no matter how many samples of $z\_i\backslash sim\; q\_\backslash phi(\backslash cdot\; |\; x)$ we take, we have by Jensen's inequality:$$\backslash mathbb\; E\_\{z\_i\; \backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[$$

\ln \left(\frac 1N \sum_i \frac{p_\theta (x, z_i)}{q_\phi(z_i|x)}\right)

\right] \leq \ln \mathbb E_{z_i \sim q_\phi(\cdot|x)}\left[

\frac 1N \sum_i \frac{p_\theta (x, z_i)}{q_\phi(z_i|x)}

\right] = \ln p_\theta(x) Subtracting the right side, we see that the problem comes down to a biased estimator of zero:$$\backslash mathbb\; E\_\{z\_i\; \backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[$$

\ln \left(\frac 1N \sum_i \frac{p_\theta (z_i|x)}{q_\phi(z_i|x)}\right)

\right] \leq 0At this point, we could branch off towards the development of an importance-weighted autoencoder{{NoteTag|note=By the delta method, we have$$\backslash mathbb\; E\_\{z\_i\; \backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[$$

\ln \left(\frac 1N \sum_i \frac{p_\theta (z_i|x)}{q_\phi(z_i|x)}\right)

\right] \approx -\frac{1}{2N} \mathbb V_{z \sim q_\phi(\cdot|x)}\left[\frac{p_\theta (z|x)}{q_\phi(z|x)}\right] = O(N^{-1})If we continue with this, we would obtain the importance-weighted autoencoder.{{Cite arXiv |last1=Burda |first1=Yuri |last2=Grosse |first2=Roger |last3=Salakhutdinov |first3=Ruslan |date=2015-09-01 |title=Importance Weighted Autoencoders | class=stat.ML | eprint=1509.00519}}|name=importance-weighted}}, but we will instead continue with the simplest case with $N=1$:$$\backslash ln\; p\_\backslash theta(x)=\; \backslash ln\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; \backslash geq\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\backslash ln\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]$$The tightness of the inequality has a closed form:$$\backslash ln\; p\_\backslash theta(x)-\; \backslash mathbb\; E\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot|x)\}\backslash left[\backslash ln\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; =\; D\_\{\backslash mathit\{KL\}\}(q\_\backslash phi(\backslash cdot\; |\; x)\backslash |\; p\_\backslash theta(\backslash cdot\; |\; x))\backslash geq\; 0$$We have thus obtained the ELBO function:$$L(\backslash phi,\; \backslash theta;\; x)\; :=\; \backslash ln\; p\_\backslash theta(x)\; -\; D\_\{\backslash mathit\{KL\}\}(q\_\backslash phi(\backslash cdot\; |\; x)\backslash |\; p\_\backslash theta(\backslash cdot\; |\; x))$$

For fixed $x$, the optimization $\backslash max\_\{\backslash theta,\; \backslash phi\}\; L(\backslash phi,\; \backslash theta;\; x)$ simultaneously attempts to maximize $\backslash ln\; p\_\backslash theta(x)$ and minimize $D\_\{\backslash mathit\{KL\}\}(q\_\backslash phi(\backslash cdot\; |\; x)\backslash |\; p\_\backslash theta(\backslash cdot\; |\; x))$. If the parametrization for $p\_\backslash theta$ and $q\_\backslash phi$ are flexible enough, we would obtain some $\backslash hat\backslash phi,\; \backslash hat\; \backslash theta$, such that we have simultaneously

$$\backslash ln\; p\_\{\backslash hat\; \backslash theta\}(x)\; \backslash approx\; \backslash max\_\backslash theta\; \backslash ln\; p\_\backslash theta(x);\; \backslash quad\; q\_\{\backslash hat\backslash phi\}(\backslash cdot\; |\; x)\backslash approx\; p\_\{\backslash hat\backslash theta\}(\backslash cdot\; |\; x)$$Since$$\backslash mathbb\{E\}\_\{x\backslash sim\; p^*(x)\}[\backslash ln\; p\_\backslash theta\; (x)]\; =\; -H(p^*)\; -\; D\_\{\backslash mathit\{KL\}\}(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$we have$$\backslash ln\; p\_\{\backslash hat\; \backslash theta\}(x)\; \backslash approx\; \backslash max\_\backslash theta\; -H(p^*)\; -\; D\_\{\backslash mathit\{KL\}\}(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$and so$$\backslash hat\backslash theta\; \backslash approx\; \backslash arg\backslash min\; D\_\{\backslash mathit\{KL\}\}(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$In other words, maximizing the ELBO would simultaneously allow us to obtain an accurate generative model $p\_\{\backslash hat\backslash theta\}\; \backslash approx\; p^*$ and an accurate discriminative model $q\_\{\backslash hat\backslash phi\}(\backslash cdot\; |\; x)\backslash approx\; p\_\{\backslash hat\backslash theta\}(\backslash cdot\; |\; x)$.{{Citation |last1=Neal |first1=Radford M. |title=A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants |date=1998 |url=http://dx.doi.org/10.1007/978-94-011-5014-9_12 |work=Learning in Graphical Models |pages=355–368 |access-date= |place=Dordrecht |publisher=Springer Netherlands |isbn=978-94-010-6104-9 |last2=Hinton |first2=Geoffrey E.|doi=10.1007/978-94-011-5014-9_12 |s2cid=17947141 |url-access=subscription }}

The ELBO has many possible expressions, each with some different emphasis.

:$$\backslash mathbb\{E\}\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot\; |\; x)\}\backslash left[\backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; =\; \backslash int\; q\_\backslash phi(z|x)\backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z|x)\}dz$$

This form shows that if we sample $z\backslash sim\; q\_\backslash phi(\backslash cdot\; |\; x)$, then $\backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z|x)\}$ is an unbiased estimator of the ELBO.

:$$\backslash ln\; p\_\backslash theta(x)\; -\; D\_\{\backslash mathit\{KL\}\}(q\_\backslash phi(\backslash cdot\; |\; x)\; \backslash ;\backslash |\backslash ;\; p\_\backslash theta(\backslash cdot\; |\; x))$$

This form shows that the ELBO is a lower bound on the evidence $\backslash ln\; p\_\backslash theta(x)$, and that maximizing the ELBO with respect to $\backslash phi$ is equivalent to minimizing the KL-divergence from $p\_\backslash theta(\backslash cdot\; |\; x)$ to $q\_\backslash phi(\backslash cdot\; |\; x)$.

:$$\backslash mathbb\{E\}\_\{z\backslash sim\; q\_\backslash phi(\backslash cdot\; |\; x)\}[\backslash ln\; p\_\backslash theta(x|z)]\; -\; D\_\{\backslash mathit\{KL\}\}(q\_\backslash phi(\backslash cdot\; |\; x)\; \backslash ;\backslash |\backslash ;\; p)$$

This form shows that maximizing the ELBO simultaneously attempts to keep $q\_\backslash phi(\backslash cdot\; |\; x)$ close to $p$ and concentrate $q\_\backslash phi(\backslash cdot\; |\; x)$ on those $z$ that maximizes $\backslash ln\; p\_\backslash theta\; (x|z)$. That is, the approximate posterior $q\_\backslash phi(\backslash cdot\; |\; x)$ balances between staying close to the prior $p$ and moving towards the maximum likelihood $\backslash arg\backslash max\_z\; \backslash ln\; p\_\backslash theta\; (x|z)$.

Suppose we take $N$ independent samples from $p^*$, and collect them in the dataset $D\; =\; \backslash \{x\_1,\; ...,\; x\_N\backslash \}$, then we have empirical distribution $q\_D(x)\; =\; \backslash frac\; 1N\; \backslash sum\_i\; \backslash delta\_\{x\_i\}$.

Fitting $p\_\backslash theta(x)$ to $q\_D(x)$ can be done, as usual, by maximizing the loglikelihood $\backslash ln\; p\_\backslash theta(D)$:$$D\_\{\backslash mathit\{KL\}\}(q\_D(x)\; \backslash |\; p\_\backslash theta(x))\; =\; -\backslash frac\; 1N\; \backslash sum\_i\; \backslash ln\; p\_\backslash theta(x\_i)\; -\; H(q\_D)=\; -\backslash frac\; 1N\; \backslash ln\; p\_\backslash theta(D)\; -\; H(q\_D)$$Now, by the ELBO inequality, we can bound $\backslash ln\; p\_\backslash theta(D)$, and thus$$D\_\{\backslash mathit\{KL\}\}(q\_D(x)\; \backslash |\; p\_\backslash theta(x))\; \backslash leq\; -\backslash frac\; 1N\; L(\backslash phi,\; \backslash theta;\; D)\; -\; H(q\_D)$$The right-hand-side simplifies to a KL-divergence, and so we get:$$D\_\{\backslash mathit\{KL\}\}(q\_D(x)\; \backslash |\; p\_\backslash theta(x))\; \backslash leq\; -\backslash frac\; 1N\; \backslash sum\_i\; L(\backslash phi,\; \backslash theta;\; x\_i)\; -\; H(q\_D)=\; D\_\{\backslash mathit\{KL\}\}(q\_\{D,\; \backslash phi\}(x,\; z);\; p\_\backslash theta(x,\; z))$$This result can be interpreted as a special case of the data processing inequality.

In this interpretation, maximizing $L(\backslash phi,\; \backslash theta;\; D)=\; \backslash sum\_i\; L(\backslash phi,\; \backslash theta;\; x\_i)$ is minimizing $D\_\{\backslash mathit\{KL\}\}(q\_\{D,\; \backslash phi\}(x,\; z);\; p\_\backslash theta(x,\; z))$, which upper-bounds the real quantity of interest $D\_\{\backslash mathit\{KL\}\}(q\_\{D\}(x);\; p\_\backslash theta(x))$ via the data-processing inequality. That is, we append a latent space to the observable space, paying the price of a weaker inequality for the sake of more computationally efficient minimization of the KL-divergence.{{Cite journal |last1=Kingma |first1=Diederik P. |last2=Welling |first2=Max |date=2019-11-27 |title=An Introduction to Variational Autoencoders |url=https://www.nowpublishers.com/article/Details/MAL-056 |journal=Foundations and Trends in Machine Learning |language=English |volume=12 |issue=4 |at=Section 2.7 |doi=10.1561/2200000056 |issn=1935-8237|arxiv=1906.02691 |s2cid=174802445 }}

{{reflist|group=note}}

Category:Theory of probability distributions