Kullback–Leibler divergence#Cross entropy

Consider two probability distributions {{mvar|P}} and {{mvar|Q}}. Usually, {{mvar|P}} represents the data, the observations, or a measured probability distribution. Distribution {{mvar|Q}} represents instead a theory, a model, a description or an approximation of {{mvar|P}}. The Kullback–Leibler divergence $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ is then interpreted as the average difference of the number of bits required for encoding samples of {{mvar|P}} using a code optimized for {{mvar|Q}} rather than one optimized for {{mvar|P}}. Note that the roles of {{mvar|P}} and {{mvar|Q}} can be reversed in some situations where that is easier to compute, such as with the expectation–maximization algorithm (EM) and evidence lower bound (ELBO) computations.

The relative entropy was introduced by Solomon Kullback and Richard Leibler in {{harvtxt|Kullback|Leibler|1951}} as "the mean information for discrimination between $H\_1$ and $H\_2$ per observation from $\backslash mu\_1$",{{sfn|Kullback|Leibler|1951|p=80}} where one is comparing two probability measures $\backslash mu\_1,\; \backslash mu\_2$, and $H\_1,\; H\_2$ are the hypotheses that one is selecting from measure $\backslash mu\_1,\; \backslash mu\_2$ (respectively). They denoted this by $I(1:2)$, and defined the "'divergence' between $\backslash mu\_1$ and $\backslash mu\_2$" as the symmetrized quantity $J(1,2)\; =\; I(1:2)\; +\; I(2:1)$, which had already been defined and used by Harold Jeffreys in 1948.{{sfn|Jeffreys|1948|p=158}} In {{harvtxt|Kullback|1959}}, the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions;{{sfn|Kullback|1959|p=7}} Kullback preferred the term discrimination information.{{cite journal

|first=S. |last=Kullback |author-link=Solomon Kullback

|year=1987

|title=Letter to the Editor: The Kullback–Leibler distance

|journal=The American Statistician

|volume=41 |issue=4 |pages=340–341

|jstor=2684769

|doi=10.1080/00031305.1987.10475510

}} The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality.{{sfn|Kullback|1959|p=6}} Numerous references to earlier uses of the symmetrized divergence and to other statistical distances are given in {{harvtxt|Kullback|1959|pp=6–7|loc=§1.3 Divergence}}. The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as the Jeffreys divergence.

For discrete probability distributions {{mvar|P}} and {{mvar|Q}} defined on the same sample space, {{nowrap|$\backslash mathcal\{X\}$,}} the relative entropy from {{mvar|Q}} to {{mvar|P}} is defined{{cite book |last=MacKay |first=David J.C. |year=2003 |title=Information Theory, Inference, and Learning Algorithms |publisher=Cambridge University Press |isbn=9780521642989 |edition=1st |page=34 |url={{google books| plainurl=y |id=AKuMj4PN_EMC}} |via=Google Books }} to be

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash sum\_\{\; x\; \backslash in\; \backslash mathcal\{X\}\; \}\; P(x)\; \backslash ,\; \backslash log\backslash frac\{\; P(x)\; \}\{\; Q(x)\; \}\; \backslash ,\; ,$$

which is equivalent to

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; -\backslash sum\_\{\; x\; \backslash in\; \backslash mathcal\{X\}\; \}\; P(x)\; \backslash ,\; \backslash log\backslash frac\{\; Q(x)\; \}\{P(x)\}\; \backslash ,.$$

In other words, it is the expectation of the logarithmic difference between the probabilities {{mvar|P}} and {{mvar|Q}}, where the expectation is taken using the probabilities {{mvar|P}}.

Relative entropy is only defined in this way if, for all {{mvar|x}}, $Q(x)\; =\; 0$ implies $P(x)\; =\; 0$ (absolute continuity). Otherwise, it is often defined as {{nobr|$+\backslash infty$,}} but the value $\backslash \; +\backslash infty\backslash$ is possible even if $Q(x)\; \backslash ne\; 0$ everywhere,{{cite web |title=What's the maximum value of Kullback-Leibler (KL) divergence? |department=Machine learning |series=Cross validated |website=Statistics Stack Exchange (stats.stackexchange.com) |url=https://stats.stackexchange.com/q/351947 }}{{cite web |title=In what situations is the integral equal to infinity? |department=Integration |website=Mathematics Stack Exchange (math.stackexchange.com) |url=https://math.stackexchange.com/q/20961}} provided that $\backslash mathcal\{X\}$ is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.

Whenever $P(x)$ is zero the contribution of the corresponding term is interpreted as zero because

$$\backslash lim\_\{x\; \backslash to\; 0^\{+\}\}\; x\; \backslash ,\; \backslash log(x)\; =\; 0\; \backslash ,.$$

For distributions {{mvar|P}} and {{mvar|Q}} of a continuous random variable, relative entropy is defined to be the integral{{cite book |last=Bishop |first= Christopher M. |title=Pattern recognition and machine learning |oclc=1334664824 |page= 55 |url=http://worldcat.org/oclc/1334664824}}

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; p(x)\; \backslash ,\; \backslash log\backslash frac\{p(x)\}\{q(x)\}\; \backslash ,\; dx\; \backslash ,\; ,$$

where {{mvar|p}} and {{mvar|q}} denote the probability densities of {{mvar|P}} and {{mvar|Q}}.

More generally, if {{mvar|P}} and {{mvar|Q}} are probability measures on a measurable space $\backslash mathcal\{X\}\backslash ,\; ,$ and {{mvar|P}} is absolutely continuous with respect to {{mvar|Q}}, then the relative entropy from {{mvar|Q}} to {{mvar|P}} is defined as

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash int\_\{\; x\; \backslash in\; \backslash mathcal\{X\}\; \}\; \backslash log\; \backslash frac\{P(dx)\}\{Q(dx)\}\; \backslash ,\; P(dx)\backslash ,\; ,$$

where $\backslash frac\{\; P(dx)\; \}\{Q(dx)\; \}$ is the Radon–Nikodym derivative of {{mvar|P}} with respect to {{mvar|Q}}, i.e. the unique {{mvar|Q}} almost everywhere defined function {{mvar|r}} on $\backslash mathcal\{X\}$ such that $P(dx)\; =\; r(x)\; Q(dx)$ which exists because {{mvar|P}} is absolutely continuous with respect to {{mvar|Q}}. Also we assume the expression on the right-hand side exists. Equivalently (by the chain rule), this can be written as

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash int\_\{\; x\; \backslash in\; \backslash mathcal\{X\}\; \}\; \backslash frac\{P(dx)\}\{Q(dx)\}\backslash \; \backslash log\backslash frac\{P(dx)\}\{Q(dx)\}\; \backslash \; Q(dx)\; \backslash ,\; ,$$

which is the entropy of {{mvar|P}} relative to {{mvar|Q}}. Continuing in this case, if $\backslash mu$ is any measure on $\backslash mathcal\{X\}$ for which densities {{mvar|p}} and {{mvar|q}} with $P(dx)\; =\; p(x)\; \backslash mu(dx)$ and $Q(dx)\; =\; q(x)\; \backslash mu(dx)$ exist (meaning that {{mvar|P}} and {{mvar|Q}} are both absolutely continuous with respect to {{nowrap|$\backslash mu$),}} then the relative entropy from {{mvar|Q}} to {{mvar|P}} is given as

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash int\_\{x\; \backslash in\; \backslash mathcal\{X\}\}\; p(x)\backslash ,\; \backslash log\backslash frac\{p(x)\}\{q(x)\}\; \backslash \; \backslash mu(dx)\; \backslash ,.$$

Note that such a measure $\backslash mu$ for which densities can be defined always exists, since one can take $\backslash mu\; =\; \backslash frac\{1\}\{2\}\; \backslash left(\; P\; +\; Q\; \backslash right)$ although in practice it will usually be one that applies in the context like counting measure for discrete distributions, or Lebesgue measure or a convenient variant thereof like Gaussian measure or the uniform measure on the sphere, Haar measure on a Lie group etc. for continuous distributions.

The logarithms in these formulae are usually taken to base 2 if information is measured in units of bits, or to base {{mvar|e}} if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm.

Various conventions exist for referring to $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ in words. Often it is referred to as the divergence between {{mvar|P}} and {{mvar|Q}}, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of {{mvar|P}} from {{mvar|Q}} or as the divergence from {{mvar|Q}} to {{mvar|P}}. This reflects the asymmetry in Bayesian inference, which starts from a prior {{mvar|Q}} and updates to the posterior {{mvar|P}}. Another common way to refer to $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ is as the relative entropy of {{mvar|P}} with respect to {{mvar|Q}} or the information gain from {{mvar|P}} over {{mvar|Q}}.

Kullback{{sfn|Kullback|1959}} gives the following example (Table 2.1, Example 2.1). Let {{mvar|P}} and {{mvar|Q}} be the distributions shown in the table and figure. {{mvar|P}} is the distribution on the left side of the figure, a binomial distribution with $N\; =\; 2$ and $p\; =\; 0.4$. {{mvar|Q}} is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes {{math|{{var|x}} {{=}} {{val|0}}, {{val|1}}, {{val|2}}}} (i.e. $\backslash mathcal\{X\}=\backslash \{0,1,2\backslash \}$), each with probability $p\; =\; 1/3$.

File:Kullback–Leibler distributions example 1.svg

Relative entropies $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ and $D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)$ are calculated as follows. This example uses the natural log with base E (mathematical constant), designated {{math|ln}} to get results in nats (see units of information):

$$\backslash begin\{align\}$$

D_\text{KL}(P \parallel Q)

&= \sum_{x\in\mathcal{X}} P(x) \, \ln\frac{P(x)}{Q(x)} \\

&= \frac{9}{25} \ln\frac{9/25}{1/3}

+ \frac{12}{25} \ln\frac{12/25}{1/3}

+ \frac{4}{25} \ln\frac{4/25}{1/3} \\

& = \frac{1}{25} \left(32 \ln 2 + 55 \ln 3 - 50 \ln 5 \right) \\

&\approx 0.0852996,

\end{align}

$$\backslash begin\{align\}$$

D_\text{KL}(Q \parallel P) &= \sum_{x\in\mathcal{X}} Q(x) \, \ln\frac{Q(x)}{P(x)} \\

&= \frac{1}{3} \, \ln\frac{1/3}{9/25}

+ \frac{1}{3} \, \ln\frac{1/3}{12/25}

+ \frac{1}{3} \, \ln\frac{1/3}{4/25} \\

&= \frac{1}{3} \left(-4 \ln 2 - 6 \ln 3 + 6 \ln 5 \right) \\

&\approx 0.097455.

\end{align}

In the field of statistics, the Neyman–Pearson lemma states that the most powerful way to distinguish between the two distributions {{mvar|P}} and {{mvar|Q}} based on an observation {{mvar|Y}} (drawn from one of them) is through the log of the ratio of their likelihoods: $\backslash log\; P(Y)\; -\; \backslash log\; Q(Y)$. The KL divergence is the expected value of this statistic if {{mvar|Y}} is actually drawn from {{mvar|P}}. Kullback motivated the statistic as an expected log likelihood ratio.{{sfn|Kullback|1959|p=5}}

In the context of coding theory, $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ can be constructed by measuring the expected number of extra bits required to code samples from {{mvar|P}} using a code optimized for {{mvar|Q}} rather than the code optimized for {{mvar|P}}.

In the context of machine learning, $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ is often called the information gain achieved if {{mvar|P}} would be used instead of {{mvar|Q}} which is currently used. By analogy with information theory, it is called the relative entropy of {{mvar|P}} with respect to {{mvar|Q}}.

Expressed in the language of Bayesian inference, $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ is a measure of the information gained by revising one's beliefs from the prior probability distribution {{mvar|Q}} to the posterior probability distribution {{mvar|P}}. In other words, it is the amount of information lost when {{mvar|Q}} is used to approximate {{mvar|P}}.{{cite book |last1=Burnham |first1=K. P. |last2=Anderson |first2=D. R. |year=2002 |title=Model Selection and Multi-Model Inference |publisher=Springer |edition=2nd |page=[https://archive.org/details/modelselectionmu0000burn/page/51 51] |url=https://archive.org/details/modelselectionmu0000burn |url-access=registration |isbn=9780387953649 }}

In applications, {{mvar|P}} typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while {{mvar|Q}} typically represents a theory, model, description, or approximation of {{mvar|P}}. In order to find a distribution {{mvar|Q}} that is closest to {{mvar|P}}, we can minimize the KL divergence and compute an information projection.

While it is a statistical distance, it is not a metric, the most familiar type of distance, but instead it is a divergence.{{sfn|Amari|2016|p=11}} While metrics are symmetric and generalize linear distance, satisfying the triangle inequality, divergences are asymmetric and generalize squared distance, in some cases satisfying a generalized Pythagorean theorem. In general $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ does not equal $D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)$, and the asymmetry is an important part of the geometry.{{sfn|Amari|2016|p=11}} The infinitesimal form of relative entropy, specifically its Hessian, gives a metric tensor that equals the Fisher information metric; see {{slink||Fisher information metric}}. Fisher information metric on the certain probability distribution let determine the natural gradient for information-geometric optimization algorithms.{{Cite journal |last1=Abdulkadirov |first1=Ruslan |last2=Lyakhov |first2=Pavel |last3=Nagornov |first3=Nikolay |date=January 2023 |title=Survey of Optimization Algorithms in Modern Neural Networks |journal=Mathematics |language=en |volume=11 |issue=11 |pages=2466 |doi=10.3390/math11112466 |doi-access=free |issn=2227-7390}} Its quantum version is Fubini-study metric.{{Cite journal |last=Matassa |first=Marco |date=December 2021 |title=Fubini-Study metrics and Levi-Civita connections on quantum projective spaces |url=https://linkinghub.elsevier.com/retrieve/pii/S0001870821005405 |journal=Advances in Mathematics |volume=393 |pages=108101 |doi=10.1016/j.aim.2021.108101 |issn=0001-8708|arxiv=2010.03291 }} Relative entropy satisfies a generalized Pythagorean theorem for exponential families (geometrically interpreted as dually flat manifolds), and this allows one to minimize relative entropy by geometric means, for example by information projection and in maximum likelihood estimation.{{sfn|Amari|2016|p=28}}

The relative entropy is the Bregman divergence generated by the negative entropy, but it is also of the form of an f-divergence. For probabilities over a finite alphabet, it is unique in being a member of both of these classes of statistical divergences. The application of Bregman divergence can be found in mirror descent.{{Cite journal |last=Lan |first=Guanghui |date=March 2023 |title=Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes |url=https://link.springer.com/article/10.1007/s10107-022-01816-5 |journal=Mathematical Programming |language=en |volume=198 |issue=1 |pages=1059–1106 |doi=10.1007/s10107-022-01816-5 |issn=1436-4646|arxiv=2102.00135 }}

Consider a growth-optimizing investor in a fair game with mutually exclusive outcomes

(e.g. a “horse race” in which the official odds add up to one).

The rate of return expected by such an investor is equal to the relative entropy

between the investor's believed probabilities and the official odds.{{cite journal |last=Kelly |first=J. L. Jr. |year=1956 |title=A New Interpretation of Information Rate |journal=Bell Syst. Tech. J. |volume=2 |issue=4 |pages=917–926 |doi=10.1002/j.1538-7305.1956.tb03809.x }}

This is a special case of a much more general connection between financial returns and divergence measures.{{cite journal |last=Soklakov |first=A. N. |year=2020 |title=Economics of Disagreement—Financial Intuition for the Rényi Divergence |journal=Entropy |volume=22 |issue=8 |page=860 |doi=10.3390/e22080860 |pmid=33286632 |pmc=7517462 |arxiv=1811.08308 |bibcode=2020Entrp..22..860S |doi-access=free }}

Financial risks are connected to $D\_\; \backslash text\{KL\}$ via information geometry.{{cite journal |last=Soklakov |first=A. N. |year=2023 |title= Information Geometry of Risks and Returns |journal=Risk |volume=June |ssrn=4134885}} Investors' views, the prevailing market view, and risky scenarios form triangles on the relevant manifold of probability distributions. The shape of the triangles determines key financial risks (both qualitatively and quantitatively). For instance, obtuse triangles in which investors' views and risk scenarios appear on “opposite sides” relative to the market describe negative risks, acute triangles describe positive exposure, and the right-angled situation in the middle corresponds to zero risk. Extending this concept, relative entropy can be hypothetically utilised to identify the behaviour of informed investors, if one takes this to be represented by the magnitude and deviations away from the prior expectations of fund flows, for example.{{cite journal |last1=Henide |first1=Karim |title=Flow Rider: Tradable Ecosystems' Relative Entropy of Flows As a Determinant of Relative Value |journal=The Journal of Investing |date=30 September 2024 |volume=33 |issue=6 |pages=34–58 |doi=10.3905/joi.2024.1.321}}

File:KL-Gauss-Example.pngs. The typical asymmetry is clearly visible.]]

In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value $x\_i$ out of a set of possibilities {{mvar|X}} can be seen as representing an implicit probability distribution $q(x\_i)=2^\{-\backslash ell\_i\}$ over {{mvar|X}}, where $\backslash ell\_i$ is the length of the code for $x\_i$ in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution {{mvar|Q}} is used, compared to using a code based on the true distribution {{mvar|P}}: it is the excess entropy.

$$\backslash begin\{align\}$$

D_\text{KL}(P\parallel Q) &= \sum_{x\in\mathcal{X}} p(x) \log \frac{1}{q(x)} - \sum_{x\in\mathcal{X}} p(x) \log \frac{1}{p(x)} \\[5pt]

&= \Eta(P, Q) - \Eta(P)

\end{align}

where $\backslash Eta(P,Q)$ is the cross entropy of {{mvar|Q}} relative to {{mvar|P}} and $\backslash Eta(P)$ is the entropy of {{mvar|P}} (which is the same as the cross-entropy of P with itself).

The relative entropy $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ can be thought of geometrically as a statistical distance, a measure of how far the distribution {{mvar|Q}} is from the distribution {{mvar|P}}. Geometrically it is a divergence: an asymmetric, generalized form of squared distance. The cross-entropy $H(P,Q)$ is itself such a measurement (formally a loss function), but it cannot be thought of as a distance, since $H(P,P)=:H(P)$ is not zero. This can be fixed by subtracting $H(P)$ to make $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ agree more closely with our notion of distance, as the excess loss. The resulting function is asymmetric, and while this can be symmetrized (see {{slink||Symmetrised divergence}}), the asymmetric form is more useful. See {{slink||Interpretations}} for more on the geometric interpretation.

Relative entropy relates to "rate function" in the theory of large deviations.{{cite journal | last1 = Sanov | first1 = I.N. | year = 1957 | title = On the probability of large deviations of random magnitudes | journal = Mat. Sbornik | volume = 42 | issue = 84| pages = 11–44 }}Novak S.Y. (2011), Extreme Value Methods with Applications to Finance ch. 14.5 (Chapman & Hall). {{isbn|978-1-4398-3574-6}}.

Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used characterization of entropy.{{cite book|last1=Hobson|first1=Arthur|title=Concepts in statistical mechanics.| date=1971| publisher=Gordon and Breach|location=New York|isbn=978-0677032405}} Consequently, mutual information is the only measure of mutual dependence that obeys certain related conditions, since it can be defined in terms of Kullback–Leibler divergence.

• Relative entropy is always non-negative, $$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; \backslash geq\; 0,$$ a result known as Gibbs' inequality, with $D\_\backslash text\{KL\}(P\backslash parallel\; Q)$ equals zero if and only if $P\; =\; Q$ as measures.

In particular, if $P(dx)\; =\; p(x)\; \backslash mu(dx)$ and $Q(dx)\; =\; q(x)\backslash mu(dx)$, then $p(x)\; =\; q(x)$ $\backslash mu$-almost everywhere. The entropy $\backslash Eta(P)$ thus sets a minimum value for the cross-entropy $\backslash Eta(P,\; Q)$, the expected number of bits required when using a code based on {{mvar|Q}} rather than {{mvar|P}}; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value {{mvar|x}} drawn from {{mvar|X}}, if a code is used corresponding to the probability distribution {{mvar|Q}}, rather than the "true" distribution {{mvar|P}}.

• No upper-bound exists for the general case. However, it is shown that if {{mvar|P}} and {{mvar|Q}} are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of $D\_\backslash text\{KL\}(P\backslash parallel\; Q)$ can be calculated.{{cite arXiv

|last1=Bonnici |first1=V. |author-link1=Vincenzo Bonnici

|year = 2020

|title = Kullback-Leibler divergence between quantum distributions, and its upper-bound

|class=cs.LG |eprint = 2008.05932

}}

• Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under parameter transformations. For example, if a transformation is made from variable {{mvar|x}} to variable $y(x)$, then, since $P(dx)\; =\; p(x)\; \backslash ,\; dx\; =\; \backslash tilde\{p\}(y)\; \backslash ,\; dy\; =\; \backslash tilde\{p\}(y(x))\; \backslash left|\backslash tfrac\{dy\}\{dx\}(x)\backslash right|\; \backslash ,dx$ and $Q(dx)=\; q(x)\backslash ,\; dx\; =\; \backslash tilde\{q\}(y)\; \backslash ,\; dy\; =\; \backslash tilde\{q\}(y)\; \backslash left|\backslash tfrac\{dy\}\{dx\}(x)\backslash right|\; dx$ where $\backslash left|\backslash tfrac\{dy\}\{dx\}(x)\backslash right|$ is the absolute value of the derivative or more generally of the Jacobian, the relative entropy may be rewritten: $$\backslash begin\{align\}$$

D_\text{KL}(P \parallel Q)

&= \int_{x_a}^{x_b} p(x) \, \log\frac{p(x)}{q(x)} \, dx \\[6pt]

&= \int_{x_a}^{x_b} \tilde{p}(y(x)) \left|\frac{dy}{dx}\right| \log\frac{\tilde{p}(y(x))\, \left|\frac{dy}{dx}\right|}{\tilde{q}(y(x))\, \left|\frac{dy}{dx}\right|}\, dx \\

&= \int_{y_a}^{y_b} \tilde{p}(y) \, \log\frac{\tilde{p}(y)}{\tilde{q}(y)} \, dy

\end{align} where $y\_a\; =\; y(x\_a)$ and $y\_b\; =\; y(x\_b)$. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a dimensionally consistent quantity, since if {{mvar|x}} is a dimensioned variable, $p(x)$ and $q(x)$ are also dimensioned, since e.g. $P(dx)\; =\; p(x)\; \backslash ,\; dx$ is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theorySee the section "differential entropy – 4" in [http://videolectures.net/nips09_verdu_re/ Relative Entropy] video lecture by Sergio Verdú NIPS 2009 (such as self-information or Shannon entropy), which can become undefined or negative for non-discrete probabilities.

• Relative entropy is additive for independent distributions in much the same way as Shannon entropy. If $P\_1,\; P\_2$ are independent distributions, and $P(dx,\; dy)\; =\; P\_1(dx)P\_2(dy)$, and likewise $Q(dx,\; dy)\; =\; Q\_1(dx)Q\_2(dy)$ for independent distributions $Q\_1,\; Q\_2$ then $$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; D\_\backslash text\{KL\}(P\_1\; \backslash parallel\; Q\_1)\; +\; D\_\backslash text\{KL\}(P\_2\; \backslash parallel\; Q\_2).$$
• Relative entropy $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ is convex in the pair of probability measures $(P,Q)$, i.e. if $(P\_1,Q\_1)$ and $(P\_2,Q\_2)$ are two pairs of probability measures then $$D\_\backslash text\{KL\}(\backslash lambda\; P\_1\; +\; (1\; -\; \backslash lambda)\; P\_2\; \backslash parallel\; \backslash lambda\; Q\_1\; +\; (1\; -\; \backslash lambda)\; Q\_2)\; \backslash le$$

\lambda D_\text{KL}(P_1 \parallel Q_1) + (1 - \lambda)D_\text{KL}(P_2 \parallel Q_2) \text{ for } 0 \le \lambda \le 1.

• $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ may be Taylor expanded about its minimum (i.e. $P=Q$) as $$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash sum\_\{n=2\}^\backslash infty\; \backslash frac\; \{1\}\{n(n-1)\}\; \backslash sum\_\{x\; \backslash in\; \backslash mathcal\{X\}\}\; \backslash frac\{(Q(x)\; -\; P(x))^n\}\{Q(x)^\{n-1\}\}$$ which converges if and only if $P\; \backslash leq\; 2Q$ almost surely w.r.t $Q$.

{{hidden begin|style=width:100%|ta1=right|border=1px #aaa solid|title=[Proof]|ta2=left}}

Denote $f(\backslash alpha)\; :=\; D\_\backslash text\{KL\}((1-\backslash alpha)\; Q\; +\; \backslash alpha\; P\; \backslash parallel\; Q)$ and note that $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; f(1)$. The first derivative of $f$ may be derived and evaluated as follows

\begin{align}

f'(\alpha) &= \sum_{x \in \mathcal{X}} (P(x) - Q(x)) \left(\log\left(\frac{(1-\alpha) Q(x) + \alpha P(x)}{Q(x)}\right) + 1\right)

\\ &= \sum_{x \in \mathcal{X}} (P(x) - Q(x)) \log\left(\frac{(1-\alpha) Q(x) + \alpha P(x)}{Q(x)}\right)

\\ f'(0) &= 0

\end{align}

Further derivatives may be derived and evaluated as follows

\begin{align}

f''(\alpha) &= \sum_{x \in \mathcal{X}} \frac{(P(x) - Q(x))^2}{(1-\alpha) Q(x) + \alpha P(x)}

\\ f''(0) &= \sum_{x \in \mathcal{X}} \frac{(P(x) - Q(x))^2}{Q(x)}

\\ f^{(n)}(\alpha) &= (-1)^n (n-2)!\sum_{x \in \mathcal{X}} \frac{(P(x) - Q(x))^n}{\left((1-\alpha) Q(x) + \alpha P(x)\right)^{n-1}}

\\ f^{(n)}(0) &= (-1)^n (n-2)!\sum_{x \in \mathcal{X}} \frac{(P(x) - Q(x))^n}{Q(x)^{n-1}}

\end{align}

Hence solving for $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ via the Taylor expansion of $f$ about $0$ evaluated at $\backslash alpha=1$ yields

\begin{align}

D_\text{KL}(P \parallel Q) &= \sum_{n=0}^\infty \frac{f^{(n)}(0)}{n!}

\\ &= \sum_{n=2}^\infty \frac{1}{n(n-1)} \sum_{x \in \mathcal{X}} \frac{(Q(x) - P(x))^n}{Q(x)^{n-1}}

\end{align}

$P\; \backslash leq\; 2Q$ a.s. is a sufficient condition for convergence of the series by the following absolute convergence argument

\begin{align}

\sum_{n=2}^\infty \left\vert\frac{1}{n(n-1)} \sum_{x \in \mathcal{X}} \frac{(Q(x) - P(x))^n}{Q(x)^{n-1}}\right\vert &= \sum_{n=2}^\infty \frac{1}{n(n-1)} \sum_{x \in \mathcal{X}} \left\vert Q(x) - P(x) \right\vert \left\vert 1 - \frac{P(x)}{Q(x)} \right\vert^{n-1}

\\ &\leq \sum_{n=2}^\infty \frac{1}{n(n-1)} \sum_{x \in \mathcal{X}} \left\vert Q(x) - P(x) \right\vert

\\ &\leq \sum_{n=2}^\infty \frac{1}{n(n-1)}

\\ &= 1

\end{align}

$P\; \backslash leq\; 2Q$ a.s. is also a necessary condition for convergence of the series by the following proof by contradiction. Assume that $P\; >\; 2Q$ with measure strictly greater than $0$. It then follows that there must exist some values $\backslash varepsilon\; >\; 0$, $\backslash rho\; >\; 0$, and such that $P\; \backslash geq\; 2Q\; +\; \backslash varepsilon$ and $Q\; \backslash leq\; U$ with measure $\backslash rho$. The previous proof of sufficiency demonstrated that the measure $1-\backslash rho$ component of the series where $P\; \backslash leq\; 2Q$ is bounded, so we need only concern ourselves with the behavior of the measure $\backslash rho$ component of the series where $P\; \backslash geq\; 2Q\; +\; \backslash varepsilon$. The absolute value of the $n$th term of this component of the series is then lower bounded by $\backslash frac1\{n(n-1)\}\; \backslash rho\; \backslash left(1\; +\; \backslash frac\{\backslash varepsilon\}\{U\}\backslash right)^n$, which is unbounded as $n\; \backslash to\; \backslash infty$, so the series diverges.

{{hidden end}}

The following result, due to Donsker and Varadhan,{{Cite journal | last1 = Donsker | first1 = Monroe D. | last2 = Varadhan | first2 = SR Srinivasa | title = Asymptotic evaluation of certain Markov process expectations for large time. IV.|journal=Communications on Pure and Applied Mathematics|year=1983| volume = 36 | issue = 2 |pages=183–212| doi = 10.1002/cpa.3160360204 }} is known as Donsker and Varadhan's variational formula.

{{Math theorem

| name = {{EquationRef|3|Theorem [Duality Formula for Variational Inference]}}

| note =

| math_statement =

Let $\backslash Theta$ be a set endowed with an appropriate $\backslash sigma$-field $\backslash mathcal\{F\}$, and two probability measures {{mvar|P}} and {{mvar|Q}}, which formulate two probability spaces $(\backslash Theta,\backslash mathcal\{F\},P)$ and $(\backslash Theta,\backslash mathcal\{F\},Q)$, with $Q\; \backslash ll\; P$. ($Q\; \backslash ll\; P$ indicates that {{mvar|Q}} is absolutely continuous with respect to {{mvar|P}}.) Let {{mvar|h}} be a real-valued integrable random variable on $(\backslash Theta,\backslash mathcal\{F\},P)$. Then the following equality holds

$$\backslash log\; E\_P[\backslash exp\; h]\; =\; \backslash operatorname\{sup\}\_\{Q\; \backslash ll\; P\}\; \backslash \{\; E\_Q[h]\; -\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)\backslash \}.$$

Further, the supremum on the right-hand side is attained if and only if it holds

$$\backslash frac\{Q(d\backslash theta)\}\{P(d\backslash theta)\}\; =\; \backslash frac\{\backslash exp\; h(\backslash theta)\}\{E\_P[\backslash exp\; h]\},$$

almost surely with respect to probability measure {{mvar|P}}, where $\backslash frac\{Q(d\backslash theta)\}\{P(d\backslash theta)\}$ denotes the Radon-Nikodym derivative of {{mvar|Q}} with respect to {{mvar|P}} .

}}

{{Math proof|title=Proof|drop=hidden|proof=

For a short proof assuming integrability of $\backslash exp(h)$ with respect to {{mvar|P}}, let $Q^*$ have {{mvar|P}}-density $\backslash frac\{\backslash exp\; h(\backslash theta)\}\{E\_P[\backslash exp\; h]\}$, i.e. $Q^*(d\backslash theta)\; =\; \backslash frac\{\backslash exp\; h(\backslash theta)\}\{E\_P[\backslash exp\; h]\}\; P(d\backslash theta)$ Then

$$D\_\backslash text\{KL\}(Q\; \backslash parallel\; Q^*)\; -\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)\; =\; -E\_Q[h]\; +\; \backslash log\; E\_P[\backslash exp\; h].$$

Therefore,

$$E\_Q[h]\; -\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)\; =\; \backslash log\; E\_P[\backslash exp\; h]\; -\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; Q^*)\; \backslash le\; \backslash log\; E\_P[\backslash exp\; h],$$

where the last inequality follows from $D\_\backslash text\{KL\}(Q\; \backslash parallel\; Q^*)\; \backslash ge\; 0$, for which equality occurs if and only if $Q=Q^*$. The conclusion follows.

}}

Suppose that we have two multivariate normal distributions, with means $\backslash mu\_0,\; \backslash mu\_1$ and with (non-singular) covariance matrices $\backslash Sigma\_0,\; \backslash Sigma\_1.$ If the two distributions have the same dimension, {{mvar|k}}, then the relative entropy between the distributions is as follows:{{cite web|last=Duchi J.|url=https://web.stanford.edu/~jduchi/projects/general_notes.pdf |title=Derivations for Linear Algebra and Optimization|page=13}}

D_\text{KL}\left(\mathcal{N}_0 \parallel \mathcal{N}_1\right) =

\frac{1}{2}\left[ \operatorname{tr}\left(\Sigma_1^{-1}\Sigma_0\right) - k +

\left(\mu_1 - \mu_0\right)^\mathsf{T} \Sigma_1^{-1}\left(\mu_1 - \mu_0\right) +

\ln \frac{\det\Sigma_1}{\det\Sigma_0}

\right].

The logarithm in the last term must be taken to base e (mathematical constant) since all terms apart from the last are base-{{mvar|e}} logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in nats. Dividing the entire expression above by $\backslash ln(2)$ yields the divergence in bits.

In a numerical implementation, it is helpful to express the result in terms of the Cholesky decompositions $L\_0,\; L\_1$ such that $\backslash Sigma\_0\; =\; L\_0\; L\_0^T$ and $\backslash Sigma\_1\; =\; L\_1\; L\_1^T$. Then with {{mvar|M}} and {{mvar|y}} solutions to the triangular linear systems $L\_1\; M\; =\; L\_0$, and $L\_1\; y\; =\; \backslash mu\_1\; -\; \backslash mu\_0$,

D_\text{KL}\left(\mathcal{N}_0 \parallel \mathcal{N}_1\right) =

\frac{1}{2}\left(

\sum_{i,j=1}^k {\left(M_{ij}\right)}^2 - k +

|y|^2 +

2\sum_{i=1}^k \ln \frac{(L_1)_{ii}}{(L_0)_{ii}}

\right).

A special case, and a common quantity in variational inference, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):

D_\text{KL}\left(

\mathcal{N}\left(\left(\mu_1, \ldots, \mu_k\right)^\mathsf{T}, \operatorname{diag} \left(\sigma_1^2, \ldots, \sigma_k^2\right)\right) \parallel

\mathcal{N}\left(\mathbf{0}, \mathbf{I}\right)

\right) =

\frac{1}{2} \sum_{i=1}^k \left[\sigma_i^2 + \mu_i^2 - 1 - \ln\left(\sigma_i^2\right)\right].

For two univariate normal distributions {{mvar|p}} and {{mvar|q}} the above simplifies to{{Cite journal |last1=Belov |first1=Dmitry I. |last2=Armstrong |first2=Ronald D. |date=2011-04-15 |title=Distributions of the Kullback-Leibler divergence with applications |url=http://dx.doi.org/10.1348/000711010x522227 |journal=British Journal of Mathematical and Statistical Psychology |volume=64 |issue=2 |pages=291–309 |doi=10.1348/000711010x522227 |pmid=21492134 |issn=0007-1102}}

D_\text{KL}\left(\mathcal{p} \parallel \mathcal{q}\right) = \log \frac{\sigma_1}{\sigma_0} + \frac{\sigma_0^2 + {\left(\mu_0 - \mu_1\right)}^2}{2\sigma_1^2} - \frac{1}{2}

In the case of co-centered normal distributions with $k=\backslash sigma\_1/\backslash sigma\_0$, this simplifies{{Cite book |last=Buchner|first=Johannes |url=http://worldcat.org/oclc/1363563215 |title=An intuition for physicists: information gain from experiments |date=2022-04-29 |oclc=1363563215}} to:

D_\text{KL}\left(\mathcal{p} \parallel \mathcal{q}\right) = \log_2 k + (k^{-2}-1)/2/\ln(2) \mathrm{bits}

Consider two uniform distributions, with the support of $p=[A,B]$ enclosed within $q=[C,D]$ (

$$D\_\backslash text\{KL\}\backslash left(\backslash mathcal\{p\}\; \backslash parallel\; \backslash mathcal\{q\}\backslash right)\; =\; \backslash log\; \backslash frac\{D-C\}\{B-A\}$$

Intuitively, the information gain to a {{mvar|k}} times narrower uniform distribution contains $\backslash log\_2\; k$ bits. This connects with the use of bits in computing, where $\backslash log\_2k$ bits would be needed to identify one element of a {{mvar|k}} long stream.

The exponential family of distribution is given by

$$p\_X(x|\backslash theta)\; =\; h(x)\; \backslash exp\backslash left(\backslash theta^\backslash mathsf\{T\}\; T(x)\; -\; A(\backslash theta)\backslash right)$$

where $h(x)$ is reference measure, $T(x)$ is sufficient statistics, $\backslash theta$ is canonical natural parameters, and $A(\backslash theta)$ is the log-partition function.

The KL divergence between two distributions $p(x|\backslash theta\_1)$ and $p(x|\backslash theta\_2)$ is given by{{Cite arXiv |last1=Nielsen |first1=Frank |last2=Garcia |first2=Vincent |date=2011 |title=Statistical exponential families: A digest with flash cards |class=cs.LG |eprint=0911.4863 }}

$$D\_\{\backslash text\{KL\}\}(\backslash theta\_1\; \backslash parallel\; \backslash theta\_2)\; =\; \{\backslash left(\backslash theta\_1\; -\; \backslash theta\_2\backslash right)\}^\backslash mathsf\{T\}\; \backslash mu\_1\; -\; A(\backslash theta\_1)\; +\; A(\backslash theta\_2)$$

where $\backslash mu\_1\; =\; E\_\{\backslash theta\_1\}[T(X)]\; =\; \backslash nabla\; A(\backslash theta\_1)$ is the mean parameter of $p(x|\backslash theta\_1)$.

For example, for the Poisson distribution with mean $\backslash lambda$, the sufficient statistics $T(x)\; =\; x$, the natural parameter $\backslash theta\; =\; \backslash log\; \backslash lambda$, and log partition function $A(\backslash theta)\; =\; e^\backslash theta$. As such, the divergence between two Poisson distributions with means $\backslash lambda\_1$ and $\backslash lambda\_2$ is

$$D\_\{\backslash text\{KL\}\}(\backslash lambda\_1\; \backslash parallel\; \backslash lambda\_2)\; =\; \backslash lambda\_1\; \backslash log\; \backslash frac\{\backslash lambda\_1\}\{\backslash lambda\_2\}\; -\; \backslash lambda\_1\; +\; \backslash lambda\_2.$$

As another example, for a normal distribution with unit variance $N(\backslash mu,\; 1)$, the sufficient statistics $T(x)\; =\; x$, the natural parameter $\backslash theta\; =\; \backslash mu$, and log partition function $A(\backslash theta)=\backslash mu^2/2$. Thus, the divergence between two normal distributions $N(\backslash mu\_1,\; 1)$ and $N(\backslash mu\_2,\; 1)$ is

$$D\_\{\backslash text\{KL\}\}(\backslash mu\_1\; \backslash parallel\; \backslash mu\_2)\; =\; \backslash left(\backslash mu\_1\; -\; \backslash mu\_2\backslash right)\; \backslash mu\_1\; -\; \backslash frac\{\backslash mu\_1^2\}\{2\}\; +\; \backslash frac\{\backslash mu\_2^2\}\{2\}\; =\; \backslash frac\{\{\backslash left(\backslash mu\_2\; -\; \backslash mu\_1\backslash right)\}^2\}\{2\}.$$

As final example, the divergence between a normal distribution with unit variance $N(\backslash mu,\; 1)$ and a Poisson distribution with mean $\backslash lambda$ is

$$D\_\{\backslash text\{KL\}\}(\backslash mu\; \backslash parallel\; \backslash lambda)\; =\; (\backslash mu\; -\; \backslash log\; \backslash lambda)\; \backslash mu\; -\; \backslash frac\{\backslash mu^2\}\{2\}\; +\; \backslash lambda.$$

While relative entropy is a statistical distance, it is not a metric on the space of probability distributions, but instead it is a divergence.{{sfn|Amari|2016|p=11}} While metrics are symmetric and generalize linear distance, satisfying the triangle inequality, divergences are asymmetric in general and generalize squared distance, in some cases satisfying a generalized Pythagorean theorem. In general $D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)$ does not equal $D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)$, and while this can be symmetrized (see {{slink||Symmetrised divergence}}), the asymmetry is an important part of the geometry.{{sfn|Amari|2016|p=11}}

It generates a topology on the space of probability distributions. More concretely, if $\backslash \{P\_1,P\_2,\backslash ldots\backslash \}$ is a sequence of distributions such that

$$\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; D\_\backslash text\{KL\}(P\_n\backslash parallel\; Q)\; =\; 0,$$

then it is said that

$$P\_n\; \backslash xrightarrow\{D\}\; \backslash ,\; Q\; .$$

Pinsker's inequality entails that

$$P\_n\; \backslash xrightarrow\{D\}\; P\; \backslash Rightarrow\; P\_n\; \backslash xrightarrow\{TV\}\; P\; ,$$

where the latter stands for the usual convergence in total variation.

Relative entropy is directly related to the Fisher information metric. This can be made explicit as follows. Assume that the probability distributions {{mvar|P}} and {{mvar|Q}} are both parameterized by some (possibly multi-dimensional) parameter $\backslash theta$. Consider then two close by values of $P\; =\; P(\backslash theta)$ and $Q\; =\; P(\backslash theta\_0)$ so that the parameter $\backslash theta$ differs by only a small amount from the parameter value $\backslash theta\_0$. Specifically, up to first order one has (using the Einstein summation convention)

$$P(\backslash theta)\; =\; P(\backslash theta\_0)\; +\; \backslash Delta\backslash theta\_j\; \backslash ,\; P\_j(\backslash theta\_0)\; +\; \backslash cdots$$

with $\backslash Delta\backslash theta\_j\; =\; (\backslash theta\; -\; \backslash theta\_0)\_j$ a small change of $\backslash theta$ in the {{mvar|j}} direction, and $P\_j\backslash left(\backslash theta\_0\backslash right)\; =\; \backslash frac\{\backslash partial\; P\}\{\backslash partial\; \backslash theta\_j\}(\backslash theta\_0)$ the corresponding rate of change in the probability distribution. Since relative entropy has an absolute minimum 0 for $P\; =\; Q$, i.e. $\backslash theta\; =\; \backslash theta\_0$, it changes only to second order in the small parameters $\backslash Delta\backslash theta\_j$. More formally, as for any minimum, the first derivatives of the divergence vanish

$$\backslash left.\backslash frac\{\backslash partial\}\{\backslash partial\backslash theta\_j\}\backslash right|\_\{\backslash theta\; =\; \backslash theta\_0\}\; D\_\backslash text\{KL\}(P(\backslash theta)\; \backslash parallel\; P(\backslash theta\_0))\; =\; 0,$$

and by the Taylor expansion one has up to second order

$$D\_\backslash text\{KL\}(P(\backslash theta)\; \backslash parallel\; P(\backslash theta\_0))\; =\; \backslash frac\{1\}\{2\}\; \backslash ,\; \backslash Delta\backslash theta\_j\; \backslash ,\; \backslash Delta\backslash theta\_k\; \backslash ,\; g\_\{jk\}(\backslash theta\_0)\; +\; \backslash cdots$$

where the Hessian matrix of the divergence

$$g\_\{jk\}(\backslash theta\_0)\; =\; \backslash left.\backslash frac\{\backslash partial^2\}\{\backslash partial\backslash theta\_j\backslash ,\; \backslash partial\backslash theta\_k\}\; \backslash right|\_\{\backslash theta\; =\; \backslash theta\_0\}\; D\_\backslash text\{KL\}(P(\backslash theta)\; \backslash parallel\; P(\backslash theta\_0))$$

must be positive semidefinite. Letting $\backslash theta\_0$ vary (and dropping the subindex 0) the Hessian $g\_\{jk\}(\backslash theta)$ defines a (possibly degenerate) Riemannian metric on the {{mvar|θ}} parameter space, called the Fisher information metric.

When $p\_\{(x,\; \backslash rho)\}$ satisfies the following regularity conditions:

$$\backslash frac\{\backslash partial\; \backslash log(p)\}\{\backslash partial\; \backslash rho\},\; \backslash frac\{\backslash partial^2\; \backslash log(p)\}\{\backslash partial\; \backslash rho^2\},\; \backslash frac\{\backslash partial^3\; \backslash log(p)\}\{\backslash partial\; \backslash rho^3\}$$ exist,

$$\backslash begin\{align\}$$

\left|\frac{\partial p}{\partial \rho}\right| &< F(x): \int_{x=0}^\infty F(x)\,dx < \infty, \\

\left|\frac{\partial^2 p}{\partial \rho^2}\right| &< G(x): \int_{x=0}^\infty G(x)\,dx < \infty \\

\left|\frac{\partial^3 \log(p)}{\partial \rho^3}\right| &< H(x): \int_{x=0}^\infty p(x, 0)H(x)\,dx < \xi < \infty

\end{align}

where {{mvar|ξ}} is independent of {{mvar|ρ}}

\left.\int_{x=0}^\infty \frac{\partial p(x, \rho)}{\partial \rho}\right|_{\rho=0}\, dx =

\left.\int_{x=0}^\infty \frac{\partial^2 p(x, \rho)}{\partial \rho^2}\right|_{\rho=0}\, dx = 0

then:

$$\backslash mathcal\{D\}(p(x,\; 0)\; \backslash parallel\; p(x,\; \backslash rho))\; =\; \backslash frac\{c\backslash rho^2\}\{2\}\; +\; \backslash mathcal\{O\}\backslash left(\backslash rho^3\backslash right)\; \backslash text\{\; as\; \}\; \backslash rho\; \backslash to\; 0.$$

Another information-theoretic metric is variation of information, which is roughly a symmetrization of conditional entropy. It is a metric on the set of partitions of a discrete probability space.

MAUVE is a measure of the statistical gap between two text distributions, such as the difference between text generated by a model and human-written text. This measure is computed using Kullback–Leibler divergences between the two distributions in a quantized embedding space of a foundation model.

Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.

{{Main|Information content}}

The self-information, also known as the information content of a signal, random variable, or event is defined as the negative logarithm of the probability of the given outcome occurring.

When applied to a discrete random variable, the self-information can be represented as{{Citation needed|reason=Needs verification|date=August 2017}}

$$\backslash operatorname\; \backslash operatorname\{I\}(m)\; =\; D\_\backslash text\{KL\}\backslash left(\backslash delta\_\backslash text\{im\}\; \backslash parallel\; \backslash \{p\_i\backslash \}\backslash right),$$

is the relative entropy of the probability distribution $P(i)$ from a Kronecker delta representing certainty that $i\; =\; m$ — i.e. the number of extra bits that must be transmitted to identify {{mvar|i}} if only the probability distribution $P(i)$ is available to the receiver, not the fact that $i\; =\; m$.

The mutual information,

$$\backslash begin\{align\}$$

\operatorname{I}(X; Y)

&= D_\text{KL}(P(X, Y) \parallel P(X)P(Y)) \\[5pt]

&= \operatorname{E}_X \{D_\text{KL}(P(Y \mid X) \parallel P(Y))\} \\[5pt]

&= \operatorname{E}_Y \{D_\text{KL}(P(X \mid Y) \parallel P(X))\}

\end{align}

is the relative entropy of the joint probability distribution $P(X,Y)$ from the product $P(X)P(Y)$ of the two marginal probability distributions — i.e. the expected number of extra bits that must be transmitted to identify {{mvar|X}} and {{mvar|Y}} if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability $P(X,Y)$ is known, it is the expected number of extra bits that must on average be sent to identify {{mvar|Y}} if the value of {{mvar|X}} is not already known to the receiver.

The Shannon entropy,

$$\backslash begin\{align\}$$

\Eta(X) &= \operatorname{E} \left[\operatorname{I}_X(x)\right] \\

&= \log N - D_\text{KL}{\left(p_X(x) \parallel P_U(X)\right)}

\end{align}

is the number of bits which would have to be transmitted to identify {{mvar|X}} from {{mvar|N}} equally likely possibilities, less the relative entropy of the uniform distribution on the random variates of {{mvar|X}}, $P\_U(X)$, from the true distribution $P(X)$ — i.e. less the expected number of bits saved, which would have had to be sent if the value of {{mvar|X}} were coded according to the uniform distribution $P\_U(X)$ rather than the true distribution $P(X)$. This definition of Shannon entropy forms the basis of E.T. Jaynes's alternative generalization to continuous distributions, the limiting density of discrete points (as opposed to the usual differential entropy), which defines the continuous entropy as

$$\backslash lim\_\{N\; \backslash to\; \backslash infty\}\; H\_\{N\}(X)\; =\; \backslash log\; N\; -\; \backslash int\; p(x)\backslash log\backslash frac\{p(x)\}\{m(x)\}\backslash ,dx,$$

which is equivalent to:

$$\backslash log(N)\; -\; D\_\backslash text\{KL\}(p(x)||m(x))$$

The conditional entropy{{r|CoverThomas}},

$$\backslash begin\{align\}$$

\Eta(X \mid Y)

&= \log N - D_\text{KL}(P(X, Y) \parallel P_U(X) P(Y)) \\[5pt]

&= \log N - D_\text{KL}(P(X, Y) \parallel P(X) P(Y)) - D_\text{KL}(P(X) \parallel P_U(X)) \\[5pt]

&= \Eta(X) - \operatorname{I}(X; Y) \\[5pt]

&= \log N - \operatorname{E}_Y \left[D_\text{KL}\left(P\left(X \mid Y\right) \parallel P_U(X)\right)\right]

\end{align}

is the number of bits which would have to be transmitted to identify {{mvar|X}} from {{mvar|N}} equally likely possibilities, less the relative entropy of the product distribution $P\_U(X)\; P(Y)$ from the true joint distribution $P(X,Y)$ — i.e. less the expected number of bits saved which would have had to be sent if the value of {{mvar|X}} were coded according to the uniform distribution $P\_U(X)$ rather than the conditional distribution $P(X|Y)$ of {{mvar|X}} given {{mvar|Y}}.

When we have a set of possible events, coming from the distribution {{mvar|p}}, we can encode them (with a lossless data compression) using entropy encoding. This compresses the data by replacing each fixed-length input symbol with a corresponding unique, variable-length, prefix-free code (e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be encoded as the bits (0, 10, 11)). If we know the distribution {{mvar|p}} in advance, we can devise an encoding that would be optimal (e.g.: using Huffman coding). Meaning the messages we encode will have the shortest length on average (assuming the encoded events are sampled from {{mvar|p}}), which will be equal to Shannon's Entropy of {{mvar|p}} (denoted as $\backslash Eta(p)$). However, if we use a different probability distribution ({{mvar|q}}) when creating the entropy encoding scheme, then a larger number of bits will be used (on average) to identify an event from a set of possibilities. This new (larger) number is measured by the cross entropy between {{mvar|p}} and {{mvar|q}}.

The cross entropy between two probability distributions ({{mvar|p}} and {{mvar|q}}) measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution {{mvar|q}}, rather than the "true" distribution {{mvar|p}}. The cross entropy for two distributions {{mvar|p}} and {{mvar|q}} over the same probability space is thus defined as follows.

$$\backslash Eta(p,\; q)\; =\; \backslash operatorname\{E\}\_p[-\backslash log\; q]\; =\; \backslash Eta(p)\; +\; D\_\backslash text\{KL\}(p\; \backslash parallel\; q).$$

For explicit derivation of this, see the Motivation section above.

Under this scenario, relative entropies (kl-divergence) can be interpreted as the extra number of bits, on average, that are needed (beyond $\backslash Eta(p)$) for encoding the events because of using {{mvar|q}} for constructing the encoding scheme instead of {{mvar|p}}.

In Bayesian statistics, relative entropy can be used as a measure of the information gain in moving from a prior distribution to a posterior distribution: $p(x)\; \backslash to\; p(x\backslash mid\; I)$. If some new fact $Y\; =\; y$ is discovered, it can be used to update the posterior distribution for {{mvar|X}} from $p(x\backslash mid\; I)$ to a new posterior distribution $p(x\backslash mid\; y,I)$ using Bayes' theorem:

$$p(x\; \backslash mid\; y,\; I)\; =\; \backslash frac\{p(y\; \backslash mid\; x,\; I)\; p(x\; \backslash mid\; I)\}\{p(y\; \backslash mid\; I)\}$$

This distribution has a new entropy:

$$\backslash Eta\backslash big(p(x\; \backslash mid\; y,\; I)\backslash big)\; =\; -\backslash sum\_x\; p(x\; \backslash mid\; y,\; I)\; \backslash log\; p(x\; \backslash mid\; y,\; I),$$

which may be less than or greater than the original entropy $\backslash Eta(p(x\backslash mid\; I))$. However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on $p(x\backslash mid\; I)$ instead of a new code based on $p(x\backslash mid\; y,\; I)$ would have added an expected number of bits:

$$D\_\backslash text\{KL\}\backslash big(p(x\; \backslash mid\; y,\; I)\; \backslash parallel\; p(x\; \backslash mid\; I)\; \backslash big)\; =\; \backslash sum\_x\; p(x\; \backslash mid\; y,\; I)\; \backslash log\; \backslash frac\{p(x\; \backslash mid\; y,\; I)\}\{p(x\; \backslash mid\; I)\}$$

to the message length. This therefore represents the amount of useful information, or information gain, about {{mvar|X}}, that has been learned by discovering $Y\; =\; y$.

If a further piece of data, $Y\_2\; =\; y\_2$, subsequently comes in, the probability distribution for {{mvar|x}} can be updated further, to give a new best guess $p(x\; \backslash mid\; y\_1,\; y\_2,\; I)$. If one reinvestigates the information gain for using $p(x\; \backslash mid\; y\_1,I)$ rather than $p(x\; \backslash mid\; I)$, it turns out that it may be either greater or less than previously estimated:

$$\backslash sum\_x\; p(x\; \backslash mid\; y\_1,\; y\_2,\; I)\; \backslash log\; \backslash frac\{p(x\; \backslash mid\; y\_1,\; y\_2,\; I)\}\{p(x\; \backslash mid\; I)\}$$ may be ≤ or > than $\backslash sum\_x\; p(x\; \backslash mid\; y\_1,\; I)\; \backslash log\; \backslash frac\{p(x\; \backslash mid\; y\_1,\; I)\}\{p(x\; \backslash mid\; I)\}$

and so the combined information gain does not obey the triangle inequality:

$$D\_\backslash text\{KL\}\; \backslash big(\; p(x\; \backslash mid\; y\_1,\; y\_2,\; I)\; \backslash parallel\; p(x\; \backslash mid\; I)\; \backslash big)$$ may be <, = or > than $D\_\backslash text\{KL\}\backslash big(\; p(x\; \backslash mid\; y\_1,\; y\_2,\; I)\; \backslash parallel\; p(x\; \backslash mid\; y\_1,\; I)\backslash big)\; +\; D\_\backslash text\{KL\}\backslash big(p(x\; \backslash mid\; y\_1,\; I)\; \backslash parallel\; p(x\; \backslash mid\; I)\backslash big)$

All one can say is that on average, averaging using $p(y\_2\; \backslash mid\; y\_1,\; x,\; I)$, the two sides will average out.

A common goal in Bayesian experimental design is to maximise the expected relative entropy between the prior and the posterior.{{cite journal | last1 = Chaloner | first1 = K. | last2 = Verdinelli | first2 = I. | year = 1995 | title = Bayesian experimental design: a review | journal = Statistical Science | volume = 10 | issue = 3| pages = 273–304 | doi = 10.1214/ss/1177009939 | doi-access = free | hdl = 11299/199630 | hdl-access = free }} When posteriors are approximated to be Gaussian distributions, a design maximising the expected relative entropy is called Bayes d-optimal.

Relative entropy $D\_\backslash text\{KL\}\backslash bigl(p(x\; \backslash mid\; H\_1)\; \backslash parallel\; p(x\; \backslash mid\; H\_0)\backslash bigr)$ can also be interpreted as the expected discrimination information for $H\_1$ over $H\_0$: the mean information per sample for discriminating in favor of a hypothesis $H\_1$ against a hypothesis $H\_0$, when hypothesis $H\_1$ is true.{{Cite book | last1=Press | first1 = W.H. | last2=Teukolsky | first2=S.A. | last3=Vetterling | first3=W.T. | last4=Flannery | first4=B.P. | year=2007 | title=Numerical Recipes: The Art of Scientific Computing | edition=3rd | publisher=Cambridge University Press | isbn=978-0-521-88068-8 | chapter=Section 14.7.2. Kullback–Leibler Distance | chapter-url=http://apps.nrbook.com/empanel/index.html#pg=756 | title-link = Numerical Recipes }} Another name for this quantity, given to it by I. J. Good, is the expected weight of evidence for $H\_1$ over $H\_0$ to be expected from each sample.

The expected weight of evidence for $H\_1$ over $H\_0$ is not the same as the information gain expected per sample about the probability distribution $p(H)$ of the hypotheses,

$$D\_\backslash text\{KL\}(p(x\; \backslash mid\; H\_1)\; \backslash parallel\; p(x\; \backslash mid\; H\_0))\; \backslash neq\; IG\; =\; D\_\backslash text\{KL\}(p(H\; \backslash mid\; x)\; \backslash parallel\; p(H\; \backslash mid\; I)).$$

Either of the two quantities can be used as a utility function in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.

On the entropy scale of information gain there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the logit scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the Riemann hypothesis is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of loss function for uncertainty are both useful, according to how well each reflects the particular circumstances of the problem in question.

The idea of relative entropy as discrimination information led Kullback to propose the Principle of {{visible anchor|Minimum Discrimination Information}} (MDI): given new facts, a new distribution {{mvar|f}} should be chosen which is as hard to discriminate from the original distribution $f\_0$ as possible; so that the new data produces as small an information gain $D\_\backslash text\{KL\}(f\; \backslash parallel\; f\_0)$ as possible.

For example, if one had a prior distribution $p(x,a)$ over {{mvar|x}} and {{mvar|a}}, and subsequently learnt the true distribution of {{mvar|a}} was $u(a)$, then the relative entropy between the new joint distribution for {{mvar|x}} and {{mvar|a}}, $q(x\backslash mid\; a)u(a)$, and the earlier prior distribution would be:

$$D\_\backslash text\{KL\}(q(x\; \backslash mid\; a)u(a)\; \backslash parallel\; p(x,\; a))\; =\; \backslash operatorname\{E\}\_\{u(a)\}\backslash left\backslash \{D\_\backslash text\{KL\}(q(x\; \backslash mid\; a)\; \backslash parallel\; p(x\; \backslash mid\; a))\backslash right\backslash \}\; +\; D\_\backslash text\{KL\}(u(a)\; \backslash parallel\; p(a)),$$

i.e. the sum of the relative entropy of $p(a)$ the prior distribution for {{mvar|a}} from the updated distribution $u(a)$, plus the expected value (using the probability distribution $u(a)$) of the relative entropy of the prior conditional distribution $p(x\backslash mid\; a)$ from the new conditional distribution $q(x\backslash mid\; a)$. (Note that often the later expected value is called the conditional relative entropy (or conditional Kullback–Leibler divergence) and denoted by $D\_\backslash text\{KL\}(q(x\backslash mid\; a)\; \backslash parallel\; p(x\backslash mid\; a))${{sfn|Kullback|1959}}{{citation|

last1=Cover|first1=Thomas M.|last2=Thomas |first2= Joy A.

|title=Elements of Information Theory

| date=1991|page=22

|publisher=John Wiley & Sons}}) This is minimized if $q(x\backslash mid\; a)=p(x\backslash mid\; a)$ over the whole support of $u(a)$; and we note that this result incorporates Bayes' theorem, if the new distribution $u(a)$ is in fact a δ function representing certainty that {{mvar|a}} has one particular value.

MDI can be seen as an extension of Laplace's Principle of Insufficient Reason, and the Principle of Maximum Entropy of E.T. Jaynes. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see differential entropy), but the relative entropy continues to be just as relevant.

In the engineering literature, MDI is sometimes called the Principle of Minimum Cross-Entropy (MCE) or Minxent for short. Minimising relative entropy from {{mvar|m}} to {{mvar|p}} with respect to {{mvar|m}} is equivalent to minimizing the cross-entropy of {{mvar|p}} and {{mvar|m}}, since

$$\backslash Eta(p,\; m)\; =\; \backslash Eta(p)\; +\; D\_\backslash text\{KL\}(p\; \backslash parallel\; m),$$

which is appropriate if one is trying to choose an adequate approximation to {{mvar|p}}. However, this is just as often not the task one is trying to achieve. Instead, just as often it is {{mvar|m}} that is some fixed prior reference measure, and {{mvar|p}} that one is attempting to optimise by minimising $D\_\backslash text\{KL\}(p\; \backslash parallel\; m)$ subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be $D\_\backslash text\{KL\}(p\; \backslash parallel\; m)$, rather than $\backslash Eta(p,m)$ {{Citation needed|date=January 2023}}.

File:ArgonKLdivergence.png

Surprisals{{Cite book |last=Tribus |first=Myron |url={{google books |plainurl=y |id=eyrYrQEACAAJ}} |title=Thermostatics and Thermodynamics: An Introduction to Energy, Information and States of Matter, with Engineering Applications |date=1959 |publisher=Van Nostrand |language=en}} add where probabilities multiply. The surprisal for an event of probability {{mvar|p}} is defined as $s\; =\; -\; k\; \backslash ln\; p$. If {{mvar|k}} is $\backslash left\backslash \{1,\; 1/\backslash ln\; 2,\; 1.38\; \backslash times\; 10^\{-23\}\backslash right\backslash \}$ then surprisal is in $\backslash \{$nats, bits, or $J/K\backslash \}$ so that, for instance, there are {{mvar|N}} bits of surprisal for landing all "heads" on a toss of {{mvar|N}} coins.

Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the average surprisal {{mvar|S}} (entropy) for a given set of control parameters (like pressure {{mvar|P}} or volume {{mvar|V}}). This constrained entropy maximization, both classically{{cite journal| last1 = Jaynes | first1 = E. T. | year = 1957 | title = Information theory and statistical mechanics | url = http://bayes.wustl.edu/etj/articles/theory.1.pdf| journal = Physical Review | volume = 106 | issue = 4| pages = 620–630 | doi = 10.1103/physrev.106.620 |bibcode = 1957PhRv..106..620J | s2cid = 17870175 }} and quantum mechanically,{{cite journal | last1 = Jaynes | first1 = E. T. | year = 1957 | title = Information theory and statistical mechanics II | url = http://bayes.wustl.edu/etj/articles/theory.2.pdf| journal = Physical Review | volume = 108 | issue = 2| pages = 171–190 | doi = 10.1103/physrev.108.171 |bibcode = 1957PhRv..108..171J }} minimizes Gibbs availability in entropy units{{Cite book |last=Gibbs |first=Josiah Willard |url={{google books |plainurl=y |id=6ijzXwAACAAJ}} |title=A Method of Geometrical Representation of the Thermodynamic Properties of Substances by Means of Surfaces |date=1871 |publisher=The Academy |language=en}} footnote page 52. $A\; \backslash equiv\; -k\; \backslash ln\; Z$ where {{mvar|Z}} is a constrained multiplicity or partition function.

When temperature {{mvar|T}} is fixed, free energy ($T\; \backslash times\; A$) is also minimized. Thus if $T,\; V$ and number of molecules {{mvar|N}} are constant, the Helmholtz free energy $F\; \backslash equiv\; U\; -\; TS$ (where {{mvar|U}} is energy and {{mvar|S}} is entropy) is minimized as a system "equilibrates." If {{mvar|T}} and {{mvar|P}} are held constant (say during processes in your body), the Gibbs free energy $G\; =\; U\; +\; PV\; -\; TS$ is minimized instead. The change in free energy under these conditions is a measure of available work that might be done in the process. Thus available work for an ideal gas at constant temperature $T\_o$ and pressure $P\_o$ is $W\; =\; \backslash Delta\; G\; =\; NkT\_o\backslash Theta(V/V\_o)$ where $V\_o\; =\; NkT\_o/P\_o$ and $\backslash Theta(x)\; =\; x\; -\; 1\; -\; \backslash ln\; x\; \backslash ge\; 0$ (see also Gibbs inequality).

More generally{{cite journal | last1 = Tribus | first1 = M. | last2 = McIrvine | first2 = E. C. | year = 1971 | title = Energy and information | journal = Scientific American | volume = 224 | issue = 3| pages = 179–186 | doi=10.1038/scientificamerican0971-179| bibcode = 1971SciAm.225c.179T }} the work available relative to some ambient is obtained by multiplying ambient temperature $T\_o$ by relative entropy or net surprisal $\backslash Delta\; I\; \backslash ge\; 0,$ defined as the average value of $k\backslash ln(p/p\_o)$ where $p\_o$ is the probability of a given state under ambient conditions. For instance, the work available in equilibrating a monatomic ideal gas to ambient values of $V\_o$ and $T\_o$ is thus $W\; =\; T\_o\; \backslash Delta\; I$, where relative entropy

$$\backslash Delta\; I\; =\; N\; k\; \backslash left[\backslash Theta\{\backslash left(\backslash frac\{V\}\{V\_o\}\backslash right)\}\; +\; \backslash frac\{3\}\{2\}\; \backslash Theta\{\backslash left(\backslash frac\{T\}\{T\_o\}\backslash right)\}\backslash right].$$

The resulting contours of constant relative entropy, shown at right for a mole of Argon at standard temperature and pressure, for example put limits on the conversion of hot to cold as in flame-powered air-conditioning or in the unpowered device to convert boiling-water to ice-water discussed here.{{cite journal | last1 = Fraundorf | first1 = P. | year = 2007 | title = Thermal roots of correlation-based complexity | url = http://www3.interscience.wiley.com/cgi-bin/abstract/117861985/ABSTRACT| archive-url = https://archive.today/20110813083358/http://www3.interscience.wiley.com/cgi-bin/abstract/117861985/ABSTRACT| url-status = dead| archive-date = 2011-08-13| journal = Complexity | volume = 13 | issue = 3| pages = 18–26 | doi= 10.1002/cplx.20195| bibcode = 2008Cmplx..13c..18F| arxiv = 1103.2481| s2cid = 20794688 }} Thus relative entropy measures thermodynamic availability in bits.

For density matrices {{mvar|P}} and {{mvar|Q}} on a Hilbert space, the quantum relative entropy from {{mvar|Q}} to {{mvar|P}} is defined to be

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; =\; \backslash operatorname\{Tr\}(P(\backslash log\; P\; -\; \backslash log\; Q)).$$

In quantum information science the minimum of $D\_\backslash text\{KL\}(P\backslash parallel\; Q)$ over all separable states {{mvar|Q}} can also be used as a measure of entanglement in the state {{mvar|P}}.

{{See also|Maximum likelihood estimation#Relation to minimizing Kullback–Leibler divergence and cross entropy}}

{{further|Model selection}}

Just as relative entropy of "actual from ambient" measures thermodynamic availability, relative entropy of "reality from a model" is also useful even if the only clues we have about reality are some experimental measurements. In the former case relative entropy describes distance to equilibrium or (when multiplied by ambient temperature) the amount of available work, while in the latter case it tells you about surprises that reality has up its sleeve or, in other words, how much the model has yet to learn.

Although this tool for evaluating models against systems that are accessible experimentally may be applied in any field, its application to selecting a statistical model via Akaike information criterion are particularly well described in papers{{cite journal| last1 = Burnham | first1 = K.P. | last2 = Anderson | first2 = D.R. | year = 2001 | title = Kullback–Leibler information as a basis for strong inference in ecological studies | journal = Wildlife Research | volume = 28 | issue = 2| pages = 111–119 | doi = 10.1071/WR99107 | doi-access = free }} and a book{{Cite book |last=Burnham|first=Kenneth P. |url=http://worldcat.org/oclc/878132909 |title=Model selection and multimodel inference : a practical information-theoretic approach |date=December 2010 |publisher=Springer |isbn=978-1-4419-2973-0 |oclc=878132909}} by Burnham and Anderson. In a nutshell the relative entropy of reality from a model may be estimated, to within a constant additive term, by a function of the deviations observed between data and the model's predictions (like the mean squared deviation) . Estimates of such divergence for models that share the same additive term can in turn be used to select among models.

When trying to fit parametrized models to data there are various estimators which attempt to minimize relative entropy, such as maximum likelihood and maximum spacing estimators.{{citation needed|date=January 2020}}

{{harvtxt|Kullback|Leibler|1951}}

also considered the symmetrized function:{{sfn|Kullback|Leibler|1951|p=80}}

$$D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\; +\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; P)$$

which they referred to as the "divergence", though today the "KL divergence" refers to the asymmetric function (see {{slink||Etymology}} for the evolution of the term). This function is symmetric and nonnegative, and had already been defined and used by Harold Jeffreys in 1948;{{sfn|Jeffreys|1948|p=158}} it is accordingly called the Jeffreys divergence.

This quantity has sometimes been used for feature selection in classification problems, where {{mvar|P}} and {{mvar|Q}} are the conditional pdfs of a feature under two different classes. In the Banking and Finance industries, this quantity is referred to as Population Stability Index (PSI), and is used to assess distributional shifts in model features through time.

An alternative is given via the $\backslash lambda$-divergence,

$$D\_\backslash lambda(P\; \backslash parallel\; Q)\; =\; \backslash lambda\; D\_\backslash text\{KL\}(P\; \backslash parallel\; \backslash lambda\; P\; +\; (1\; -\; \backslash lambda)Q)\; +\; (1\; -\; \backslash lambda)\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; \backslash lambda\; P\; +\; (1\; -\; \backslash lambda)Q),$$

which can be interpreted as the expected information gain about {{mvar|X}} from discovering which probability distribution {{mvar|X}} is drawn from, {{mvar|P}} or {{mvar|Q}}, if they currently have probabilities $\backslash lambda$ and $1-\backslash lambda$ respectively.{{Clarify|date=May 2018}} {{Citation needed|date=May 2018}}

The value $\backslash lambda\; =\; 0.5$ gives the Jensen–Shannon divergence, defined by

$$D\_\backslash text\{JS\}\; =\; \backslash tfrac\{1\}\{2\}\; D\_\backslash text\{KL\}\; (P\; \backslash parallel\; M)\; +\; \backslash tfrac\{1\}\{2\}\; D\_\backslash text\{KL\}(Q\; \backslash parallel\; M)$$

where {{mvar|M}} is the average of the two distributions,

$$M\; =\; \backslash tfrac\{1\}\{2\}\; \backslash left(P\; +\; Q\backslash right).$$

We can also interpret $D\_\backslash text\{JS\}$ as the capacity of a noisy information channel with two inputs giving the output distributions {{mvar|P}} and {{mvar|Q}}. The Jensen–Shannon divergence, like all {{mvar|f}}-divergences, is locally proportional to the Fisher information metric. It is similar to the Hellinger metric (in the sense that it induces the same affine connection on a statistical manifold).

Furthermore, the Jensen–Shannon divergence can be generalized using abstract statistical M-mixtures relying on an abstract mean M.{{cite journal | last1=Nielsen |first1=Frank | year = 2019 | title = On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means | journal = Entropy | volume = 21 |issue=5 | pages=485 | doi=10.3390/e21050485 | pmid=33267199 | pmc=7514974 | arxiv=1904.04017 | bibcode=2019Entrp..21..485N | doi-access=free}}{{cite journal | last1=Nielsen |first1=Frank | year = 2020 | title = On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid | journal = Entropy | volume = 22 |issue=2 | pages=221 | doi=10.3390/e22020221|pmid=33285995 | pmc=7516653 | arxiv=1912.00610 | bibcode=2020Entrp..22..221N | doi-access=free}}

There are many other important measures of probability distance. Some of these are particularly connected with relative entropy. For example:

• The total-variation distance, $\backslash delta(p,q)$. This is connected to the divergence through Pinsker's inequality: $$\backslash delta(P,\; Q)\; \backslash le\; \backslash sqrt\{\backslash tfrac\{1\}\{2\}\; D\_\backslash text\{KL\}(P\; \backslash parallel\; Q)\}.$$ Pinsker's inequality is vacuous for any distributions where $D\_\{\backslash mathrm\{KL\}\}(P\backslash parallel\; Q)>2$, since the total variation distance is at most {{mvar|1}}. For such distributions, an alternative bound can be used, due to Bretagnolle and Huber{{Citation |last1=Bretagnolle |first1=J. |title=Séminaire de Probabilités XII |date=1978 |url=http://dx.doi.org/10.1007/bfb0064610 |series=Lecture Notes in Mathematics |pages=342–363 |place=Berlin, Heidelberg |publisher=Springer Berlin Heidelberg |isbn=978-3-540-08761-8 |access-date=2023-02-14 |last2=Huber |first2=C.|chapter=Estimation des densités : Risque minimax |volume=649 |doi=10.1007/bfb0064610 |s2cid=122597694 |language=fr}} Lemma 2.1 (see, also, Tsybakov{{Cite book |last=B.) |first=Tsybakov, A. B. (Alexandre |url=http://worldcat.org/oclc/757859245 |title=Introduction to nonparametric estimation |date=2010 |publisher=Springer |isbn=978-1-4419-2709-5 |oclc=757859245}} Equation 2.25.): $$\backslash delta(P,Q)\; \backslash le\; \backslash sqrt\{1-e^\{\; -D\_\{\backslash mathrm\{KL\}\}(P\backslash parallel\; Q)\; \}\}.$$
• The family of Rényi divergences generalize relative entropy. Depending on the value of a certain parameter, $\backslash alpha$, various inequalities may be deduced.

Other notable measures of distance include the Hellinger distance, histogram intersection, Chi-squared statistic, quadratic form distance, match distance, Kolmogorov–Smirnov distance, and earth mover's distance.{{cite journal | last1 = Rubner | first1 = Y. | author-link3 = Leonidas J. Guibas | last2 = Tomasi | first2 = C. | last3 = Guibas | first3 = L. J. | year = 2000 | title = The earth mover's distance as a metric for image retrieval | journal = International Journal of Computer Vision | volume = 40 | issue = 2| pages = 99–121 | doi = 10.1023/A:1026543900054 | s2cid = 14106275 }}

{{Main|Data differencing}}

Just as absolute entropy serves as theoretical background for data compression, relative entropy serves as theoretical background for data differencing – the absolute entropy of a set of data in this sense being the data required to reconstruct it (minimum compressed size), while the relative entropy of a target set of data, given a source set of data, is the data required to reconstruct the target given the source (minimum size of a patch).

{{Div col|colwidth=20em}}

• Akaike information criterion
• Bayesian information criterion
• Bregman divergence
• Cross-entropy
• Deviance information criterion
• Entropic value at risk
• Entropy power inequality
• Hellinger distance
• Information gain in decision trees
• Information gain ratio
• Information theory and measure theory
• Jensen–Shannon divergence
• Quantum relative entropy
• Solomon Kullback and Richard Leibler
• Bhattacharyya distance

{{div col end}}

{{Reflist|30em}}

{{refbegin}}

• {{cite book | last = Amari | first = Shun-ichi | author-link = Shun'ichi Amari | title = Information Geometry and Its Applications | year = 2016 | series = Applied Mathematical Sciences | volume = 194 | publisher = Springer Japan | isbn = 978-4-431-55977-1 | doi = 10.1007/978-4-431-55978-8 | pages = XIII, 374 }}
• {{citation|last1=Kullback|first1=Solomon |title=Information Theory and Statistics |date=1959 |publisher=John Wiley & Sons}}. Republished by Dover Publications in 1968; reprinted in 1978: {{isbn|0-8446-5625-9}}.
• {{cite book |last=Jeffreys |first=Harold |title=Theory of Probability |edition=Second |date=1948 |publisher=Oxford University Press}}

{{refend}}

• [https://bitbucket.org/szzoli/ite/ Information Theoretical Estimators Toolbox]
• [https://github.com/evansenter/diverge Ruby gem for calculating Kullback–Leibler divergence]
• [https://arxiv.org/abs/1404.2000 Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory]
• [http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file Matlab code for calculating Kullback–Leibler divergence for discrete distributions]
• Sergio Verdú, [http://videolectures.net/nips09_verdu_re/ Relative Entropy], NIPS 2009. One-hour video lecture.
• [https://arxiv.org/abs/math/0604246 A modern summary of info-theoretic divergence measures]

{{DEFAULTSORT:Kullback-Leibler Divergence}}

Category:Entropy and information

Category:F-divergences

Category:Information geometry

Category:Thermodynamics