Kullback's inequality

Let P and Q be probability distributions (measures) on the real line, whose first moments exist, and such that P << Q. Consider the natural exponential family of Q given by

$$Q\_\backslash theta(A)\; =\; \backslash frac\{\backslash int\_A\; e^\{\backslash theta\; x\}Q(dx)\}\{\backslash int\_\{-\backslash infty\}^\backslash infty\; e^\{\backslash theta\; x\}Q(dx)\}$$

= \frac{1}{M_Q(\theta)} \int_A e^{\theta x}Q(dx)

for every measurable set A, where $M\_Q$ is the moment-generating function of Q. (Note that Q₀ = Q.) Then

$$D\_\{KL\}(P\backslash parallel\; Q)\; =\; D\_\{KL\}(P\backslash parallel\; Q\_\backslash theta)$$

+ \int_{\operatorname{supp}P}\left(\log\frac{\mathrm dQ_\theta}{\mathrm dQ}\right)\mathrm dP.

By Gibbs' inequality we have $D\_\{KL\}(P\backslash parallel\; Q\_\backslash theta)\; \backslash ge\; 0$ so that

$$D\_\{KL\}(P\backslash parallel\; Q)\; \backslash ge$$

\int_{\operatorname{supp}P}\left(\log\frac{\mathrm dQ_\theta}{\mathrm dQ}\right)\mathrm dP

= \int_{\operatorname{supp}P}\left(\log\frac{e^{\theta x}}{M_Q(\theta)}\right) P(dx)

Simplifying the right side, we have, for every real θ where

$$D\_\{KL\}(P\backslash parallel\; Q)\; \backslash ge\; \backslash mu\text{'}\_1(P)\; \backslash theta\; -\; \backslash Psi\_Q(\backslash theta),$$

where $\backslash mu\text{'}\_1(P)$ is the first moment, or mean, of P, and $\backslash Psi\_Q\; =\; \backslash log\; M\_Q$ is called the cumulant-generating function. Taking the supremum completes the process of convex conjugation and yields the rate function:

$$D\_\{KL\}(P\backslash parallel\; Q)\; \backslash ge\; \backslash sup\_\backslash theta\; \backslash left\backslash \{\; \backslash mu\text{'}\_1(P)\; \backslash theta\; -\; \backslash Psi\_Q(\backslash theta)\; \backslash right\backslash \}$$

= \Psi_Q^*(\mu'_1(P)).

{{main|Cramér–Rao bound}}

Let X_θ be a family of probability distributions on the real line indexed by the real parameter θ, and satisfying certain regularity conditions. Then

$$\backslash lim\_\{h\backslash to\; 0\}\; \backslash frac\; \{D\_\{KL\}(X\_\{\backslash theta+h\}\; \backslash parallel\; X\_\backslash theta)\}\; \{h^2\}$$

\ge \lim_{h\to 0} \frac {\Psi^*_\theta (\mu_{\theta+h})}{h^2},

where $\backslash Psi^*\_\backslash theta$ is the convex conjugate of the cumulant-generating function of $X\_\backslash theta$ and $\backslash mu\_\{\backslash theta+h\}$ is the first moment of $X\_\{\backslash theta+h\}.$

The left side of this inequality can be simplified as follows:

$$\backslash begin\{align\}$$

\lim_{h\to 0} \frac {D_{KL}(X_{\theta+h}\parallel X_\theta)} {h^2} &=\lim_{h\to 0} \frac 1 {h^2} \int_{-\infty}^\infty \log \left( \frac{\mathrm dX_{\theta+h}}{\mathrm dX_\theta} \right) \mathrm dX_{\theta+h} \\

&=-\lim_{h\to 0} \frac 1 {h^2} \int_{-\infty}^\infty \log \left( \frac{\mathrm dX_{\theta}}{\mathrm dX_{\theta+h}} \right) \mathrm dX_{\theta+h} \\

&=-\lim_{h\to 0} \frac 1 {h^2} \int_{-\infty}^\infty \log\left( 1- \left (1-\frac{\mathrm dX_{\theta}}{\mathrm dX_{\theta+h}} \right ) \right) \mathrm dX_{\theta+h} \\

&= \lim_{h\to 0} \frac 1 {h^2} \int_{-\infty}^\infty \left[ \left( 1 - \frac{\mathrm dX_\theta}{\mathrm dX_{\theta+h}} \right) +\frac 1 2 \left( 1 - \frac{\mathrm dX_\theta}{\mathrm dX_{\theta+h}} \right) ^ 2

+ o \left( \left( 1 - \frac{\mathrm dX_\theta}{\mathrm dX_{\theta+h}} \right) ^ 2 \right) \right]\mathrm dX_{\theta+h} && \text{Taylor series for } \log(1-t) \\

&= \lim_{h\to 0} \frac 1 {h^2} \int_{-\infty}^\infty \left[ \frac 1 2 \left( 1 - \frac{\mathrm dX_\theta}{\mathrm dX_{\theta+h}} \right)^2 \right]\mathrm dX_{\theta+h} \\

&= \lim_{h\to 0} \frac 1 {h^2} \int_{-\infty}^\infty \left[ \frac 1 2 \left( \frac{\mathrm dX_{\theta+h} - \mathrm dX_\theta}{\mathrm dX_{\theta+h}} \right)^2 \right]\mathrm dX_{\theta+h} \\

&= \frac 1 2 \mathcal I_X(\theta)

\end{align}

which is half the Fisher information of the parameter θ.

The right side of the inequality can be developed as follows:

\lim_{h\to 0} \frac {\Psi^*_\theta (\mu_{\theta+h})}{h^2}

= \lim_{h\to 0} \frac 1 {h^2} {\sup_t \{\mu_{\theta+h}t - \Psi_\theta(t)\} }.

This supremum is attained at a value of t=τ where the first derivative of the cumulant-generating function is $\backslash Psi\text{'}\_\backslash theta(\backslash tau)\; =\; \backslash mu\_\{\backslash theta+h\},$ but we have $\backslash Psi\text{'}\_\backslash theta(0)\; =\; \backslash mu\_\backslash theta,$ so that

$$\backslash Psi\text{'}\text{'}\_\backslash theta(0)\; =\; \backslash frac\{d\backslash mu\_\backslash theta\}\{d\backslash theta\}\; \backslash lim\_\{h\; \backslash to\; 0\}\; \backslash frac\; h\; \backslash tau.$$

Moreover,

$$\backslash lim\_\{h\backslash to\; 0\}\; \backslash frac\; \{\backslash Psi^*\_\backslash theta\; (\backslash mu\_\{\backslash theta+h\})\}\{h^2\}$$

= \frac 1 {2\Psi''_\theta(0)}\left(\frac {d\mu_\theta}{d\theta}\right)^2

= \frac 1 {2\operatorname{Var}(X_\theta)}\left(\frac {d\mu_\theta}{d\theta}\right)^2.

We have:

$$\backslash frac\; 1\; 2\; \backslash mathcal\; I\_X(\backslash theta)$$

\ge \frac 1 {2\operatorname{Var}(X_\theta)}\left(\frac {d\mu_\theta}{d\theta}\right)^2,

which can be rearranged as:

$$\backslash operatorname\{Var\}(X\_\backslash theta)\; \backslash ge\; \backslash frac\{(d\backslash mu\_\backslash theta\; /\; d\backslash theta)^2\}\; \{\backslash mathcal\; I\_X(\backslash theta)\}.$$

• Kullback–Leibler divergence
• Cramér–Rao bound
• Fisher information
• Large deviations theory
• Convex conjugate
• Rate function
• Moment-generating function

{{DEFAULTSORT:Kullback's Inequality}}

Category:Information theory

Category:Statistical inequalities

Category:Estimation theory