differential entropy#Maximization in the normal distribution

Let $X$ be a random variable with a probability density function $f$ whose support is a set $\backslash mathcal\; X$. The differential entropy $h(X)$ or $h(f)$ is defined as{{cite book|first1=Thomas M.|first2=Joy A.|last1=Cover|last2=Thomas|isbn=0-471-06259-6|title=Elements of Information Theory|year=1991|publisher=Wiley|location=New York|url=https://archive.org/details/elementsofinform0000cove|url-access=registration}}{{rp|p=243}}

{{Equation box 1

|indent = :

|title=

|equation = $h(X)\; =\; \backslash operatorname\{E\}[-\backslash log\; (f(X))]\; =\; -\backslash int\_\backslash mathcal\{X\}\; f(x)\backslash log\; f(x)\backslash ,dx$

|cellpadding= 6

|border

|border colour = #0073CF

}}

For probability distributions which do not have an explicit density function expression, but have an explicit quantile function expression, $Q(p)$, then $h(Q)$ can be defined in terms of the derivative of $Q(p)$ i.e. the quantile density function $Q\text{'}(p)$ as{{Citation |last1=Vasicek |first1=Oldrich |year=1976 |title=A Test for Normality Based on Sample Entropy |journal=Journal of the Royal Statistical Society, Series B |volume=38 |issue=1 |pages=54–59 |doi=10.1111/j.2517-6161.1976.tb01566.x |jstor=2984828 |postscript=. }}{{rp|pp=54–59}}

$$h(Q)\; =\; \backslash int\_0^1\; \backslash log\; Q\text{'}(p)\backslash ,dp.$$

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure $X$.{{cite book |last=Gibbs |first=Josiah Willard |author-link=Josiah Willard Gibbs |title=Elementary Principles in Statistical Mechanics, developed with especial reference to the rational foundation of thermodynamics |year=1902 |publisher=Charles Scribner's Sons |location=New York}}{{rp|pp=183–184}} For example, the differential entropy of a quantity measured in millimeters will be {{not a typo|log(1000)}} more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of {{not a typo|log(1000)}} more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution $\backslash mathcal\{U\}(0,1/2)$ has negative differential entropy; i.e., it is better ordered than $\backslash mathcal\{U\}(0,1)$ as shown now

$$\backslash int\_0^\backslash frac\{1\}\{2\}\; -2\backslash log(2)\backslash ,dx\; =\; -\backslash log(2)\backslash ,$$

being less than that of $\backslash mathcal\{U\}(0,1)$ which has zero differential entropy. Thus, differential entropy does not share all properties of discrete entropy.

The continuous mutual information $I(X;Y)$ has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of $X$ and $Y$ as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps),{{cite journal

| first = Alexander

| last = Kraskov

|author2=Stögbauer, Grassberger

| year = 2004

| title = Estimating mutual information

| journal = Physical Review E

| volume = 60

| issue = 6

| pages = 066138

| doi =10.1103/PhysRevE.69.066138

| pmid = 15244698

|arxiv = cond-mat/0305641 |bibcode = 2004PhRvE..69f6138K | s2cid = 1269438

}} including linear{{ cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | orig-year = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&q=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22&pg=PA8 }} transformations of $X$ and $Y$, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points.

• For probability densities $f$ and $g$, the Kullback–Leibler divergence $D\_\{KL\}(f\; \backslash parallel\; g)$ is greater than or equal to 0 with equality only if $f=g$ almost everywhere. Similarly, for two random variables $X$ and $Y$, $I(X;Y)\; \backslash ge\; 0$ and $h(X\backslash mid\; Y)\; \backslash le\; h(X)$ with equality if and only if $X$ and $Y$ are independent.
• The chain rule for differential entropy holds as in the discrete case{{rp|p=253}} $$h(X\_1,\; \backslash ldots,\; X\_n)\; =\; \backslash sum\_\{i=1\}^n\; h(X\_i\backslash mid\; X\_1,\; \backslash ldots,\; X\_\{i-1\})\; \backslash leq\; \backslash sum\_\{i=1\}^\{n\}\; h(X\_i).$$
• Differential entropy is translation invariant, i.e. for a constant $c$.{{rp|p=253}} $$h(X+c)\; =\; h(X)$$
• Differential entropy is in general not invariant under arbitrary invertible maps.{{pb}}In particular, for a constant $a$, $$h(aX)\; =\; h(X)+\; \backslash log\; |a|$${{pb}}For a vector valued random variable $\backslash mathbf\{X\}$ and an invertible (square) matrix $\backslash mathbf\{A\}${{rp|p=253}} $$h(\backslash mathbf\{A\}\backslash mathbf\{X\})\; =\; h(\backslash mathbf\{X\})+\backslash log\; \backslash left(\; \backslash left|\backslash det\; \backslash mathbf\{A\}\backslash right|\; \backslash right)$$
  • In general, for a transformation from a random vector to another random vector with same dimension $\backslash mathbf\{Y\}=m\; \backslash left(\backslash mathbf\{X\}\backslash right)$, the corresponding entropies are related via $$h(\backslash mathbf\{Y\})\; \backslash leq\; h(\backslash mathbf\{X\})\; +\; \backslash int\; f(x)\; \backslash log\; \backslash left\backslash vert\; \backslash frac\{\backslash partial\; m\}\{\backslash partial\; x\}\; \backslash right\backslash vert\; \backslash ,\; dx$$ where $\backslash left\backslash vert\; \backslash frac\{\backslash partial\; m\}\{\backslash partial\; x\}\; \backslash right\backslash vert$ is the Jacobian of the transformation $m$.{{cite web |title=proof of upper bound on differential entropy of f(X) |work=Stack Exchange |date=April 16, 2016 |url=https://math.stackexchange.com/q/1745670 }} The above inequality becomes an equality if the transform is a bijection. Furthermore, when $m$ is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and $h(Y)=h(X)$.
  • If a random vector $X\; \backslash in\; \backslash mathbb\{R\}^n$ has mean zero and covariance matrix $K$, $h(\backslash mathbf\{X\})\; \backslash leq\; \backslash frac\{1\}\{2\}\; \backslash log(\backslash det\{2\; \backslash pi\; e\; K\})\; =\; \backslash frac\{1\}\{2\}\; \backslash log[(2\backslash pi\; e)^n\; \backslash det\{K\}]$ with equality if and only if $X$ is jointly gaussian (see below).{{rp|p=254}}

  However, differential entropy does not have other desirable properties:

  • It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
  • It can be negative.

  A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

With a normal distribution, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.{{rp|p=255}}

Let $g(x)$ be a Gaussian PDF with mean {{mvar|μ}} and variance $\backslash sigma^2$ and $f(x)$ an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that $f(x)$ has the same mean of $\backslash mu$ as {{nowrap|$g(x)$.}}

Consider the Kullback–Leibler divergence between the two distributions

$$0\; \backslash leq\; D\_\{KL\}(f\; \backslash parallel\; g)\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; f(x)\; \backslash log\; \backslash left(\; \backslash frac\{f(x)\}\{g(x)\}\; \backslash right)\; \backslash ,\; dx\; =\; -h(f)\; -\; \backslash int\_\{-\backslash infty\}^\backslash infty\; f(x)\backslash log(g(x))\; \backslash ,\; dx.$$

Now note that

$$\backslash begin\{align\}$$

\int_{-\infty}^\infty f(x)\log(g(x)) \, dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) \, dx \\

&= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx \,+\, \log(e)\int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) \, dx \\

&= -\tfrac{1}{2}\log(2\pi\sigma^2) - \log(e)\frac{\sigma^2}{2\sigma^2} \\[1ex]

&= -\tfrac{1}{2}\left(\log(2\pi\sigma^2) + \log(e)\right) \\[1ex]

&= -\tfrac{1}{2}\log(2\pi e \sigma^2) \\[1ex]

&= -h(g)

\end{align}

because the result does not depend on $f(x)$ other than through the variance. Combining the two results yields

$$h(g)\; -\; h(f)\; \backslash geq\; 0\; \backslash !$$

with equality when $f(x)\; =\; g(x)$ following from the properties of Kullback–Leibler divergence.

This result may also be demonstrated using the calculus of variations. A Lagrangian function with two Lagrangian multipliers may be defined as:

$$L\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; g(x)\; \backslash log(g(x))\; \backslash ,\; dx\; -\; \backslash lambda\_0\; \backslash left(1-\backslash int\_\{-\backslash infty\}^\backslash infty\; g(x)\; \backslash ,\; dx\backslash right)\; -\; \backslash lambda\; \backslash left(\backslash sigma^2\; -\; \backslash int\_\{-\backslash infty\}^\backslash infty\; g(x)(x-\backslash mu)^2\backslash ,dx\backslash right)$$

where {{math|g(x)}} is some function with mean {{mvar|μ}}. When the entropy of {{math|g(x)}} is at a maximum and the constraint equations, which consist of the normalization condition $\backslash left(1=\backslash int\_\{-\backslash infty\}^\backslash infty\; g(x)\backslash ,dx\backslash right)$ and the requirement of fixed variance $\backslash left(\backslash sigma^2\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; g(x)(x-\backslash mu)^2\backslash ,dx\backslash right)$, are both satisfied, then a small variation {{math|δg(x)}} about {{math|g(x)}} will produce a variation {{math|δL}} about {{math|L}} which is equal to zero:

$$0=\backslash delta\; L=\backslash int\_\{-\backslash infty\}^\backslash infty\; \backslash delta\; g(x)\; \backslash left[\backslash log(g(x))\; +\; 1\; +\; \backslash lambda\_0\; +\; \backslash lambda(x-\backslash mu)^2\backslash right]\backslash ,dx$$

Since this must hold for any small {{math|δg(x)}}, the term in brackets must be zero, and solving for {{math|g(x)}} yields:

$$g(x)\; =\; e^\{-\backslash lambda\_0\; -\; 1\; -\; \backslash lambda\; (x-\backslash mu)^2\}$$

Using the constraint equations to solve for {{math|λ₀}} and {{mvar|λ}} yields the normal distribution:

$$g(x)\; =\; \backslash frac\{1\}\{\backslash sqrt\{2\backslash pi\backslash sigma^2\}\}e^\{-\backslash frac\{(x-\backslash mu)^2\}\{2\backslash sigma^2\}\}$$

Let $X$ be an exponentially distributed random variable with parameter $\backslash lambda$, that is, with probability density function

$$f(x)\; =\; \backslash lambda\; e^\{-\backslash lambda\; x\}\; \backslash text\{\; for\; \}\; x\; \backslash geq\; 0.$$

Its differential entropy is then

$$\backslash begin\{align\}$$

h_e(X)

&= -\int_0^\infty \lambda e^{-\lambda x} \log \left(\lambda e^{-\lambda x}\right) dx \\[2pt]

&= -\left(\int_0^\infty (\log \lambda)\lambda e^{-\lambda x}\,dx + \int_0^\infty (-\lambda x) \lambda e^{-\lambda x}\,dx\right) \\[2pt]

&= -\log \lambda \int_0^\infty f(x)\,dx + \lambda \operatorname{E}[X] \\[4pt]

&= -\log\lambda + 1\,.

\end{align}

Here, $h\_e(X)$ was used rather than $h(X)$ to make it explicit that the logarithm was taken to base {{math|e}}, to simplify the calculation.

The differential entropy yields a lower bound on the expected squared error of an estimator. For any random variable $X$ and estimator $\backslash widehat\{X\}$ the following holds:

$$\backslash operatorname\{E\}[(X\; -\; \backslash widehat\{X\})^2]\; \backslash ge\; \backslash frac\{1\}\{2\backslash pi\; e\}e^\{2h(X)\}$$

with equality if and only if $X$ is a Gaussian random variable and $\backslash widehat\{X\}$ is the mean of $X$.

In the table below $\backslash Gamma(x)\; =\; \backslash int\_0^\{\backslash infty\}\; e^\{-t\}\; t^\{x-1\}\; dt$ is the gamma function, $\backslash psi(x)\; =\; \backslash frac\{d\}\{dx\}\; \backslash log\backslash Gamma(x)=\backslash frac\{\backslash Gamma\text{'}(x)\}\{\backslash Gamma(x)\}$ is the digamma function, $B(p,q)\; =\; \backslash frac\{\backslash Gamma(p)\backslash Gamma(q)\}\{\backslash Gamma(p+q)\}$ is the beta function, and {{math|γ_E}} is Euler's constant.{{cite journal |last1=Park |first1=Sung Y. |last2=Bera |first2=Anil K. |year=2009 |title=Maximum entropy autoregressive conditional heteroskedasticity model |journal=Journal of Econometrics |volume=150 |issue=2 |pages=219–230 |publisher=Elsevier |doi=10.1016/j.jeconom.2008.12.014 |url=http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper-masterdownload%5C2009519932327055475115776.pdf |access-date=2011-06-02 |archive-url=https://web.archive.org/web/20160307144515/http://wise.xmu.edu.cn/uploadfiles/paper-masterdownload/2009519932327055475115776.pdf |archive-date=2016-03-07 |url-status=dead }}{{rp|pp=219–230}}

|-

| Logistic || $f(x)\; =\; \backslash frac\{e^\{-x/s\}\}\{s\{\backslash left(1\; +\; e^\{-x/s\}\backslash right)\}^2\}$|| $\backslash log\; s\; +\; 2\; \backslash ,$||$(-\backslash infty,\backslash infty)\backslash ,$

|-

| Lognormal || $f(x)\; =\; \backslash frac\{1\}\{\backslash sigma\; x\; \backslash sqrt\{2\backslash pi\}\}\; \backslash exp\backslash left(-\backslash frac\{(\backslash log\; x\; -\; \backslash mu)^2\}\{2\backslash sigma^2\}\backslash right)$ || $\backslash mu\; +\; \backslash tfrac\{1\}\{2\}\; \backslash log(2\backslash pi\; e\; \backslash sigma^2)$||$[0,\backslash infty)\backslash ,$

|-

| Maxwell–Boltzmann || $f(x)\; =\; \backslash frac\{1\}\{a^3\}\backslash sqrt\{\backslash frac\{2\}\{\backslash pi\}\}\backslash ,x^\{2\}\backslash exp\backslash left(-\backslash frac\{x^2\}\{2a^2\}\backslash right)$ || $\backslash log(a\backslash sqrt\{2\backslash pi\})\; +\; \backslash gamma\_E\; -\; \backslash tfrac\{1\}\{2\}$||$[0,\backslash infty)\backslash ,$

|-

| Generalized normal || $f(x)\; =\; \backslash frac\{2\; \backslash beta^\{\backslash frac\{\backslash alpha\}\{2\}\}\}\{\backslash Gamma(\backslash frac\{\backslash alpha\}\{2\})\}\; x^\{\backslash alpha\; -\; 1\}\; \backslash exp\backslash left(-\backslash beta\; x^2\backslash right)$|| $\backslash log\{\backslash frac\{\backslash Gamma(\backslash alpha/2)\}\{2\backslash beta^\{\backslash frac\{1\}\{2\}\}\}\}\; -\; \backslash frac\{\backslash alpha\; -\; 1\}\{2\}\; \backslash psi\backslash left(\backslash frac\{\backslash alpha\}\{2\}\backslash right)\; +\; \backslash frac\{\backslash alpha\}\{2\}$||$(-\backslash infty,\backslash infty)\backslash ,$

|-

| Pareto || $f(x)\; =\; \backslash frac\{\backslash alpha\; x\_m^\backslash alpha\}\{x^\{\backslash alpha+1\}\}$ || $\backslash log\; \backslash frac\{x\_m\}\{\backslash alpha\}\; +\; 1\; +\; \backslash frac\{1\}\{\backslash alpha\}$||$[x\_m,\backslash infty)\backslash ,$

|-

| Student's t || $f(x)\; =\; \backslash frac\{(1\; +\; x^2/\backslash nu)^\{-\backslash frac\{\backslash nu+1\}\{2\}\}\}\{\backslash sqrt\{\backslash nu\}B(\backslash frac\{1\}\{2\},\backslash frac\{\backslash nu\}\{2\})\}$ || $\backslash begin\{align\}$

&\frac{\nu\!+\!1}{2} \left[\psi{\left(\frac{\nu\!+\!1}{2}\right)} - \psi{\left(\frac{\nu}{2}\right)}\right]\\

&+\!\log \sqrt{\nu} B{\left(\frac{1}{2},\frac{\nu}{2}\right)}

\end{align}||$(-\backslash infty,\backslash infty)\backslash ,$

|-

| Triangular || $f(x)\; =\; \backslash begin\{cases\}$

\frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt]

\frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c < x \le b, \\[4pt]

\end{cases} || $\backslash frac\{1\}\{2\}\; +\; \backslash log\; \backslash frac\{b-a\}\{2\}$||$[a,b]\backslash ,$

|-

| Weibull || $f(x)\; =\; \backslash frac\{k\}\{\backslash lambda^k\}\; x^\{k-1\}\; \backslash exp\backslash left(-\backslash frac\{x^k\}\{\backslash lambda^k\}\backslash right)$ || $\backslash frac\{k-1\}\{k\}\backslash gamma\_E\; +\; \backslash log\; \backslash frac\{\backslash lambda\}\{k\}\; +\; 1$||$[0,\backslash infty)\backslash ,$

|-

| Multivariate normal || $$

f_X(\mathbf{x}) = \frac{\exp \left[ -\frac{1}{2} ( \mathbf{x} - \mathbf{\mu})^\mathsf{T} \Sigma^{-1} (\mathbf{x} - \mathbf{\mu}) \right]} {{\left(2\pi\right)}^{N/2} \left|\Sigma\right|^{1/2}}|| $\backslash tfrac\{1\}\{2\}\; \backslash log\backslash left[(2\backslash pi\; e)^N\; \backslash det(\backslash Sigma)\backslash right]$||$\backslash mathbb\{R\}^N$

|}

Many of the differential entropies are from.{{cite journal|author=Lazo, A. and P. Rathie|title=On the entropy of continuous probability distributions|journal=IEEE Transactions on Information Theory|year=1978|volume=24 |issue=1|doi=10.1109/TIT.1978.1055832|pages=120–122}}{{rp|pp=120–122}}

As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.{{cite journal |author=Jaynes, E.T. |author-link=Edwin Thompson Jaynes |title=Information Theory And Statistical Mechanics |journal=Brandeis University Summer Institute Lectures in Theoretical Physics |volume=3 |issue=sect. 4b |year=1963 |url=http://bayes.wustl.edu/etj/articles/brandeis.pdf }}{{rp|pp=181–218}}

A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If $m(x)$ is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

$$D(p\backslash parallel\; m)\; =\; \backslash int\; p(x)\backslash log\backslash frac\{p(x)\}\{m(x)\}\backslash ,dx.$$

The definition of differential entropy above can be obtained by partitioning the range of $X$ into bins of length $h$ with associated sample points $ih$ within the bins, for $X$ Riemann integrable. This gives a quantized version of $X$, defined by $X\_h\; =\; ih$ if $ih\; \backslash le\; X\; \backslash le\; (i+1)h$. Then the entropy of $X\_h\; =\; ih$ is

$$H\_h=-\backslash sum\_i\; hf(ih)\backslash log\; (f(ih))\; -\; \backslash sum\; hf(ih)\backslash log(h).$$

The first term on the right approximates the differential entropy, while the second term is approximately $-\backslash log(h)$. Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be $\backslash infty$.

• Information entropy
• Self-information
• Entropy estimation

{{reflist}}

• {{springer|title=Differential entropy|id=p/d031890}}
• {{planetmath reference|urlname=DifferentialEntropy|title=Differential entropy}}

Category:Entropy and information

Category:Information theory

Category:Statistical randomness