Entropy power inequality

For a random vector $X\; :\; \backslash Omega\; \backslash to\; \backslash mathbb\{R\}^n$ with probability density function $f\; :\; \backslash mathbb\{R\}^n\; \backslash to\; \backslash mathbb\{R\}$, the differential entropy of $X$, denoted $h(X)$, is defined to be

:$h(X)\; =\; -\; \backslash int\_\{\backslash mathbb\{R\}^n\}\; f(x)\; \backslash log\; f(x)\; \backslash ,\; dx$

and the entropy power of $X$, denoted $N(X)$, is defined to be

:$N(X)\; =\; \backslash frac\{1\}\{2\backslash pi\; e\}\; e^\{\; \backslash frac\{2\}\{n\}\; h(X)\; \}.$

In particular, $N(X)\; =\; |K|^\{1/n\}$ when $X$ is normally distributed with covariance matrix $K$.

Let $X$ and $Y$ be independent random variables with probability density functions in the $L^p$ space $L^p(\backslash mathbb\{R\}^n)$ for some $p\; >\; 1$. Then

:$N(X\; +\; Y)\; \backslash geq\; N(X)\; +\; N(Y).$

Moreover, equality holds if and only if $X$ and $Y$ are multivariate normal random variables with proportional covariance matrices.

The entropy power inequality can be rewritten in an equivalent form that does not explicitly depend on the definition of entropy power (see Costa and Cover reference below).

Let $X$ and $Y$ be independent random variables, as above. Then, let $X\text{'}$ and $Y\text{'}$ be independent random variables with Gaussian distributions such that

:$h(X\text{'})\; =\; h(X)$ and $h(Y\text{'})\; =\; h(Y)$

Then,

:$h(X\; +\; Y)\; \backslash geq\; h(X\text{'}\; +\; Y\text{'})$

• Information entropy
• Information theory
• Limiting density of discrete points
• Self-information
• Kullback–Leibler divergence
• Entropy estimation

• {{cite journal

| last = Dembo

| first = Amir |author2=Cover, Thomas M. |author3=Thomas, Joy A.

| s2cid = 845669 | title = Information-theoretic inequalities

| journal = IEEE Trans. Inf. Theory

| volume = 37

| year = 1991

| issue = 6

| pages = 1501–1518

| doi = 10.1109/18.104312

| mr = 1134291

}}

• {{cite journal

| last = Costa

| first = Max H. M.

| author2=Cover, Thomas M.

| title = On the similarity of the entropy-power inequality and the Brunn-Minkowski inequality

| journal = IEEE Trans. Inf. Theory

| volume = 30

| issue = 6

| year = 1984

| pages = 837–839

| doi = 10.1109/TIT.1984.1056983

}}

• {{cite journal

| last=Gardner

| first=Richard J.

| title=The Brunn–Minkowski inequality

| journal=Bull. Amer. Math. Soc. (N.S.)

| volume=39

| issue=3

| year=2002

| pages=355–405 (electronic)

| doi=10.1090/S0273-0979-02-00941-2

| doi-access=free

}}

• {{cite journal

| last = Shannon

| first = Claude E.

| authorlink = Claude Shannon

| title = A mathematical theory of communication

| journal = Bell System Tech. J.

| volume = 27

| issue = 3

| year = 1948

| pages = 379–423, 623–656

| doi = 10.1002/j.1538-7305.1948.tb01338.x

| hdl = 10338.dmlcz/101429

| hdl-access = free

}}

• {{cite journal

| last = Stam

| first = A. J.

| title = Some inequalities satisfied by the quantities of information of Fisher and Shannon

| journal = Information and Control

| volume = 2

| year = 1959

| pages = 101–112

| doi = 10.1016/S0019-9958(59)90348-1

| issue = 2

| doi-access = free

}}

Category:Information theory

Category:Probabilistic inequalities

Category:Statistical inequalities