Bernoulli distribution

If $X$ is a random variable with a Bernoulli distribution, then:

:$\backslash Pr(X=1)\; =\; p,\; \backslash Pr(X=0)\; =\; q\; =1\; -\; p.$

The probability mass function $f$ of this distribution, over possible outcomes k, is

:$f(k;p)\; =\; \backslash begin\{cases\}$

p & \text{if }k=1, \\

q = 1-p & \text {if } k = 0.

\end{cases}{{Cite book|title=Introduction to Probability|last=Bertsekas|author-link=Dimitri Bertsekas|first=Dimitri P.|date=2002|publisher=Athena Scientific|others=Tsitsiklis, John N., Τσιτσικλής, Γιάννης Ν.|isbn=188652940X|location=Belmont, Mass.|oclc=51441829}}

This can also be expressed as

:$f(k;p)\; =\; p^k\; (1-p)^\{1-k\}\; \backslash quad\; \backslash text\{for\; \}\; k\backslash in\backslash \{0,1\backslash \}$

or as

:$f(k;p)=pk+(1-p)(1-k)\; \backslash quad\; \backslash text\{for\; \}\; k\backslash in\backslash \{0,1\backslash \}.$

The Bernoulli distribution is a special case of the binomial distribution with $n\; =\; 1.${{cite book | last = McCullagh | first = Peter | author-link= Peter McCullagh |author2=Nelder, John |author-link2=John Nelder | title = Generalized Linear Models, Second Edition | publisher = Boca Raton: Chapman and Hall/CRC | year = 1989 | isbn = 0-412-31760-5 |ref=McCullagh1989 |at=Section 4.2.2 }}

The kurtosis goes to infinity for high and low values of $p,$ but for $p=1/2$ the two-point distributions including the Bernoulli distribution have a lower excess kurtosis, namely −2, than any other probability distribution.

The Bernoulli distributions for $0\; \backslash le\; p\; \backslash le\; 1$ form an exponential family.

The maximum likelihood estimator of $p$ based on a random sample is the sample mean.

File:PMF and CDF of a bernouli distribution.png

The expected value of a Bernoulli random variable $X$ is

:$\backslash operatorname\{E\}[X]=p$

This is because for a Bernoulli distributed random variable $X$ with $\backslash Pr(X=1)=p$ and $\backslash Pr(X=0)=q$ we find

:$\backslash operatorname\{E\}[X]\; =\; \backslash Pr(X=1)\backslash cdot\; 1\; +\; \backslash Pr(X=0)\backslash cdot\; 0$

= p \cdot 1 + q\cdot 0 = p.

The variance of a Bernoulli distributed $X$ is

:$\backslash operatorname\{Var\}[X]\; =\; pq\; =\; p(1-p)$

We first find

:$\backslash operatorname\{E\}[X^2]\; =\; \backslash Pr(X=1)\backslash cdot\; 1^2\; +\; \backslash Pr(X=0)\backslash cdot\; 0^2$

:$=\; p\; \backslash cdot\; 1^2\; +\; q\backslash cdot\; 0^2\; =\; p\; =\; \backslash operatorname\{E\}[X]$

From this follows

:$\backslash operatorname\{Var\}[X]\; =\; \backslash operatorname\{E\}[X^2]-\backslash operatorname\{E\}[X]^2\; =\; \backslash operatorname\{E\}[X]-\backslash operatorname\{E\}[X]^2$

:$=\; p-p^2\; =\; p(1-p)\; =\; pq$

With this result it is easy to prove that, for any Bernoulli distribution, its variance will have a value inside $[0,1/4]$.

The skewness is $\backslash frac\{q-p\}\{\backslash sqrt\{pq\}\}=\backslash frac\{1-2p\}\{\backslash sqrt\{pq\}\}$. When we take the standardized Bernoulli distributed random variable $\backslash frac\{X-\backslash operatorname\{E\}[X]\}\{\backslash sqrt\{\backslash operatorname\{Var\}[X]\}\}$ we find that this random variable attains $\backslash frac\{q\}\{\backslash sqrt\{pq\}\}$ with probability $p$ and attains $-\backslash frac\{p\}\{\backslash sqrt\{pq\}\}$ with probability $q$. Thus we get

:$\backslash begin\{align\}$

\gamma_1 &= \operatorname{E} \left[\left(\frac{X-\operatorname{E}[X]}{\sqrt{\operatorname{Var}[X]}}\right)^3\right] \\

&= p \cdot \left(\frac{q}{\sqrt{pq}}\right)^3 + q \cdot \left(-\frac{p}{\sqrt{pq}}\right)^3 \\

&= \frac{1}{\sqrt{pq}^3} \left(pq^3-qp^3\right) \\

&= \frac{pq}{\sqrt{pq}^3} (q^2-p^2) \\

&= \frac{(1-p)^2-p^2}{\sqrt{pq}} \\

&= \frac{1-2p}{\sqrt{pq}} = \frac{q-p}{\sqrt{pq}}.

\end{align}

The raw moments are all equal because $1^k=1$ and $0^k=0$.

:$\backslash operatorname\{E\}[X^k]\; =\; \backslash Pr(X=1)\backslash cdot\; 1^k\; +\; \backslash Pr(X=0)\backslash cdot\; 0^k\; =\; p\; \backslash cdot\; 1\; +\; q\backslash cdot\; 0\; =\; p\; =\; \backslash operatorname\{E\}[X].$

The central moment of order $k$ is given by

:$$

\mu_k =(1-p)(-p)^k +p(1-p)^k.

The first six central moments are

:$\backslash begin\{align\}$

\mu_1 &= 0, \\

\mu_2 &= p(1-p), \\

\mu_3 &= p(1-p)(1-2p), \\

\mu_4 &= p(1-p)(1-3p(1-p)), \\

\mu_5 &= p(1-p)(1-2p)(1-2p(1-p)), \\

\mu_6 &= p(1-p)(1-5p(1-p)(1-p(1-p))).

\end{align}

The higher central moments can be expressed more compactly in terms of $\backslash mu\_2$ and $\backslash mu\_3$

:$\backslash begin\{align\}$

\mu_4 &= \mu_2 (1-3\mu_2 ), \\

\mu_5 &= \mu_3 (1-2\mu_2 ), \\

\mu_6 &= \mu_2 (1-5\mu_2 (1-\mu_2 )).

\end{align}

The first six cumulants are

:$\backslash begin\{align\}$

\kappa_1 &= p, \\

\kappa_2 &= \mu_2 , \\

\kappa_3 &= \mu_3 , \\

\kappa_4 &= \mu_2 (1-6\mu_2 ), \\

\kappa_5 &= \mu_3 (1-12\mu_2 ), \\

\kappa_6 &= \mu_2 (1-30\mu_2 (1-4\mu_2 )).

\end{align}

Entropy is a measure of uncertainty or randomness in a probability distribution. For a Bernoulli random variable $X$ with success probability $p$ and failure probability $q\; =\; 1\; -\; p$, the entropy $H(X)$ is defined as:

:$\backslash begin\{align\}$

H(X) &= \mathbb{E}_p \ln (\frac{1}{P(X)}) = - [P(X = 0) \ln P(X = 0) + P(X = 1) \ln P(X = 1)] \\

H(X) &= - (q \ln q + p \ln p) , \quad q = P(X = 0), p = P(X = 1)

\end{align}

The entropy is maximized when $p\; =\; 0.5$, indicating the highest level of uncertainty when both outcomes are equally likely. The entropy is zero when $p\; =\; 0$ or $p\; =\; 1$, where one outcome is certain.

Fisher information measures the amount of information that an observable random variable $X$ carries about an unknown parameter $p$ upon which the probability of $X$ depends. For the Bernoulli distribution, the Fisher information with respect to the parameter $p$ is given by:

:$\backslash begin\{align\}$

I(p) = \frac{1}{pq}

\end{align}

Proof:

• The Likelihood Function for a Bernoulli random variable$X$ is:

:$\backslash begin\{align\}$

L(p; X) = p^X (1 - p)^{1 - X}

\end{align}

This represents the probability of observing $X$ given the parameter $p$.

• The Log-Likelihood Function is:

:$\backslash begin\{align\}$

\ln L(p; X) = X \ln p + (1 - X) \ln (1 - p)

\end{align}

• The Score Function (the first derivative of the log-likelihood w.r.t. $p$ is:

:$\backslash begin\{align\}$

\frac{\partial}{\partial p} \ln L(p; X) = \frac{X}{p} - \frac{1 - X}{1 - p}

\end{align}

• The second derivative of the log-likelihood function is:

:$\backslash begin\{align\}$

\frac{\partial^2}{\partial p^2} \ln L(p; X) = -\frac{X}{p^2} - \frac{1 - X}{(1 - p)^2}

\end{align}

• Fisher information is calculated as the negative expected value of the second derivative of the log-likelihood:

:$\backslash begin\{align\}$

I(p) = -E\left[\frac{\partial^2}{\partial p^2} \ln L(p; X)\right] = -\left(-\frac{p}{p^2} - \frac{1 - p}{(1 - p)^2}\right) = \frac{1}{p(1-p)} = \frac{1}{pq}

\end{align}

It is maximized when $p\; =\; 0.5$, reflecting maximum uncertainty and thus maximum information about the parameter $p$.

• If $X\_1,\backslash dots,X\_n$ are independent, identically distributed (i.i.d.) random variables, all Bernoulli trials with success probability p, then their sum is distributed according to a binomial distribution with parameters n and p:
• :$\backslash sum\_\{k=1\}^n\; X\_k\; \backslash sim\; \backslash operatorname\{B\}(n,p)$ (binomial distribution).

:The Bernoulli distribution is simply $\backslash operatorname\{B\}(1,\; p)$, also written as $\backslash mathrm\{Bernoulli\}\; (p).$

• The categorical distribution is the generalization of the Bernoulli distribution for variables with any constant number of discrete values.
• The Beta distribution is the conjugate prior of the Bernoulli distribution.{{Cite web |last1=Orloff |first1=Jeremy |last2=Bloom |first2=Jonathan |date= |title=Conjugate priors: Beta and normal |url=https://math.mit.edu/~dav/05.dir/class15-prep.pdf |access-date=October 20, 2023 |website=math.mit.edu}}
• The geometric distribution models the number of independent and identical Bernoulli trials needed to get one success.
• If $Y\; \backslash sim\; \backslash mathrm\{Bernoulli\}\backslash left(\backslash frac\{1\}\{2\}\backslash right)$, then $2Y\; -\; 1$ has a Rademacher distribution.

• Bernoulli process, a random process consisting of a sequence of independent Bernoulli trials
• Bernoulli sampling
• Binary entropy function
• Binary decision diagram

{{Reflist}}

• {{cite book |last1=Johnson |first1=N. L. |last2=Kotz |first2=S. |last3=Kemp |first3=A. |year=1993 |title=Univariate Discrete Distributions |edition=2nd |publisher=Wiley |isbn=0-471-54897-9 }}
• {{cite book |first=John G. |last=Peatman |title=Introduction to Applied Statistics |location=New York |publisher=Harper & Row |year=1963 |pages=162–171 }}

{{Commons category|Bernoulli distribution}}

• {{springer|title=Binomial distribution|id=p/b016420}}.
• {{MathWorld|title=Bernoulli Distribution|urlname=BernoulliDistribution}}
• Interactive graphic: [http://www.math.wm.edu/~leemis/chart/UDR/UDR.html Univariate Distribution Relationships].

{{ProbDistributions|discrete-finite}}

{{DEFAULTSORT:Bernoulli Distribution}}

Category:Discrete distributions

Category:Conjugate prior distributions

Category:Exponential family distributions