Sufficient statistic

Roughly, given a set $\backslash mathbf\{X\}$ of independent identically distributed data conditioned on an unknown parameter $\backslash theta$, a sufficient statistic is a function $T(\backslash mathbf\{X\})$ whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate). Due to the factorization theorem (see below), for a sufficient statistic $T(\backslash mathbf\{X\})$, the probability density can be written as $f\_\{\backslash mathbf\{X\}\}(x;\backslash theta)\; =\; h(x)\; \backslash ,\; g(\backslash theta,\; T(x))$. From this factorization, it can easily be seen that the maximum likelihood estimate of $\backslash theta$ will interact with $\backslash mathbf\{X\}$ only through $T(\backslash mathbf\{X\})$. Typically, the sufficient statistic is a simple function of the data, e.g. the sum of all the data points.

More generally, the "unknown parameter" may represent a vector of unknown quantities or may represent everything about the model that is unknown or not fully specified. In such a case, the sufficient statistic may be a set of functions, called a jointly sufficient statistic. Typically, there are as many functions as there are parameters. For example, for a Gaussian distribution with unknown mean and variance, the jointly sufficient statistic, from which maximum likelihood estimates of both parameters can be estimated, consists of two functions, the sum of all data points and the sum of all squared data points (or equivalently, the sample mean and sample variance).

In other words, the joint probability distribution of the data is conditionally independent of the parameter given the value of the sufficient statistic for the parameter. Both the statistic and the underlying parameter can be vectors.

A statistic t = T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ.{{cite book | last = Casella | first = George |author2=Berger, Roger L. | title = Statistical Inference, 2nd ed | publisher=Duxbury Press | year = 2002}}

Alternatively, one can say the statistic T(X) is sufficient for θ if, for all prior distributions on θ, the mutual information between θ and T(X) equals the mutual information between θ and X.{{Cite book|last=Cover|first=Thomas M.|title=Elements of Information Theory|date=2006|publisher=Wiley-Interscience|others=Joy A. Thomas|isbn=0-471-24195-4|edition=2nd|location=Hoboken, N.J.|pages=36|oclc=59879802}} In other words, the data processing inequality becomes an equality:

:$I\backslash bigl(\backslash theta\; ;\; T(X)\backslash bigr)\; =\; I(\backslash theta\; ;\; X)$

As an example, the sample mean is sufficient for the (unknown) mean μ of a normal distribution with known variance. Once the sample mean is known, no further information about μ can be obtained from the sample itself. On the other hand, for an arbitrary distribution the median is not sufficient for the mean: even if the median of the sample is known, knowing the sample itself would provide further information about the population mean. For example, if the observations that are less than the median are only slightly less, but observations exceeding the median exceed it by a large amount, then this would have a bearing on one's inference about the population mean.

Fisher's factorization theorem or factorization criterion provides a convenient characterization of a sufficient statistic. If the probability density function is ƒ_θ(x), then T is sufficient for θ if and only if nonnegative functions g and h can be found such that

:$f(x;\backslash theta)=h(x)\; \backslash ,\; g(\backslash theta,T(x)),$

i.e., the density ƒ can be factored into a product such that one factor, h, does not depend on θ and the other factor, which does depend on θ, depends on x only through T(x). A general proof of this was given by Halmos and Savage{{Cite journal |last1=Halmos |first1=P. R. |last2=Savage |first2=L. J. |date=1949 |title=Application of the Radon-Nikodym Theorem to the Theory of Sufficient Statistics |url=http://projecteuclid.org/euclid.aoms/1177730032 |journal=The Annals of Mathematical Statistics |language=en |volume=20 |issue=2 |pages=225–241 |doi=10.1214/aoms/1177730032 |issn=0003-4851|doi-access=free |url-access=subscription }} and the theorem is sometimes referred to as the Halmos–Savage factorization theorem.{{Cite web |title=Factorization theorem - Encyclopedia of Mathematics |url=https://encyclopediaofmath.org/wiki/Factorization_theorem |access-date=2022-09-07 |website=encyclopediaofmath.org}} The proofs below handle special cases, but an alternative general proof along the same lines can be given.{{Cite journal |last=Taraldsen |first=G. |date=2022 |title=The Factorization Theorem for Sufficiency |url= |journal=Preprint |language=en |doi=10.13140/RG.2.2.15068.87687}} In many simple cases the probability density function is fully specified by $\backslash theta$ and $T(x)$, and $h(x)=1$ (see Examples).

It is easy to see that if F(t) is a one-to-one function and T is a sufficient

statistic, then F(T) is a sufficient statistic. In particular we can multiply a

sufficient statistic by a nonzero constant and get another sufficient statistic.

An implication of the theorem is that when using likelihood-based inference, two sets of data yielding the same value for the sufficient statistic T(X) will always yield the same inferences about θ. By the factorization criterion, the likelihood's dependence on θ is only in conjunction with T(X). As this is the same in both cases, the dependence on θ will be the same as well, leading to identical inferences.

Due to Hogg and Craig.{{cite book | last = Hogg | first = Robert V. |author2=Craig, Allen T. | title = Introduction to Mathematical Statistics | publisher=Prentice Hall | year = 1995 | isbn=978-0-02-355722-4}} Let $X\_1,\; X\_2,\; \backslash ldots,\; X\_n$, denote a random sample from a distribution having the pdf f(x, θ) for ι < θ < δ. Let Y₁ = u₁(X₁, X₂, ..., X_n) be a statistic whose pdf is g₁(y₁; θ). What we want to prove is that Y₁ = u₁(X₁, X₂, ..., X_n) is a sufficient statistic for θ if and only if, for some function H,

:$\backslash prod\_\{i=1\}^n\; f(x\_i;\; \backslash theta)\; =\; g\_1\; \backslash left[u\_1\; (x\_1,\; x\_2,\; \backslash dots,\; x\_n);\; \backslash theta\; \backslash right]\; H(x\_1,\; x\_2,\; \backslash dots,\; x\_n).$

First, suppose that

:$\backslash prod\_\{i=1\}^n\; f(x\_i;\; \backslash theta)\; =\; g\_1\; \backslash left[u\_1\; (x\_1,\; x\_2,\; \backslash dots,\; x\_n);\; \backslash theta\; \backslash right]\; H(x\_1,\; x\_2,\; \backslash dots,\; x\_n).$

We shall make the transformation y_i = u_i(x₁, x₂, ..., x_n), for i = 1, ..., n, having inverse functions x_i = w_i(y₁, y₂, ..., y_n), for i = 1, ..., n, and Jacobian $J\; =\; \backslash left[w\_i/y\_j\; \backslash right]$. Thus,

:$$

\prod_{i=1}^n f \left[ w_i(y_1, y_2, \dots, y_n); \theta \right] =

|J| g_1 (y_1; \theta) H \left[ w_1(y_1, y_2, \dots, y_n), \dots, w_n(y_1, y_2, \dots, y_n) \right].

The left-hand member is the joint pdf g(y₁, y₂, ..., y_n; θ) of Y₁ = u₁(X₁, ..., X_n), ..., Y_n = u_n(X₁, ..., X_n). In the right-hand member, $g\_1(y\_1;\backslash theta)$ is the pdf of $Y\_1$, so that $H[\; w\_1,\; \backslash dots\; ,\; w\_n]\; |J|$ is the quotient of $g(y\_1,\backslash dots,y\_n;\backslash theta)$ and $g\_1(y\_1;\backslash theta)$; that is, it is the conditional pdf $h(y\_2,\; \backslash dots,\; y\_n\; \backslash mid\; y\_1;\; \backslash theta)$ of $Y\_2,\backslash dots,Y\_n$ given $Y\_1=y\_1$.

But $H(x\_1,x\_2,\backslash dots,x\_n)$, and thus $H\backslash left[w\_1(y\_1,\backslash dots,y\_n),\; \backslash dots,\; w\_n(y\_1,\; \backslash dots,\; y\_n))\backslash right]$, was given not to depend upon $\backslash theta$. Since $\backslash theta$ was not introduced in the transformation and accordingly not in the Jacobian $J$, it follows that $h(y\_2,\; \backslash dots,\; y\_n\; \backslash mid\; y\_1;\; \backslash theta)$ does not depend upon $\backslash theta$ and that $Y\_1$ is a sufficient statistics for $\backslash theta$.

The converse is proven by taking:

:$g(y\_1,\backslash dots,y\_n;\backslash theta)=g\_1(y\_1;\; \backslash theta)\; h(y\_2,\; \backslash dots,\; y\_n\; \backslash mid\; y\_1),$

where $h(y\_2,\; \backslash dots,\; y\_n\; \backslash mid\; y\_1)$ does not depend upon $\backslash theta$ because $Y\_2\; ...\; Y\_n$ depend only upon $X\_1\; ...\; X\_n$, which are independent on $\backslash Theta$ when conditioned by $Y\_1$, a sufficient statistics by hypothesis. Now divide both members by the absolute value of the non-vanishing Jacobian $J$, and replace $y\_1,\; \backslash dots,\; y\_n$ by the functions $u\_1(x\_1,\; \backslash dots,\; x\_n),\; \backslash dots,\; u\_n(x\_1,\backslash dots,\; x\_n)$ in $x\_1,\backslash dots,\; x\_n$. This yields

:$\backslash frac\{g\backslash left[\; u\_1(x\_1,\; \backslash dots,\; x\_n),\; \backslash dots,\; u\_n(x\_1,\; \backslash dots,\; x\_n);\; \backslash theta\; \backslash right]\}$

where $J^*$ is the Jacobian with $y\_1,\backslash dots,y\_n$ replaced by their value in terms $x\_1,\; \backslash dots,\; x\_n$. The left-hand member is necessarily the joint pdf $f(x\_1;\backslash theta)\backslash cdots\; f(x\_n;\backslash theta)$ of $X\_1,\backslash dots,X\_n$. Since $h(y\_2,\backslash dots,y\_n\backslash mid\; y\_1)$, and thus $h(u\_2,\backslash dots,u\_n\backslash mid\; u\_1)$, does not depend upon $\backslash theta$, then

:$H(x\_1,\backslash dots,x\_n)=\backslash frac\{h(u\_2,\backslash dots,u\_n\backslash mid\; u\_1)\}$

is a function that does not depend upon $\backslash theta$.

A simpler more illustrative proof is as follows, although it applies only in the discrete case.

We use the shorthand notation to denote the joint probability density of $(X,\; T(X))$ by $f\_\backslash theta(x,t)$. Since $T$ is a deterministic function of $X$, we have $f\_\backslash theta(x,t)\; =\; f\_\backslash theta(x)$, as long as $t\; =\; T(x)$ and zero otherwise. Therefore:

:$$

\begin{align}

f_\theta(x) & = f_\theta(x,t) \\[5pt]

& = f_\theta (x\mid t) f_\theta(t) \\[5pt]

& = f(x\mid t) f_\theta(t)

\end{align}

with the last equality being true by the definition of sufficient statistics. Thus $f\_\backslash theta(x)=a(x)\; b\_\backslash theta(t)$ with $a(x)\; =\; f\_\{X\; \backslash mid\; t\}(x)$ and $b\_\backslash theta(t)\; =\; f\_\backslash theta(t)$.

Conversely, if $f\_\backslash theta(x)=a(x)\; b\_\backslash theta(t)$, we have

:$$

\begin{align}

f_\theta(t) & = \sum _{x : T(x) = t} f_\theta(x, t) \\[5pt]

& = \sum _{x : T(x) = t} f_\theta(x) \\[5pt]

& = \sum _{x : T(x) = t} a(x) b_\theta(t) \\[5pt]

& = \left( \sum _{x : T(x) = t} a(x) \right) b_\theta(t).

\end{align}

With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over $t$.

Let $f\_\{X\backslash mid\; t\}(x)$ denote the conditional probability density of $X$ given $T(X)$. Then we can derive an explicit expression for this:

:$$

\begin{align}

f_{X\mid t}(x)

& = \frac{f_\theta(x, t)}{f_\theta(t)} \\[5pt]

& = \frac{f_\theta(x)}{f_\theta(t)} \\[5pt]

& = \frac{a(x) b_\theta(t)}{\left( \sum _{x : T(x) = t} a(x) \right) b_\theta(t)} \\[5pt]

& = \frac{a(x)}{\sum _{x : T(x) = t} a(x)}.

\end{align}

With the first equality by definition of conditional probability density, the second by the remark above, the third by the equality proven above, and the fourth by simplification. This expression does not depend on $\backslash theta$ and thus $T$ is a sufficient statistic.{{cite web | url=http://cnx.org/content/m11480/1.6/ | title=The Fisher–Neyman Factorization Theorem}}. Webpage at Connexions (cnx.org)

A sufficient statistic is minimal sufficient if it can be represented as a function of any other sufficient statistic. In other words, S(X) is minimal sufficient if and only ifDodge (2003) — entry for minimal sufficient statistics

1. S(X) is sufficient, and
2. if T(X) is sufficient, then there exists a function f such that S(X) = f(T(X)).

Intuitively, a minimal sufficient statistic most efficiently captures all possible information about the parameter θ.

A useful characterization of minimal sufficiency is that when the density f_θ exists, S(X) is minimal sufficient if and only if{{Citation needed|reason=The classical result (e.g., Theorem 6.2.13 in Casella and Berger) only gives a sufficient condition, not a characterization|date=September 2023}}

:$\backslash frac\{f\_\backslash theta(x)\}\{f\_\backslash theta(y)\}$ is independent of θ :$\backslash Longleftrightarrow$ S(x) = S(y)

This follows as a consequence from Fisher's factorization theorem stated above.

A case in which there is no minimal sufficient statistic was shown by Bahadur, 1954.Lehmann and Casella (1998), Theory of Point Estimation, 2nd Edition, Springer, p 37 However, under mild conditions, a minimal sufficient statistic does always exist. In particular, in Euclidean space, these conditions always hold if the random variables (associated with $P\_\backslash theta$ ) are all discrete or are all continuous.

If there exists a minimal sufficient statistic, and this is usually the case, then every complete sufficient statistic is necessarily minimal sufficientLehmann and Casella (1998), Theory of Point Estimation, 2nd Edition, Springer, page 42 (note that this statement does not exclude a pathological case in which a complete sufficient exists while there is no minimal sufficient statistic). While it is hard to find cases in which a minimal sufficient statistic does not exist, it is not so hard to find cases in which there is no complete statistic.

The collection of likelihood ratios $\backslash left\backslash \{\backslash frac\{L(X\; \backslash mid\; \backslash theta\_i)\}\{L(X\; \backslash mid\; \backslash theta\_0)\}\backslash right\backslash \}$ for $i\; =\; 1,\; ...,\; k$, is a minimal sufficient statistic if the parameter space is discrete $\backslash left\backslash \{\backslash theta\_0,\; ...,\; \backslash theta\_k\backslash right\backslash \}$.

If X₁, ...., X_n are independent Bernoulli-distributed random variables with expected value p, then the sum T(X) = X₁ + ... + X_n is a sufficient statistic for p (here 'success' corresponds to X_i = 1 and 'failure' to X_i = 0; so T is the total number of successes)

This is seen by considering the joint probability distribution:

:$\backslash Pr\backslash \{X=x\backslash \}=\backslash Pr\backslash \{X\_1=x\_1,X\_2=x\_2,\backslash ldots,X\_n=x\_n\backslash \}.$

Because the observations are independent, this can be written as

:$$

p^{x_1}(1-p)^{1-x_1} p^{x_2}(1-p)^{1-x_2}\cdots p^{x_n}(1-p)^{1-x_n}

and, collecting powers of p and 1 − p, gives

:$$

p^{\sum x_i}(1-p)^{n-\sum x_i}=p^{T(x)}(1-p)^{n-T(x)}

which satisfies the factorization criterion, with h(x) = 1 being just a constant.

Note the crucial feature: the unknown parameter p interacts with the data x only via the statistic T(x) = Σ x_i.

As a concrete application, this gives a procedure for distinguishing a fair coin from a biased coin.

{{see also|German tank problem}}

If X₁, ...., X_n are independent and uniformly distributed on the interval [0,θ], then T(X) = max(X₁, ..., X_n) is sufficient for θ — the sample maximum is a sufficient statistic for the population maximum.

To see this, consider the joint probability density function of X  (X₁,...,X_n). Because the observations are independent, the pdf can be written as a product of individual densities

:$\backslash begin\{align\}$

f_{\theta}(x_1,\ldots,x_n)

&= \frac{1}{\theta}\mathbf{1}_{\{0\leq x_1\leq\theta\}} \cdots

\frac{1}{\theta}\mathbf{1}_{\{0\leq x_n\leq\theta\}} \\[5pt]

&= \frac{1}{\theta^n} \mathbf{1}_{\{0\leq\min\{x_i\}\}}\mathbf{1}_{\{\max\{x_i\}\leq\theta\}}

\end{align}

where 1_{...} is the indicator function. Thus the density takes form required by the Fisher–Neyman factorization theorem, where h(x) = 1_{{min{xi}≥0}}, and the rest of the expression is a function of only θ and T(x) = max{x_i}.

In fact, the minimum-variance unbiased estimator (MVUE) for θ is

:$\backslash frac\{n+1\}\{n\}T(X).$

This is the sample maximum, scaled to correct for the bias, and is MVUE by the Lehmann–Scheffé theorem. Unscaled sample maximum T(X) is the maximum likelihood estimator for θ.

If $X\_1,...,X\_n$ are independent and uniformly distributed on the interval $[\backslash alpha,\; \backslash beta]$ (where $\backslash alpha$ and $\backslash beta$ are unknown parameters), then $T(X\_1^n)=\backslash left(\backslash min\_\{1\; \backslash leq\; i\; \backslash leq\; n\}X\_i,\backslash max\_\{1\; \backslash leq\; i\; \backslash leq\; n\}X\_i\backslash right)$ is a two-dimensional sufficient statistic for $(\backslash alpha\backslash ,\; ,\; \backslash ,\; \backslash beta)$.

To see this, consider the joint probability density function of $X\_1^n=(X\_1,\backslash ldots,X\_n)$. Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

:$\backslash begin\{align\}$

f_{X_1^n}(x_1^n)

&= \prod_{i=1}^n \left({1 \over \beta-\alpha}\right) \mathbf{1}_{ \{ \alpha \leq x_i \leq \beta \} }

= \left({1 \over \beta-\alpha}\right)^n \mathbf{1}_{ \{ \alpha \leq x_i \leq \beta, \, \forall \, i = 1,\ldots,n\}} \\

&= \left({1 \over \beta-\alpha}\right)^n \mathbf{1}_{ \{ \alpha \, \leq \, \min_{1 \leq i \leq n}X_i \} } \mathbf{1}_{ \{ \max_{1 \leq i \leq n}X_i \, \leq \, \beta \} }.

\end{align}

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

:$\backslash begin\{align\}$

h(x_1^n)= 1, \quad

g_{(\alpha, \beta)}(x_1^n)= \left({1 \over \beta-\alpha}\right)^n \mathbf{1}_{ \{ \alpha \, \leq \, \min_{1 \leq i \leq n}X_i \} } \mathbf{1}_{ \{ \max_{1 \leq i \leq n}X_i \, \leq \, \beta \} }.

\end{align}

Since $h(x\_1^n)$ does not depend on the parameter $(\backslash alpha,\; \backslash beta)$ and $g\_\{(\backslash alpha\; \backslash ,\; ,\; \backslash ,\; \backslash beta)\}(x\_1^n)$ depends only on $x\_1^n$ through the function $T(X\_1^n)=\; \backslash left(\backslash min\_\{1\; \backslash leq\; i\; \backslash leq\; n\}X\_i,\backslash max\_\{1\; \backslash leq\; i\; \backslash leq\; n\}X\_i\backslash right),$

the Fisher–Neyman factorization theorem implies $T(X\_1^n)\; =\; \backslash left(\backslash min\_\{1\; \backslash leq\; i\; \backslash leq\; n\}X\_i,\backslash max\_\{1\; \backslash leq\; i\; \backslash leq\; n\}X\_i\backslash right)$ is a sufficient statistic for $(\backslash alpha\backslash ,\; ,\; \backslash ,\; \backslash beta)$.

If X₁, ...., X_n are independent and have a Poisson distribution with parameter λ, then the sum T(X) = X₁ + ... + X_n is a sufficient statistic for λ.

To see this, consider the joint probability distribution:

:$$

\Pr(X=x)=P(X_1=x_1,X_2=x_2,\ldots,X_n=x_n).

Because the observations are independent, this can be written as

:$$

{e^{-\lambda} \lambda^{x_1} \over x_1 !} \cdot

{e^{-\lambda} \lambda^{x_2} \over x_2 !} \cdots

{e^{-\lambda} \lambda^{x_n} \over x_n !}

which may be written as

:$$

e^{-n\lambda} \lambda^{(x_1+x_2+\cdots+x_n)} \cdot

{1 \over x_1 ! x_2 !\cdots x_n ! }

which shows that the factorization criterion is satisfied, where h(x) is the reciprocal of the product of the factorials. Note the parameter λ interacts with the data only through its sum T(X).

If $X\_1,\backslash ldots,X\_n$ are independent and normally distributed with expected value $\backslash theta$ (a parameter) and known finite variance $\backslash sigma^2,$ then

:$T(X\_1^n)=\backslash overline\{x\}=\backslash frac1n\backslash sum\_\{i=1\}^nX\_i$

is a sufficient statistic for $\backslash theta.$

To see this, consider the joint probability density function of $X\_1^n=(X\_1,\backslash dots,X\_n)$. Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

:$\backslash begin\{align\}$

f_{X_1^n}(x_1^n)

& = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left (-\frac{(x_i-\theta)^2}{2\sigma^2} \right ) \\ [6pt]

&= (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left ( -\sum_{i=1}^n \frac{(x_i-\theta)^2}{2\sigma^2} \right ) \\ [6pt]

& = (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left (-\sum_{i=1}^n \frac{ \left ( \left (x_i-\overline{x} \right ) - \left (\theta-\overline{x} \right ) \right )^2}{2\sigma^2} \right ) \\ [6pt]

& = (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left( -{1\over2\sigma^2} \left(\sum_{i=1}^n(x_i-\overline{x})^2 + \sum_{i=1}^n(\theta-\overline{x})^2 -2\sum_{i=1}^n(x_i-\overline{x})(\theta-\overline{x})\right) \right) \\ [6pt]

&= (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left( -{1\over2\sigma^2} \left (\sum_{i=1}^n(x_i-\overline{x})^2 + n(\theta-\overline{x})^2 \right ) \right ) && \sum_{i=1}^n(x_i-\overline{x})(\theta-\overline{x})=0 \\ [6pt]

&= (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left( -{1\over2\sigma^2} \sum_{i=1}^n (x_i-\overline{x})^2 \right ) \exp \left (-\frac{n}{2\sigma^2} (\theta-\overline{x})^2 \right )

\end{align}

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

:$\backslash begin\{align\}$

h(x_1^n) &= (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left( -{1\over2\sigma^2} \sum_{i=1}^n (x_i-\overline{x})^2 \right ) \\[6pt]

g_\theta(x_1^n) &= \exp \left (-\frac{n}{2\sigma^2} (\theta-\overline{x})^2 \right )

\end{align}

Since $h(x\_1^n)$ does not depend on the parameter $\backslash theta$ and $g\_\{\backslash theta\}(x\_1^n)$ depends only on $x\_1^n$ through the function

:$T(X\_1^n)=\backslash overline\{x\}=\backslash frac1n\backslash sum\_\{i=1\}^nX\_i,$

the Fisher–Neyman factorization theorem implies $T(X\_1^n)$ is a sufficient statistic for $\backslash theta$.

If $\backslash sigma^2$ is unknown and since $s^2\; =\; \backslash frac\{1\}\{n-1\}\; \backslash sum\_\{i=1\}^n\; \backslash left(x\_i\; -\; \backslash overline\{x\}\; \backslash right)^2$, the above likelihood can be rewritten as

:$\backslash begin\{align\}$

f_{X_1^n}(x_1^n)= (2\pi\sigma^2)^{-n/2} \exp \left( -\frac{n-1}{2\sigma^2}s^2 \right) \exp \left (-\frac{n}{2\sigma^2} (\theta-\overline{x})^2 \right ) .

\end{align}

The Fisher–Neyman factorization theorem still holds and implies that $(\backslash overline\{x\},s^2)$ is a joint sufficient statistic for $(\; \backslash theta\; ,\; \backslash sigma^2)$.

If $X\_1,\backslash dots,X\_n$ are independent and exponentially distributed with expected value θ (an unknown real-valued positive parameter), then $T(X\_1^n)=\backslash sum\_\{i=1\}^nX\_i$ is a sufficient statistic for θ.

To see this, consider the joint probability density function of $X\_1^n=(X\_1,\backslash dots,X\_n)$. Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

:$\backslash begin\{align\}$

f_{X_1^n}(x_1^n)

&= \prod_{i=1}^n {1 \over \theta} \, e^{ {-1 \over \theta}x_i }

= {1 \over \theta^n}\, e^{ {-1 \over \theta} \sum_{i=1}^nx_i }.

\end{align}

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

:$\backslash begin\{align\}$

h(x_1^n)= 1,\,\,\,

g_{\theta}(x_1^n)= {1 \over \theta^n}\, e^{ {-1 \over \theta} \sum_{i=1}^nx_i }.

\end{align}

Since $h(x\_1^n)$ does not depend on the parameter $\backslash theta$ and $g\_\{\backslash theta\}(x\_1^n)$ depends only on $x\_1^n$ through the function $T(X\_1^n)=\backslash sum\_\{i=1\}^nX\_i$

the Fisher–Neyman factorization theorem implies $T(X\_1^n)=\backslash sum\_\{i=1\}^nX\_i$ is a sufficient statistic for $\backslash theta$.

If $X\_1,\backslash dots,X\_n$ are independent and distributed as a $\backslash Gamma(\backslash alpha\; \backslash ,\; ,\; \backslash ,\; \backslash beta)$, where $\backslash alpha$ and $\backslash beta$ are unknown parameters of a Gamma distribution, then $T(X\_1^n)\; =\; \backslash left(\; \backslash prod\_\{i=1\}^n\{X\_i\}\; ,\; \backslash sum\_\{i=1\}^n\; X\_i\; \backslash right)$ is a two-dimensional sufficient statistic for $(\backslash alpha,\; \backslash beta)$.

To see this, consider the joint probability density function of $X\_1^n=(X\_1,\backslash dots,X\_n)$. Because the observations are independent, the pdf can be written as a product of individual densities, i.e.

:$\backslash begin\{align\}$

f_{X_1^n}(x_1^n)

&= \prod_{i=1}^n \left({1 \over \Gamma(\alpha) \beta^\alpha}\right) x_i^{\alpha -1} e^{(-1/\beta)x_i} \\[5pt]

&= \left({1 \over \Gamma(\alpha) \beta^\alpha}\right)^n \left(\prod_{i=1}^n x_i\right)^{\alpha-1} e^{{-1 \over \beta} \sum_{i=1}^n x_i}.

\end{align}

The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting

:$\backslash begin\{align\}$

h(x_1^n)= 1,\,\,\,

g_{(\alpha \, , \, \beta)}(x_1^n)= \left({1 \over \Gamma(\alpha) \beta^{\alpha}}\right)^n \left(\prod_{i=1}^n x_i\right)^{\alpha-1} e^{{-1 \over \beta} \sum_{i=1}^n x_i}.

\end{align}

Since $h(x\_1^n)$ does not depend on the parameter $(\backslash alpha\backslash ,\; ,\; \backslash ,\; \backslash beta)$ and $g\_\{(\backslash alpha\; \backslash ,\; ,\; \backslash ,\; \backslash beta)\}(x\_1^n)$ depends only on $x\_1^n$ through the function $T(x\_1^n)=\; \backslash left(\; \backslash prod\_\{i=1\}^n\; x\_i,\; \backslash sum\_\{i=1\}^n\; x\_i\; \backslash right),$

the Fisher–Neyman factorization theorem implies $T(X\_1^n)=\; \backslash left(\; \backslash prod\_\{i=1\}^n\; X\_i,\; \backslash sum\_\{i=1\}^n\; X\_i\; \backslash right)$ is a sufficient statistic for $(\backslash alpha\backslash ,\; ,\; \backslash ,\; \backslash beta).$

Sufficiency finds a useful application in the Rao–Blackwell theorem, which states that if g(X) is any kind of estimator of θ, then typically the conditional expectation of g(X) given sufficient statistic T(X) is a better (in the sense of having lower variance) estimator of θ, and is never worse. Sometimes one can very easily construct a very crude estimator g(X), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.

{{main|Exponential family}}

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases. Intuitively, this states that nonexponential families of distributions on the real line require nonparametric statistics to fully capture the information in the data.

Less tersely, suppose $X\_n,\; n\; =\; 1,\; 2,\; 3,\; \backslash dots$ are independent identically distributed real random variables whose distribution is known to be in some family of probability distributions, parametrized by $\backslash theta$, satisfying certain technical regularity conditions, then that family is an exponential family if and only if there is a $\backslash R^m$-valued sufficient statistic $T(X\_1,\; \backslash dots,\; X\_n)$ whose number of scalar components $m$ does not increase as the sample size n increases.{{Cite journal |last1=Tikochinsky |first1=Y. |last2=Tishby |first2=N. Z. |last3=Levine |first3=R. D. |date=1984-11-01 |title=Alternative approach to maximum-entropy inference |url=http://dx.doi.org/10.1103/physreva.30.2638 |journal=Physical Review A |volume=30 |issue=5 |pages=2638–2644 |doi=10.1103/physreva.30.2638 |bibcode=1984PhRvA..30.2638T |issn=0556-2791|url-access=subscription }}

This theorem shows that the existence of a finite-dimensional, real-vector-valued sufficient statistics sharply restricts the possible forms of a family of distributions on the real line.

When the parameters or the random variables are no longer real-valued, the situation is more complex.{{Cite journal |last=Andersen |first=Erling Bernhard |date=September 1970 |title=Sufficiency and Exponential Families for Discrete Sample Spaces |url=http://dx.doi.org/10.1080/01621459.1970.10481160 |journal=Journal of the American Statistical Association |volume=65 |issue=331 |pages=1248–1255 |doi=10.1080/01621459.1970.10481160 |issn=0162-1459|url-access=subscription }}

An alternative formulation of the condition that a statistic be sufficient, set in a Bayesian context, involves the posterior distributions obtained by using the full data-set and by using only a statistic. Thus the requirement is that, for almost every x,

:$\backslash Pr(\backslash theta\backslash mid\; X=x)\; =\; \backslash Pr(\backslash theta\backslash mid\; T(X)=t(x)).$

More generally, without assuming a parametric model, we can say that the statistics T is predictive sufficient if

:$\backslash Pr(X\text{'}=x\text{'}\backslash mid\; X=x)\; =\; \backslash Pr(X\text{'}=x\text{'}\backslash mid\; T(X)=t(x)).$

It turns out that this "Bayesian sufficiency" is a consequence of the formulation above,{{cite book

|last1=Bernardo |first1=J.M. |author-link1=José-Miguel Bernardo

|last2=Smith |first2=A.F.M. |author-link2=Adrian Smith (academic)

|year=1994

|title=Bayesian Theory

|publisher=Wiley

|isbn=0-471-92416-4

|chapter=Section 5.1.4

}} however they are not directly equivalent in the infinite-dimensional case.{{cite journal

|last1=Blackwell |first1=D. |author-link1=David Blackwell

|last2=Ramamoorthi |first2=R. V.

|title=A Bayes but not classically sufficient statistic.

|journal=Annals of Statistics

|volume=10 |year=1982 |issue=3 |pages=1025–1026

|doi=10.1214/aos/1176345895 |mr=663456 | zbl = 0485.62004

|doi-access=free }} A range of theoretical results for sufficiency in a Bayesian context is available.{{cite journal

|last1=Nogales |first1=A.G.

|last2=Oyola |first2=J.A.

|last3=Perez |first3=P.

|year=2000

|title=On conditional independence and the relationship between sufficiency and invariance under the Bayesian point of view

|journal=Statistics & Probability Letters

|volume=46 |issue=1 |pages=75–84

|doi=10.1016/S0167-7152(99)00089-9 |mr=1731351 | zbl = 0964.62003

|url=https://dialnet.unirioja.es/servlet/oaiart?codigo=118597

|url-access=subscription}}

A concept called "linear sufficiency" can be formulated in a Bayesian context,{{cite journal |first1=M. |last1=Goldstein |first2=A. |last2=O'Hagan |year=1996 |title=Bayes Linear Sufficiency and Systems of Expert Posterior Assessments |journal=Journal of the Royal Statistical Society |series=Series B |volume=58 |issue=2 |pages=301–316 |doi=10.1111/j.2517-6161.1996.tb02083.x |jstor=2345978 }} and more generally.{{cite journal |last=Godambe |first=V. P. |year=1966 |title=A New Approach to Sampling from Finite Populations. II Distribution-Free Sufficiency |journal=Journal of the Royal Statistical Society |series=Series B |volume=28 |issue=2 |pages=320–328 |doi=10.1111/j.2517-6161.1966.tb00645.x |jstor=2984375 }} First define the best linear predictor of a vector Y based on X as $\backslash hat\; E[Y\backslash mid\; X]$. Then a linear statistic T(x) is linear sufficient{{cite journal |last=Witting |first=T. |year=1987 |title=The linear Markov property in credibility theory |journal=ASTIN Bulletin |volume=17 |issue=1 |pages=71–84 |doi= 10.2143/ast.17.1.2014984|doi-access=free |hdl=20.500.11850/422507 |hdl-access=free }} if

:$\backslash hat\; E[\backslash theta\backslash mid\; X]=\; \backslash hat\; E[\backslash theta\backslash mid\; T(X)]\; .$

• Completeness of a statistic
• Basu's theorem on independence of complete sufficient and ancillary statistics
• Lehmann–Scheffé theorem: a complete sufficient estimator is the best estimator of its expectation
• Rao–Blackwell theorem
• Chentsov's theorem
• Sufficient dimension reduction
• Ancillary statistic

{{reflist|30em}}

• {{Springer|title=Sufficient statistic|id=S/s091070|first=A.S.|last=Kholevo}}
• {{cite book

| last = Lehmann

| first = E. L.

|author2=Casella, G.

| title = Theory of Point Estimation

| year = 1998

| publisher = Springer

| isbn = 0-387-98502-6

| edition = 2nd

| pages = Chapter 4

| no-pp = y }}

• Dodge, Y. (2003) The Oxford Dictionary of Statistical Terms, OUP. {{isbn|0-19-920613-9}}

{{Statistics|inference|collapsed}}

{{DEFAULTSORT:Sufficient Statistic}}

Category:Statistical theory

Category:Statistical principles

Category:Articles containing proofs

Category:factorization