Cochran's theorem

If X₁, ..., X_n are independent normally distributed random variables with mean μ and standard deviation σ then

:$U\_i\; =\; \backslash frac\{X\_i-\backslash mu\}\{\backslash sigma\}$

is standard normal for each i. Note that the total Q is equal to sum of squared Us as shown here:

:$\backslash sum\_iQ\_i=\backslash sum\_\{jik\}\; U\_j\; B\_\{jk\}^\{(i)\}\; U\_k\; =\; \backslash sum\_\{jk\}\; U\_j\; U\_k\; \backslash sum\_i\; B\_\{jk\}^\{(i)\}\; =$

\sum_{jk} U_j U_k\delta_{jk} = \sum_{j} U_j^2

which stems from the original assumption that $B\_\{1\}\; +\; B\_\{2\}\; \backslash ldots\; =\; I$.

So instead we will calculate this quantity and later separate it into Q_i's. It is possible to write

:$$

\sum_{i=1}^n U_i^2=\sum_{i=1}^n\left(\frac{X_i-\overline{X}}{\sigma}\right)^2

+ n\left(\frac{\overline{X}-\mu}{\sigma}\right)^2

(here $\backslash overline\{X\}$ is the sample mean). To see this identity, multiply throughout by $\backslash sigma^2$ and note that

:$$

\sum(X_i-\mu)^2=

\sum(X_i-\overline{X}+\overline{X}-\mu)^2

and expand to give

:$$

\sum(X_i-\mu)^2=

\sum(X_i-\overline{X})^2+\sum(\overline{X}-\mu)^2+

2\sum(X_i-\overline{X})(\overline{X}-\mu).

The third term is zero because it is equal to a constant times

:$\backslash sum(\backslash overline\{X\}-X\_i)=0,$

and the second term has just n identical terms added together. Thus

:$$

\sum(X_i-\mu)^2 = \sum(X_i-\overline{X})^2+n(\overline{X}-\mu)^2 ,

and hence

:$$

\sum\left(\tfrac{X_i-\mu}{\sigma}\right)^2=

\sum\left(\tfrac{X_i-\overline{X}}{\sigma}\right)^2

+n\left(\tfrac{\overline{X}-\mu}{\sigma}\right)^2=

\overbrace{\sum_i\left(U_i-\tfrac{1}{n}\sum_j{U_j}\right)^2}^{Q_1}

+\overbrace{\tfrac{1}{n}\left(\sum_j{U_j}\right)^2}^{Q_2}=

Q_1+Q_2.

Now $B^\{(2)\}=\backslash frac\{J\_n\}\{n\}$ with $J\_n$ the matrix of ones which has rank 1. In turn $B^\{(1)\}=\; I\_n-\backslash frac\{J\_n\}\{n\}$ given that $I\_n=B^\{(1)\}+B^\{(2)\}$. This expression can be also obtained by expanding $Q\_1$ in matrix notation. It can be shown that the rank of $B^\{(1)\}$ is $n-1$ as the addition of all its rows is equal to zero. Thus the conditions for Cochran's theorem are met.

Cochran's theorem then states that Q₁ and Q₂ are independent, with chi-squared distributions with n − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property characterizes the normal distribution – for no other distribution are the sample mean and sample variance independent.{{cite journal

|doi=10.2307/2983669

|first=R.C. |last=Geary |author-link=Roy C. Geary

|year=1936

|title=The Distribution of "Student's" Ratio for Non-Normal Samples

|journal=Supplement to the Journal of the Royal Statistical Society

|volume=3 |issue=2 |pages=178–184

|jfm=63.1090.03

|jstor=2983669

}}

The result for the distributions is written symbolically as

:$$

\sum\left(X_i-\overline{X}\right)^2 \sim \sigma^2 \chi^2_{n-1}.

:$$

n(\overline{X}-\mu)^2\sim \sigma^2 \chi^2_1,

Both these random variables are proportional to the true but unknown variance σ². Thus their ratio does not depend on σ² and, because they are statistically independent. The distribution of their ratio is given by

:$$

\frac{n\left(\overline{X}-\mu\right)^2}

{\frac{1}{n-1}\sum\left(X_i-\overline{X}\right)^2}\sim \frac{\chi^2_1}{\frac{1}{n-1}\chi^2_{n-1}}

\sim F_{1,n-1}

where F_1,n − 1 is the F-distribution with 1 and n − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the definition of a random variable having the F-distribution.

To estimate the variance σ², one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution

:$$

\widehat{\sigma}^2=

\frac{1}{n}\sum\left(

X_i-\overline{X}\right)^2.

Cochran's theorem shows that

:$$

\frac{n\widehat{\sigma}^2}{\sigma^2}\sim\chi^2_{n-1}

and the properties of the chi-squared distribution show that

:$\backslash begin\{align\}$

E \left(\frac{n \widehat{\sigma}^2}{\sigma^2}\right) &= E \left(\chi^2_{n-1}\right) \\

\frac{n}{\sigma^2}E \left(\widehat{\sigma}^2\right) &= (n-1) \\

E \left(\widehat{\sigma}^2\right) &= \frac{\sigma^2 (n-1)}{n}

\end{align}

The following version is often seen when considering linear regression.{{Cite web|url=https://yangfengstat.github.io/assets/teaching/cochran's-theorem.pdf|title=Cochran's Theorem (A quick tutorial)}} Suppose that $Y\backslash sim\; N\_n(0,\backslash sigma^2I\_n)$ is a standard multivariate normal random vector (here $I\_n$ denotes the n-by-n identity matrix), and if $A\_1,\backslash ldots,A\_k$ are all n-by-n symmetric matrices with $\backslash sum\_\{i=1\}^kA\_i=I\_n$. Then, on defining $r\_i=\; \backslash operatorname\{Rank\}(A\_i)$, any one of the following conditions implies the other two:

• $\backslash sum\_\{i=1\}^kr\_i=n\; ,$
• $Y^TA\_iY\backslash sim\backslash sigma^2\backslash chi^2\_\{r\_i\}$ (thus the $A\_i$ are positive semidefinite)
• $Y^TA\_iY$ is independent of $Y^TA\_jY$ for $i\backslash neq\; j\; .$

Let U₁, ..., U_N be i.i.d. standard normally distributed random variables, and $U\; =\; [U\_1,\; ...,\; U\_N]^T$. Let $B^\{(1)\},B^\{(2)\},\backslash ldots,\; B^\{(k)\}$be symmetric matrices. Define r_i to be the rank of $B^\{(i)\}$. Define $Q\_i=U^T\; B^\{(i)\}U$, so that the Q_i are quadratic forms. Further assume $\backslash sum\_i\; Q\_i\; =\; U^T\; U$.

Cochran's theorem states that the following are equivalent:

• $r\_1+\backslash cdots\; +r\_k=N$,
• the Q_i are independent
• each Q_i has a chi-squared distribution with r_i degrees of freedom.{{Citation |title=Cochran's theorem |date=2008-01-01 |url=https://www.oxfordreference.com/view/10.1093/acref/9780199541454.001.0001/acref-9780199541454-e-294 |work=A Dictionary of Statistics |publisher=Oxford University Press |language=en |doi=10.1093/acref/9780199541454.001.0001 |isbn=978-0-19-954145-4 |access-date=2022-05-18 |last1=Upton |first1=Graham |last2=Cook |first2=Ian |url-access=subscription }}

Often it's stated as $\backslash sum\_i\; A\_i\; =\; A$, where $A$ is idempotent, and $\backslash sum\_i\; r\_i\; =\; N$ is replaced by $\backslash sum\_i\; r\_i\; =\; rank(A)$. But after an orthogonal transform, $A\; =\; diag(I\_M,\; 0)$, and so we reduce to the above theorem.

Claim: Let $X$ be a standard Gaussian in $\backslash R^n$, then for any symmetric matrices $Q,\; Q\text{'}$, if $X^T\; Q\; X$ and $X^T\; Q\text{'}\; X$ have the same distribution, then $Q,\; Q\text{'}$ have the same eigenvalues (up to multiplicity).

{{Math proof|title=Proof|proof=

Let the eigenvalues of $Q$ be $\backslash lambda\_1,\; ...,\; \backslash lambda\_n$, then calculate the characteristic function of $X^T\; Q\; X$. It comes out to be

$\backslash phi(t)\; =\backslash left(\backslash prod\_j\; (1-2i\; \backslash lambda\_j\; t)\backslash right)^\{-1/2\}$

(To calculate it, first diagonalize $Q$, change into that frame, then use the fact that the characteristic function of the sum of independent variables is the product of their characteristic functions.)

For $X^T\; Q\; X$ and $X^T\; Q\text{'}\; X$ to be equal, their characteristic functions must be equal, so $Q,\; Q\text{'}$ have the same eigenvalues (up to multiplicity).

}}

Claim: $I\; =\; \backslash sum\_i\; B\_i$.

{{Math proof|title=Proof|proof=

$U^T\; (I\; -\; \backslash sum\_i\; B\_i)\; U\; =\; 0$. Since $(I\; -\; \backslash sum\_i\; B\_i)$ is symmetric, and $U^T\; (I\; -\; \backslash sum\_i\; B\_i)\; U\; =^d\; U^T\; 0\; U$, by the previous claim, $(I\; -\; \backslash sum\_i\; B\_i)$ has the same eigenvalues as 0.

}}

Lemma: If $\backslash sum\_i\; M\_i\; =\; I$, all $M\_i$ symmetric, and have eigenvalues 0, 1, then they are simultaneously diagonalizable.

{{Math proof|title=Proof|proof=

Fix i, and consider the eigenvectors v of $M\_i$

such that $M\_i\; v\; =\; v$. Then we have $v^T\; v\; =\; v^T\; I\; v\; =\; v^T\; v\; +\; \backslash sum\_\{j\backslash neq\; i\}\; v^T\; M\_j\; v$, so all $v^T\; M\_j\; v\; =\; 0$. Thus we obtain a split of $\backslash R^N$ into $V\backslash oplus\; V^\backslash perp$, such that V is the 1-eigenspace of $M\_i$

, and in the 0-eigenspaces of all other $M\_j$

. Now induct by moving into $V^\backslash perp$.

}}

Now we prove the original theorem. We prove that the three cases are equivalent by proving that each case implies the next one in a cycle ($1\; \backslash to\; 2\; \backslash to\; 3\; \backslash to\; 1$).

{{Math proof|title=Proof|proof=

Case: All $Q\_i$ are independent

Fix some $i$, define $C\_i\; =\; I\; -\; B\_i\; =\; \backslash sum\_\{j\backslash neq\; i\}\; B\_j$, and diagonalize $B\_i$ by an orthogonal transform $O$. Then consider $O\; C\_i\; O^T\; =\; I\; -\; O\; B\_i\; O^T$. It is diagonalized as well.

Let $W\; =\; OU$, then it is also standard Gaussian. Then we have

$Q\_i\; =\; W^T\; (OB\_i\; O^T)\; W;\; \backslash quad\; \backslash sum\_\{j\backslash neq\; i\}\; Q\_j\; =\; W^T\; (I\; -\; OB\_i\; O^T)\; W$

Inspect their diagonal entries, to see that $Q\_i\; \backslash perp\; \backslash sum\_\{j\backslash neq\; i\}\; Q\_j$ implies that their nonzero diagonal entries are disjoint.

Thus all eigenvalues of $B\_i$ are 0, 1, so $Q\_i$ is a $\backslash chi^2$ dist with $r\_i$ degrees of freedom.

Case: Each $Q\_i$ is a $\backslash chi^2(r\_i)$ distribution.

Fix any $i$, diagonalize it by orthogonal transform $O$, and reindex, so that $O\; B\_i\; O^T\; =\; diag(\backslash lambda\_1,\; ...,\; \backslash lambda\_\{r\_i\},\; 0,\; ...,\; 0)$. Then $Q\_i\; =\; \backslash sum\_j\; \backslash lambda\_j\; \{U\text{'}\}\_j^2$ for some $U\text{'}\_j$, a spherical rotation of $U\_i$.

Since $Q\_i\backslash sim\; \backslash chi^2(r\_i)$, we get all $\backslash lambda\_j\; =\; 1$. So all $B\_i\backslash succeq\; 0$, and have eigenvalues $0,\; 1$.

So diagonalize them simultaneously, add them up, to find $\backslash sum\_i\; r\_i\; =\; N$.

Case: $r\_1+\backslash cdots\; +r\_k=N$.

We first show that the matrices B⁽ⁱ⁾ can be simultaneously diagonalized by an orthogonal matrix and that their non-zero eigenvalues are all equal to +1. Once that's shown, take this orthogonal transform to this simultaneous eigenbasis, in which the random vector $[U\_1,\; ...,\; U\_N]^T$ becomes $[U\text{'}\_1,\; ...,\; U\text{'}\_N]^T$, but all $U\_i\text{'}$ are still independent and standard Gaussian. Then the result follows.

Each of the matrices B⁽ⁱ⁾ has rank r_i and thus r_i non-zero eigenvalues. For each i, the sum $C^\{(i)\}\; \backslash equiv\; \backslash sum\_\{j\backslash ne\; i\}B^\{(j)\}$ has at most rank $\backslash sum\_\{j\backslash ne\; i\}r\_j\; =\; N-r\_i$. Since $B^\{(i)\}+C^\{(i)\}\; =\; I\_\{N\; \backslash times\; N\}$, it follows that C⁽ⁱ⁾ has exactly rank N − r_i.

Therefore B⁽ⁱ⁾ and C⁽ⁱ⁾ can be simultaneously diagonalized. This can be shown by first diagonalizing B⁽ⁱ⁾, by the spectral theorem. In this basis, it is of the form:

:$\backslash begin\{bmatrix\}$

\lambda_1 & 0 & 0 & \cdots & \cdots & & 0 \\

0 & \lambda_2 & 0 & \cdots & \cdots & & 0 \\

0 & 0 & \ddots & & & & \vdots \\

\vdots & \vdots & & \lambda_{r_i} & & \\

\vdots & \vdots & & & 0 & \\

0 & \vdots & & & & \ddots \\

0 & 0 & \ldots & & & & 0

\end{bmatrix}.

Thus the lower $(N-r\_i)$ rows are zero. Since $C^\{(i)\}\; =\; I\; -\; B^\{(i)\}$, it follows that these rows in C⁽ⁱ⁾ in this basis contain a right block which is a $(N-r\_i)\backslash times(N-r\_i)$ unit matrix, with zeros in the rest of these rows. But since C⁽ⁱ⁾ has rank N − r_i, it must be zero elsewhere. Thus it is diagonal in this basis as well. It follows that all the non-zero eigenvalues of both B⁽ⁱ⁾ and C⁽ⁱ⁾ are +1. This argument applies for all i, thus all B⁽ⁱ⁾ are positive semidefinite.

Moreover, the above analysis can be repeated in the diagonal basis for $C^\{(1)\}\; =\; B^\{(2)\}\; +\; \backslash sum\_\{j>2\}B^\{(j)\}$. In this basis $C^\{(1)\}$ is the identity of an $(N-r\_1)\backslash times(N-r\_1)$ vector space, so it follows that both B⁽²⁾ and $\backslash sum\_\{j>2\}B^\{(j)\}$ are simultaneously diagonalizable in this vector space (and hence also together with B⁽¹⁾). By iteration it follows that all B-s are simultaneously diagonalizable.

Thus there exists an orthogonal matrix $S$ such that for all $i$, $S^\backslash mathrm\{T\}B^\{(i)\}\; S\; \backslash equiv\; B^\{(i)\backslash prime\}$ is diagonal, where any entry $B^\{(i)\backslash prime\}\_\{x,y\}$ with indices $x\; =\; y$, , is equal to 1, while any entry with other indices is equal to 0.

}}

• Cramér's theorem, on decomposing normal distribution
• Infinite divisibility (probability)

{{refimprove|date=July 2011}}

{{Experimental design|state=expanded}}

{{DEFAULTSORT:Cochran's Theorem}}

Category:Theorems in statistics

Category:Characterization of probability distributions