inverse-variance weighting

Suppose an experimenter wishes to measure the value of a quantity, say the acceleration due to gravity of Earth, whose true value happens to be $\backslash mu$. A careful experimenter makes multiple measurements, which we denote with $n$ random variables $X\_1,\; X\_2\; ,\; ...\; ,\; X\_n$. If they are all noisy but unbiased, i.e., the measuring device does not systematically overestimate or underestimate the true value and the errors are scattered symmetrically, then the expectation value $E[X\_i]\; =\; \backslash mu$ $\backslash forall\; i$. The scatter in the measurement is then characterised by the variance of the random variables $Var(X\_i)\; :=\; \backslash sigma\_i^2$, and if the measurements are performed under identical scenarios, then all the $\backslash sigma\_i$ are the same, which we shall refer to by $\backslash sigma$. Given the $n$ measurements, a typical estimator for $\backslash mu$, denoted as $\backslash hat\{\backslash mu\}$, is given by the simple average $\backslash overline\{X\}\; =\; \backslash frac\{1\}\{n\}\; \backslash sum\_i\; X\_i$. Note that this empirical average is also a random variable, whose expectation value $E[\backslash overline\{X\}]$ is $\backslash mu$ but also has a scatter. If the individual measurements are uncorrelated, the square of the error in the estimate is given by

Var(\overline{X}) = \frac{1}{n^2}\sum_i \sigma_i^2 = \left(\frac{\sigma}{\sqrt{n}}\right)^2

. Hence, if all the $\backslash sigma\_i$ are equal, then the error in the estimate decreases with increase in $n$ as $1/\backslash sqrt\{n\}$, thus making more observations preferred.

Instead of $n$ repeated measurements with one instrument, if the experimenter makes $n$ of the same quantity with $n$ different instruments with varying quality of measurements, then there is no reason to expect the different $\backslash sigma\_i$ to be the same. Some instruments could be noisier than others. In the example of measuring the acceleration due to gravity, the different "instruments" could be measuring $g$ from a simple pendulum, from analysing a projectile motion etc. The simple average is no longer an optimal estimator, since the error in $\backslash overline\{X\}$ might actually exceed the error in the least noisy measurement if different measurements have very different errors. Instead of discarding the noisy measurements that increase the final error, the experimenter can combine all the measurements with appropriate weights so as to give more importance to the least noisy measurements and vice versa. Given the knowledge of $\backslash sigma\_1^2,\; \backslash sigma\_2^2,\; ...\; ,\; \backslash sigma\_n^2$, an optimal estimator to measure $\backslash mu$ would be a weighted mean of the measurements $\backslash hat\{\backslash mu\}\; =\; \backslash frac\{\backslash sum\_i\; w\_i\; X\_i\}\{\backslash sum\_i\; w\_i\}$, for the particular choice of the weights $w\_i\; =\; 1/\backslash sigma\_i^2$. The variance of the estimator $Var(\backslash hat\{\backslash mu\})\; =\; \backslash frac\{\; \backslash sum\_i\; w\_i^2\; \backslash sigma\_i^2\; \}\{\; \backslash left(\; \backslash sum\_i\; w\_i\; \backslash right)^2\; \}$, which for the optimal choice of the weights become $$

Var(\hat{\mu}_\text{opt}) = \left( \sum_{i} \sigma_i^{-2} \right)^{-1} .

Note that since , the estimator has a scatter smaller than the scatter in any individual measurement. Furthermore, the scatter in $\backslash hat\{\backslash mu\}\_\backslash text\{opt\}$ decreases with adding more measurements, however noisier those measurements may be.

Consider a generic weighted sum $Y=\; \backslash sum\_i\; w\_i\; X\_i$, where the weights $w\_i$ are normalised such that $\backslash sum\_i\; w\_i\; =\; 1$.

If the $X\_i$ are all independent, the variance of $Y$ is given by (see Bienaymé's identity)

:$$

Var(Y) = \sum_i w_i^2 \sigma_i^2.

For optimality, we wish to minimise $Var(Y)$ which can be done by equating the gradient with respect to the weights of $Var(Y)$ to zero, while maintaining the constraint that $\backslash sum\_i\; w\_i\; =\; 1$. Using a Lagrange multiplier $w\_0$ to enforce the constraint, we express the variance:

:$$

Var(Y) = \sum_i w_i^2 \sigma_i^2 - w_0(\sum_i w_i - 1).

For $k>0$,

:$$

0 = \frac{\partial}{\partial w_k} Var(Y) = 2w_k\sigma_k^2 - w_0,

which implies that:

:$w\_k\; =\; \backslash frac\{w\_0/2\}\{\backslash sigma\_k^2\}.$

The main takeaway here is that $w\_k\; \backslash propto\; 1/\backslash sigma\_k^2$. Since $\backslash sum\_i\; w\_i\; =\; 1$,

:$$

\frac{2}{w_0} = \sum_i \frac{1}{\sigma_i^2} := \frac{1}{\sigma_0^2}.

The individual normalised weights are:

:$$

w_k = \frac{1}{\sigma_k^2}\left( \sum_i \frac{1}{\sigma_i^2} \right)^{-1}.

It is easy to see that this extremum solution corresponds to the minimum from the second partial derivative test by noting that the variance is a quadratic function of the weights. Thus, the minimum variance of the estimator is then given by:

:$$

Var(Y) = \sum_i \frac{\sigma_0^4}{\sigma_i^4}\sigma_i^2 = \sigma_0^4\sum_i \frac{1}{\sigma_i^2} = \sigma_0^4\frac{1}{\sigma_0^2} = \sigma_0^2 = \frac{1}{\sum_i 1/\sigma_i^2}.

{{Missing|correlated measurements should involve the complete covariance matrix|date=October 2024}}

For normally distributed random variables inverse-variance weighted averages can also be derived as the maximum likelihood estimate for the true value. Furthermore, from a Bayesian perspective the posterior distribution for the true value given normally distributed observations $y\_i$ and a flat prior is a normal distribution with the inverse-variance weighted average as a mean and variance $Var(Y)$.

{{See also|generalized variance}}

For potentially correlated multivariate distributions an equivalent argument leads to an optimal weighting based on the covariance matrices $\backslash mathbf\{C\}\_i$ of the individual vector-valued estimates $\backslash mathbf\{x\}\_i$:

:$\backslash mathbf\{\backslash hat\{x\}\}\; =\; \backslash left(\backslash sum\_i\; \backslash mathbf\{C\}\_i^\{-1\}\backslash right)^\{-1\}\backslash sum\_i\; \backslash mathbf\{C\}\_i^\{-1\}\; \backslash mathbf\{x\}\_i$

:$\backslash mathbf\{\backslash hat\{C\}\}\; =\; \backslash left(\backslash sum\_i\; \backslash mathbf\{C\}\_i^\{-1\}\backslash right)^\{-1\}$

For multivariate distributions the term "precision-weighted" average is more commonly used.

The Kalman filter gain minimizes the determinant of the posterior covariance (i.e. the generalized variance) of the estimated mean Proof that the Kalman gain minimizes the generalized variance,

Eviatar Bach https://arxiv.org/abs/2103.07275. This step of the Kalman filter, therefore, employs inverse-variance weighting, where more precise (i.e., lower variance) sources of information are weighted more heavily in the update step of the Kalman filter.

• Weighted least squares
• Portfolio theory
• Cramér-Rao bound

{{Reflist}}

{{More citations needed|date=September 2012}}

Category:Meta-analysis

Category:Estimation methods