delta method

The delta method was derived from propagation of error, and the idea behind was known in the early 20th century.{{cite journal

|last=Portnoy

|first=Stephen

|year=2013

|title=Letter to the Editor

|journal=The American Statistician

|volume=67 |number=3 |pages=190 |doi=10.1080/00031305.2013.820668|s2cid=219596186

}} Its statistical application can be traced as far back as 1928 by T. L. Kelley.{{cite book

|last=Kelley

|first=Truman L.

|year=1928

|title=Crossroads in the Mind of Man: A Study of Differentiable Mental Abilities

|pages=49–50 |isbn=978-1-4338-0048-1}} A formal description of the method was presented by J. L. Doob in 1935.{{cite journal

|jstor=2957546

|last=Doob

|first=J. L.

|year=1935

|title=The Limiting Distributions of Certain Statistics

|journal=Annals of Mathematical Statistics

|volume=6 |issue=3

|pages=160–169

|doi=10.1214/aoms/1177732594|doi-access=free}} Robert Dorfman also described a version of it in 1938.{{cite journal |jstor=23339471 |last=Ver Hoef |first=J. M. |year=2012 |title=Who invented the delta method? |journal=The American Statistician |volume=66 |issue=2 |pages=124–127 |doi=10.1080/00031305.2012.687494}}

While the delta method generalizes easily to a multivariate setting, careful motivation of the technique is more easily demonstrated in univariate terms. Roughly, if there is a sequence of random variables {{mvar|X_n}} satisfying

:$\{\backslash sqrt\{n\}[X\_n-\backslash theta]\backslash ,\backslash xrightarrow\{D\}\backslash ,\backslash mathcal\{N\}(0,\backslash sigma^2)\},$

where θ and σ² are finite valued constants and $\backslash xrightarrow\{D\}$ denotes convergence in distribution, then

:$\{\backslash sqrt\{n\}[g(X\_n)-g(\backslash theta)]\backslash ,\backslash xrightarrow\{D\}\backslash ,\backslash mathcal\{N\}(0,\backslash sigma^2\backslash cdot[g\text{'}(\backslash theta)]^2)\}$

for any function g satisfying the property that its first derivative, evaluated at $\backslash theta$, $g\text{'}(\backslash theta)$ exists and is non-zero valued.

The intuition of the delta method is that any such g function, in a "small enough" range of the function, can be approximated via a first order Taylor series (which is basically a linear function). If the random variable is roughly normal then a linear transformation of it is also normal. Small range can be achieved when approximating the function around the mean, when the variance is "small enough". When g is applied to a random variable such as the mean, the delta method would tend to work better as the sample size increases, since it would help reduce the variance, and thus the Taylor approximation would be applied to a smaller range of the function g at the point of interest.

Demonstration of this result is fairly straightforward under the assumption that $g(x)$ is differentiable near the neighborhood of $\backslash theta$ and $g\text{'}(x)$ is continuous at $\backslash theta$ with $g\text{'}(\backslash theta)\backslash neq\; 0$. To begin, we use the mean value theorem (i.e.: the first order approximation of a Taylor series using Taylor's theorem):

:$g(X\_n)=g(\backslash theta)+g\text{'}(\backslash tilde\{\backslash theta\})(X\_n-\backslash theta),$

where $\backslash tilde\{\backslash theta\}$ lies between {{mvar|X_n}} and θ.

Note that since $X\_n\backslash ,\backslash xrightarrow\{P\}\backslash ,\backslash theta$ and , it must be that $\backslash tilde\{\backslash theta\}\; \backslash ,\backslash xrightarrow\{P\}\backslash ,\backslash theta$ and since {{math|g′(θ)}} is continuous, applying the continuous mapping theorem yields

:$g\text{'}(\backslash tilde\{\backslash theta\})\backslash ,\backslash xrightarrow\{P\}\backslash ,g\text{'}(\backslash theta),$

where $\backslash xrightarrow\{P\}$ denotes convergence in probability.

Rearranging the terms and multiplying by $\backslash sqrt\{n\}$ gives

:$\backslash sqrt\{n\}[g(X\_n)-g(\backslash theta)]=g\text{'}\; \backslash left\; (\backslash tilde\{\backslash theta\}\; \backslash right\; )\backslash sqrt\{n\}[X\_n-\backslash theta].$

Since

:$\{\backslash sqrt\{n\}[X\_n-\backslash theta]\; \backslash xrightarrow\{D\}\; \backslash mathcal\{N\}(0,\backslash sigma^2)\}$

by assumption, it follows immediately from appeal to Slutsky's theorem that

:$\{\backslash sqrt\{n\}[g(X\_n)-g(\backslash theta)]\; \backslash xrightarrow\{D\}\; \backslash mathcal\{N\}(0,\backslash sigma^2[g\text{'}(\backslash theta)]^2)\}.$

This concludes the proof.

Alternatively, one can add one more step at the end, to obtain the order of approximation:

:$$

\begin{align}

\sqrt{n}[g(X_n)-g(\theta)]&=g' \left (\tilde{\theta} \right )\sqrt{n}[X_n-\theta]\\[5pt]

&=\sqrt{n}[X_n-\theta]\left[ g'(\tilde{\theta} )+g'(\theta)-g'(\theta)\right]\\[5pt]

&=\sqrt{n}[X_n-\theta]\left[g'(\theta)\right]+\sqrt{n}[X_n-\theta]\left[ g'(\tilde{\theta} )-g'(\theta)\right]\\[5pt]

&=\sqrt{n}[X_n-\theta]\left[g'(\theta)\right]+O_p(1)\cdot o_p(1)\\[5pt]

&=\sqrt{n}[X_n-\theta]\left[g'(\theta)\right]+o_p(1)

\end{align}

This suggests that the error in the approximation converges to 0 in probability.

By definition, a consistent estimator B converges in probability to its true value β, and often a central limit theorem can be applied to obtain asymptotic normality:

:$\backslash sqrt\{n\}\backslash left(B-\backslash beta\backslash right)\backslash ,\backslash xrightarrow\{D\}\backslash ,N\backslash left(0,\; \backslash Sigma\; \backslash right),$

where n is the number of observations and Σ is a (symmetric positive semi-definite) covariance matrix. Suppose we want to estimate the variance of a scalar-valued function h of the estimator B. Keeping only the first two terms of the Taylor series, and using vector notation for the gradient, we can estimate h(B) as

:$h(B)\; \backslash approx\; h(\backslash beta)\; +\; \backslash nabla\; h(\backslash beta)^T\; \backslash cdot\; (B-\backslash beta)$

which implies the variance of h(B) is approximately

:$\backslash begin\{align\}$

\operatorname{Var}\left(h(B)\right) & \approx \operatorname{Var}\left(h(\beta) + \nabla h(\beta)^T \cdot (B-\beta)\right) \\[5pt]

& = \operatorname{Var}\left(h(\beta) + \nabla h(\beta)^T \cdot B - \nabla h(\beta)^T \cdot \beta\right) \\[5pt]

& = \operatorname{Var}\left(\nabla h(\beta)^T \cdot B\right) \\[5pt]

& = \nabla h(\beta)^T \cdot \operatorname{Cov}(B) \cdot \nabla h(\beta) \\[5pt]

& = \nabla h(\beta)^T \cdot \frac{\Sigma}{n} \cdot \nabla h(\beta)

\end{align}

One can use the mean value theorem (for real-valued functions of many variables) to see that this does not rely on taking first order approximation.

The delta method therefore implies that

:$\backslash sqrt\{n\}\backslash left(h(B)-h(\backslash beta)\backslash right)\backslash ,\backslash xrightarrow\{D\}\backslash ,N\backslash left(0,\; \backslash nabla\; h(\backslash beta)^T\; \backslash cdot\; \backslash Sigma\; \backslash cdot\; \backslash nabla\; h(\backslash beta)\backslash right)$

or in univariate terms,

:$\backslash sqrt\{n\}\backslash left(h(B)-h(\backslash beta)\backslash right)\backslash ,\backslash xrightarrow\{D\}\backslash ,N\backslash left(0,\; \backslash sigma^2\; \backslash cdot\; \backslash left(h^\backslash prime(\backslash beta)\backslash right)^2\; \backslash right).$

Suppose X_n is binomial with parameters $p\; \backslash in\; (0,1]$ and n. Since

:$\{\backslash sqrt\{n\}\; \backslash left[\; \backslash frac\{X\_n\}\{n\}-p\; \backslash right]\backslash ,\backslash xrightarrow\{D\}\backslash ,N(0,p\; (1-p))\},$

we can apply the Delta method with {{math|g(θ) {{=}} log(θ)}} to see

:$\{\backslash sqrt\{n\}\; \backslash left[\; \backslash log\backslash left(\; \backslash frac\{X\_n\}\{n\}\backslash right)-\backslash log(p)\backslash right]\; \backslash ,\backslash xrightarrow\{D\}\backslash ,N(0,p\; (1-p)\; [1/p]^2)\}$

Hence, even though for any finite n, the variance of $\backslash log\backslash left(\backslash frac\{X\_n\}\{n\}\backslash right)$ does not actually exist (since X_n can be zero), the asymptotic variance of $\backslash log\; \backslash left(\; \backslash frac\{X\_n\}\{n\}\; \backslash right)$ does exist and is equal to

:$\backslash frac\{1-p\}\{np\}.$

Note that since p>0, $\backslash Pr\; \backslash left(\; \backslash frac\{X\_n\}\{n\}\; >\; 0\; \backslash right)\; \backslash rightarrow\; 1$ as $n\; \backslash rightarrow\; \backslash infty$, so with probability converging to one, $\backslash log\backslash left(\backslash frac\{X\_n\}\{n\}\backslash right)$ is finite for large n.

Moreover, if $\backslash hat\; p$ and $\backslash hat\; q$ are estimates of different group rates from independent samples of sizes n and m respectively, then the logarithm of the estimated relative risk $\backslash frac\{\backslash hat\; p\}\{\backslash hat\; q\}$ has asymptotic variance equal to

:$\backslash frac\{1-p\}\{p\; \backslash ,\; n\}+\backslash frac\{1-q\}\{q\; \backslash ,\; m\}.$

This is useful to construct a hypothesis test or to make a confidence interval for the relative risk.

The delta method is often used in a form that is essentially identical to that above, but without the assumption that {{mvar|X_n}} or B is asymptotically normal. Often the only context is that the variance is "small". The results then just give approximations to the means and covariances of the transformed quantities. For example, the formulae presented in Klein (1953, p. 258) are:{{cite book |last=Klein |first=L. R. |author-link=Lawrence Klein |year=1953 |title=A Textbook of Econometrics |page=258 |url=https://books.google.com/books?id=uzwiAAAAMAAJ&pg=PA258 }}

:$\backslash begin\{align\}$

\operatorname{Var} \left(h_r \right) = & \sum_i \left( \frac{\partial h_r}{\partial B_i} \right)^2 \operatorname{Var}\left( B_i \right) + \sum_i \sum_{j \neq i} \left( \frac{ \partial h_r }{ \partial B_i } \right) \left( \frac{ \partial h_r }{ \partial B_j } \right) \operatorname{Cov}\left( B_i, B_j \right) \\

\operatorname{Cov}\left( h_r, h_s \right) = & \sum_i \left( \frac{ \partial h_r }{ \partial B_i } \right) \left( \frac{\partial h_s }{ \partial B_i } \right) \operatorname{Var}\left( B_i \right) + \sum_i \sum_{j \neq i} \left( \frac{\partial h_r}{\partial B_i} \right) \left(\frac{\partial h_s}{\partial B_j} \right) \operatorname{Cov}\left( B_i, B_j \right)

\end{align}

where {{mvar|h_r}} is the rth element of h(B) and B_i is the ith element of B.

When {{math|1=g′(θ) = 0}} the delta method cannot be applied. However, if {{math|g′′(θ)}} exists and is not zero, the second-order delta method can be applied. By the Taylor expansion, $n[g(X\_n)-g(\backslash theta)]=\backslash frac\{1\}\{2\}n[X\_n-\backslash theta]^2\backslash left[g\text{'}\text{'}(\backslash theta)\backslash right]+o\_p(1)$, so that the variance of $g\backslash left(X\_n\backslash right)$ relies on up to the 4th moment of $X\_n$.

The second-order delta method is also useful in conducting a more accurate approximation of $g\backslash left(X\_n\backslash right)$'s distribution when sample size is small.

$\backslash sqrt\{n\}[g(X\_n)-g(\backslash theta)]=\backslash sqrt\{n\}[X\_n-\backslash theta]\; g\text{'}(\backslash theta)+\backslash frac\{1\}\{2\}\backslash frac\{n[X\_n-\backslash theta]^2\}\{\backslash sqrt\{n\}\}\; g\text{'}\text{'}(\backslash theta)\; +o\_p(1)$.

For example, when $X\_n$ follows the standard normal distribution, $g\backslash left(X\_n\backslash right)$ can be approximated as the weighted sum of a standard normal and a chi-square with 1 degree of freedom.

A version of the delta method exists in nonparametric statistics. Let $X\_i\; \backslash sim\; F$ be an independent and identically distributed random variable with a sample of size $n$ with an empirical distribution function $\backslash hat\{F\}\_n$, and let $T$ be a functional. If $T$ is Hadamard differentiable with respect to the Chebyshev metric, then

:$\backslash frac\{\; T(\backslash hat\{F\}\_n)\; -\; T(F)\; \}\{\; \backslash widehat\{\backslash text\{se\}\}\; \}\; \backslash xrightarrow\{D\}\; N(0,\; 1)$

where $\backslash widehat\{\backslash text\{se\}\}\; =\; \backslash frac\{\backslash hat\{\backslash tau\}\}\{\backslash sqrt\{n\}\}$ and $\backslash hat\{\backslash tau\}^2\; =\; \backslash frac\{1\}\{n\}\backslash sum\_\{i=1\}^n\; \backslash hat\{L\}^2(X\_i)$, with $\backslash hat\{L\}(x)\; =\; L\_\{\backslash hat\{F\}\_n\}(\backslash delta\_x)$ denoting the empirical influence function for $T$. A nonparametric $(1-\backslash alpha)$ pointwise asymptotic confidence interval for $T(F)$ is therefore given by

:$T(\backslash hat\{F\}\_n)\; \backslash pm\; z\_\{\backslash alpha/2\}\; \backslash widehat\{\backslash text\{se\}\}$

where $z\_q$ denotes the $q$-quantile of the standard normal. See Wasserman (2006) p. 19f. for details and examples.

• Taylor expansions for the moments of functions of random variables
• Variance-stabilizing transformation

{{Reflist}}

• {{cite journal |last=Oehlert |first=G. W. |year=1992 |title=A Note on the Delta Method |journal=The American Statistician |volume=46 |issue=1 |pages=27–29 |jstor=2684406 |doi=10.1080/00031305.1992.10475842}}
• {{cite book |first=Kirk M. |last=Wolter |chapter=Taylor Series Methods |title=Introduction to Variance Estimation |location=New York |publisher=Springer |year=1985 |isbn=0-387-96119-4 |pages=221–247 |chapter-url=https://books.google.com/books?id=EadxTw0t2dMC&pg=PA221 }}
• {{cite book |first=Larry |last=Wasserman |title=All of Nonparametric Statistics| location=New York |publisher=Springer |year=2006 |isbn=0-387-25145-6 |pages=19–20 }}

• {{cite web|first=Søren|last=Asmussen|publisher=Aarhus University|title=Some Applications of the Delta Method|url=https://data.math.au.dk/courses/advsimmethod/Fall05/notes/1209.pdf|url-status=dead|archive-url=https://web.archive.org/web/20150525191234/http://data.math.au.dk/courses/advsimmethod/Fall05/notes/1209.pdf|archive-date=May 25, 2015|work=Lecture notes|date=2005}}
• {{cite web |title=Explanation of the delta method |first=Alan H. |last=Feiveson |publisher=Stata Corp. |url=https://www.stata.com/support/faqs/statistics/delta-method/ }}

Category:Estimation methods

Category:Statistical approximations

Category:Articles containing proofs

Category:Statistics articles needing expert attention