maximum a posteriori estimation

Assume that we want to estimate an unobserved population parameter $\backslash theta$ on the basis of observations $x$. Let $f$ be the sampling distribution of $x$, so that $f(x\backslash mid\backslash theta)$ is the probability of $x$ when the underlying population parameter is $\backslash theta$. Then the function:

:$\backslash theta\; \backslash mapsto\; f(x\; \backslash mid\; \backslash theta)\; \backslash !$

is known as the likelihood function and the estimate:

:$\backslash hat\{\backslash theta\}\_\{\backslash mathrm\{MLE\}\}(x)\; =\; \backslash underset\{\backslash theta\}\{\backslash operatorname\{arg\backslash ,max\}\}\; \backslash \; f(x\; \backslash mid\; \backslash theta)\; \backslash !$

is the maximum likelihood estimate of $\backslash theta$.

Now assume that a prior distribution $g$ over $\backslash theta$ exists. This allows us to treat $\backslash theta$ as a random variable as in Bayesian statistics. We can calculate the posterior density of $\backslash theta$ using Bayes' theorem:

:$\backslash theta\; \backslash mapsto\; f(\backslash theta\; \backslash mid\; x)\; =\; \backslash frac\{f(x\; \backslash mid\; \backslash theta)\; \backslash ,\; g(\backslash theta)\}\{\backslash displaystyle\backslash int\_\{\backslash Theta\}\; f(x\; \backslash mid\; \backslash vartheta)\; \backslash ,\; g(\backslash vartheta)\; \backslash ,\; d\backslash vartheta\}\; \backslash !$

where $g$ is density function of $\backslash theta$, $\backslash Theta$ is the domain of $g$.

The method of maximum a posteriori estimation then estimates $\backslash theta$ as the mode of the posterior density of this random variable:

:$\backslash begin\{align\}$

\hat{\theta}_{\mathrm{MAP}}(x) &

= \underset{\theta}{\operatorname{arg\,max}} \ f(\theta \mid x) \\

& = \underset{\theta}{\operatorname{arg\,max}} \ \frac{f(x \mid \theta) \, g(\theta)}

{\displaystyle\int_{\Theta} f(x \mid \vartheta) \, g(\vartheta) \, d\vartheta} \\

& = \underset{\theta}{\operatorname{arg\,max}} \ f(x \mid \theta) \, g(\theta).

\end{align}

\!

The denominator of the posterior density (the marginal likelihood of the model) is always positive and does not depend on $\backslash theta$ and therefore plays no role in the optimization. Observe that the MAP estimate of $\backslash theta$ coincides with the ML estimate when the prior $g$ is uniform (i.e., $g$ is a constant function), which occurs whenever the prior distribution is taken as the reference measure, as is typical in function-space applications.

When the loss function is of the form

:$$

L(\theta, a) =

\begin{cases}

0, & \text{if } |a-\theta|

1, & \text{otherwise}, \\

\end{cases}

as $c$ goes to 0, the Bayes estimator approaches the MAP estimator, provided that the distribution of $\backslash theta$ is quasi-concave.{{Cite journal|last=Bassett|first=Robert|last2=Deride|first2=Julio|date=2018-01-30|title=Maximum a posteriori estimators as a limit of Bayes estimators|journal=Mathematical Programming|language=en|pages=1–16|doi=10.1007/s10107-018-1241-0|issn=0025-5610|arxiv=1611.05917}} But generally a MAP estimator is not a Bayes estimator unless $\backslash theta$ is discrete.

MAP estimates can be computed in several ways:

1. Analytically, when the mode(s) of the posterior density can be given in closed form. This is the case when conjugate priors are used.
2. Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
3. Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
4. Via a Monte Carlo method using simulated annealing

While only mild conditions are required for MAP estimation to be a limiting case of Bayes estimation (under the 0–1 loss function), it is not representative of Bayesian methods in general. This is because MAP estimates are point estimates, and depend on the arbitrary choice of reference measure, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with credible intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively—which are more representative of typical loss functions—and for a continuous posterior distribution there is no loss function which suggests the MAP is the optimal point estimator. In addition, the posterior density may often not have a simple analytic form: in this case, the distribution can be simulated using Markov chain Monte Carlo techniques, while optimization to find the mode(s) of the density may be difficult or impossible.{{citation needed|date=August 2012}}

File:Bimodal density.svg in which the highest mode is uncharacteristic of the majority of the distribution]]

In many types of models, such as mixture models, the posterior may be multi-modal. In such a case, the usual recommendation is that one should choose the highest mode: this is not always feasible (global optimization is a difficult problem), nor in some cases even possible (such as when identifiability issues arise). Furthermore, the highest mode may be uncharacteristic of the majority of the posterior, especially in many dimensions.

Finally, unlike ML estimators, the MAP estimate is not invariant under reparameterization. Switching from one parameterization to another involves introducing a Jacobian that impacts on the location of the maximum.{{cite book|last1=Murphy|first1=Kevin P.|title=Machine learning : a probabilistic perspective|date=2012|publisher=MIT Press|location=Cambridge, Massachusetts|isbn=978-0-262-01802-9 |pages=151–152}} In contrast, Bayesian posterior expectations are invariant under reparameterization.

As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using a MAP estimate, consider the case where there is a need to classify inputs $x$ as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification $h\_1$, $h\_2$ and $h\_3$ with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, $x$, $h\_1$ classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier $h\_1$, $x$ is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify $x$ as negative.

Suppose that we are given a sequence $(x\_1,\; \backslash dots,\; x\_n)$ of IID $N(\backslash mu,\backslash sigma\_v^2\; )$ random variables and a prior distribution of $\backslash mu$ is given by $N(\backslash mu\_0,\backslash sigma\_m^2\; )$. We wish to find the MAP estimate of $\backslash mu$. Note that the normal distribution is its own conjugate prior, so we will be able to find a closed-form solution analytically.

The function to be maximized is then given by{{Cite book |last=Young |first=G. A. |url=https://www.cambridge.org/core/books/essentials-of-statistical-inference/7CDE4B08DD68DE7EE0B00F778FC29CCD |title=Essentials of Statistical Inference |last2=Smith |first2=R. L. |date=2005 |publisher=Cambridge University Press |isbn=978-0-521-83971-6 |series=Cambridge Series in Statistical and Probabilistic Mathematics |location=Cambridge}}

:$g(\backslash mu)\; f(x\; \backslash mid\; \backslash mu)=\backslash pi(\backslash mu)\; L(\backslash mu)\; =\; \backslash frac\{1\}\{\backslash sqrt\{2\; \backslash pi\}\; \backslash sigma\_m\}\; \backslash exp\backslash left(-\backslash frac\{1\}\{2\}\; \backslash left(\backslash frac\{\backslash mu-\backslash mu\_0\}\{\backslash sigma\_m\}\backslash right)^2\backslash right)\; \backslash prod\_\{j=1\}^n\; \backslash frac\{1\}\{\backslash sqrt\{2\; \backslash pi\}\; \backslash sigma\_v\}\; \backslash exp\backslash left(-\backslash frac\{1\}\{2\}\; \backslash left(\backslash frac\{x\_j\; -\; \backslash mu\}\{\backslash sigma\_v\}\backslash right)^2\backslash right),$

which is equivalent to minimizing the following function of $\backslash mu$:

:$\backslash sum\_\{j=1\}^n\; \backslash left(\backslash frac\{x\_j\; -\; \backslash mu\}\{\backslash sigma\_v\}\backslash right)^2\; +\; \backslash left(\backslash frac\{\backslash mu-\backslash mu\_0\}\{\backslash sigma\_m\}\backslash right)^2.$

Thus, we see that the MAP estimator for μ is given by

:$\backslash hat\{\backslash mu\}\_\backslash mathrm\{MAP\}\; =\; \backslash frac\{\backslash sigma\_m^2\backslash ,n\}\{\backslash sigma\_m^2\; \backslash ,n+\; \backslash sigma\_v^2\; \}\; \backslash left(\backslash frac\{1\}\{n\}\; \backslash sum\_\{j=1\}^n\; x\_j\; \backslash right)\; +\; \backslash frac\{\backslash sigma\_v^2\}\{\backslash sigma\_m^2\; \backslash ,n+\; \backslash sigma\_v^2\; \}\; \backslash ,\backslash mu\_0$

=\frac{\sigma_m^2\left(\sum_{j=1}^n x_j\right) + \sigma_v^2 \,\mu_0}{\sigma_m^2\,n + \sigma_v^2 }.

which turns out to be a linear interpolation between the prior mean and the sample mean weighted by their respective covariances.

The case of $\backslash sigma\_m\; \backslash to\; \backslash infty$ is called a non-informative prior and leads to an improper probability distribution; in this case $\backslash hat\{\backslash mu\}\_\backslash mathrm\{MAP\}\; \backslash to\; \backslash hat\{\backslash mu\}\_\backslash mathrm\{MLE\}.$

{{Reflist}}

• {{cite book |first=M. |last=DeGroot |title=Optimal Statistical Decisions |publisher=McGraw-Hill |year=1970 |isbn=0-07-016242-5 }}
• {{cite book |first=Harold W. |last=Sorenson |year=1980 |title=Parameter Estimation: Principles and Problems |publisher=Marcel Dekker |isbn=0-8247-6987-2 }}
• {{cite book |first=Anders |last=Hald |chapter=Gauss's Derivation of the Normal Distribution and the Method of Least Squares, 1809 |title=A History of Parametric Statistical Inference from Bernoulli to Fisher, 1713–1935 |year=2007 |publisher=Springer |location=New York |pages=55–61 |isbn=978-0-387-46409-1 }}

{{Statistics|inference}}

Category:Bayesian estimation

Category:Logic and statistics

Estimation