Normal-gamma distribution

For a pair of random variables, (X,T), suppose that the conditional distribution of X given T is given by

:$X\backslash mid\; T\; \backslash sim\; N(\backslash mu,1\; /(\backslash lambda\; T))\; \backslash ,\backslash !\; ,$

meaning that the conditional distribution is a normal distribution with mean $\backslash mu$ and precision $\backslash lambda\; T$ — equivalently, with variance $1\; /\; (\backslash lambda\; T)\; .$

Suppose also that the marginal distribution of T is given by

:$T\; \backslash mid\; \backslash alpha,\; \backslash beta\; \backslash sim\; \backslash operatorname\{Gamma\}(\backslash alpha,\backslash beta),$

where this means that T has a gamma distribution. Here λ, α and β are parameters of the joint distribution.

Then (X,T) has a normal-gamma distribution, and this is denoted by

:$(X,T)\; \backslash sim\; \backslash operatorname\{NormalGamma\}(\backslash mu,\backslash lambda,\backslash alpha,\backslash beta).$

The joint probability density function of (X,T) is

: $$f(x,\backslash tau\backslash mid\backslash mu,\backslash lambda,\backslash alpha,\backslash beta)\; =\; \backslash frac\{\backslash beta^\backslash alpha\; \backslash sqrt\{\backslash lambda\}\}\{\backslash Gamma(\backslash alpha)\backslash sqrt\{2\backslash pi\}\}\; \backslash ,\; \backslash tau^\{\backslash alpha-\backslash frac\{1\}\{2\}\}\backslash ,e^\{-\backslash beta\backslash tau\}\backslash exp\backslash left(\; -\backslash frac\{\; \backslash lambda\; \backslash tau\; (x-\; \backslash mu)^2\}\{2\}\backslash right),$$

where the conditional probability for $$f(x,\; \backslash tau\backslash mid\backslash mu,\backslash lambda,\backslash alpha,\backslash beta)\; =\; f(x\backslash mid\backslash tau,\backslash mu,\backslash lambda,\backslash alpha,\backslash beta)f(\backslash tau\backslash mid\backslash mu,\backslash lambda,\backslash alpha,\backslash beta)$$

was used.

By construction, the marginal distribution of $\backslash tau$ is a gamma distribution, and the conditional distribution of $x$ given $\backslash tau$ is a Gaussian distribution. The marginal distribution of $x$ is a three-parameter non-standardized Student's t-distribution with parameters $(\backslash nu,\; \backslash mu,\; \backslash sigma^2)=(2\backslash alpha,\; \backslash mu,\; \backslash beta/(\backslash lambda\backslash alpha))$.{{citation needed|date=April 2013}}

The normal-gamma distribution is a four-parameter exponential family with natural parameters $\backslash alpha-1/2,\; -\backslash beta-\backslash lambda\backslash mu^2/2,\; \backslash lambda\backslash mu,\; -\backslash lambda/2$ and natural statistics $\backslash ln\backslash tau,\; \backslash tau,\; \backslash tau\; x,\; \backslash tau\; x^2$.{{citation needed|date=April 2013}}

The following moments can be easily computed using the moment generating function of the sufficient statistic:{{Citation |last=Wasserman |first=Larry |title=Parametric Inference |date=2004 |url=http://dx.doi.org/10.1007/978-0-387-21736-9_9 |work=Springer Texts in Statistics |pages=119–148 |access-date=2023-12-08 |place=New York, NY |publisher=Springer New York |isbn=978-1-4419-2322-6}}

:$\backslash operatorname\{E\}(\backslash ln\; T)=\backslash psi\backslash left(\backslash alpha\backslash right)\; -\; \backslash ln\backslash beta,$

where $\backslash psi\backslash left(\backslash alpha\backslash right)$ is the digamma function,

:$$

\begin{align}

\operatorname{E}(T) & =\frac{\alpha}{\beta}, \\[5pt]

\operatorname{E}(TX) & =\mu \frac{\alpha}{\beta}, \\[5pt]

\operatorname{E}(TX^2) & =\frac{1}{\lambda} + \mu^2 \frac{\alpha}{\beta}.

\end{align}

If $(X,T)\; \backslash sim\; \backslash mathrm\{NormalGamma\}(\backslash mu,\backslash lambda,\backslash alpha,\backslash beta),$ then for any $b>0,\; (bX,bT)$ is distributed as{{citation needed|date=April 2013}} $\{\backslash rm\; NormalGamma\}(b\backslash mu,\; \backslash lambda/\; b^3,\; \backslash alpha,\; \backslash beta/\; b\; ).$

Assume that x is distributed according to a normal distribution with unknown mean $\backslash mu$ and precision $\backslash tau$.

:$x\; \backslash sim\; \backslash mathcal\{N\}(\backslash mu,\; \backslash tau^\{-1\})$

and that the prior distribution on $\backslash mu$ and $\backslash tau$, $(\backslash mu,\backslash tau)$, has a normal-gamma distribution

:$$

(\mu,\tau) \sim \text{NormalGamma}(\mu_0,\lambda_0,\alpha_0,\beta_0) ,

for which the density {{pi}} satisfies

:$$

\pi(\mu,\tau) \propto \tau^{\alpha_0-\frac{1}{2}}\,\exp[-\beta_0\tau]\,\exp\left[ -\frac{\lambda_0\tau(\mu-\mu_0)^2} 2 \right].

Suppose

: $$

x_1,\ldots,x_n \mid \mu,\tau \sim \operatorname{{i.}{i.}{d.}} \operatorname N\left( \mu, \tau^{-1} \right),

i.e. the components of $\backslash mathbf\; X\; =\; (x\_1,\backslash ldots,x\_n)$ are conditionally independent given $\backslash mu,\backslash tau$ and the conditional distribution of each of them given $\backslash mu,\backslash tau$ is normal with expected value $\backslash mu$ and variance $1\; /\; \backslash tau.$ The posterior distribution of $\backslash mu$ and $\backslash tau$ given this dataset $\backslash mathbb\; X$ can be analytically determined by Bayes' theorem{{cite web |url=http://www.trinity.edu/cbrown/bayesweb/ |title=Bayes' Theorem: Introduction |access-date=2014-08-05 |url-status=live |archive-url=https://web.archive.org/web/20140807091855/http://www.trinity.edu/cbrown/bayesweb/ |archive-date=2014-08-07 }} explicitly,

:$\backslash mathbf\{P\}(\backslash tau,\backslash mu\; \backslash mid\; \backslash mathbf\{X\})\; \backslash propto\; \backslash mathbf\{L\}(\backslash mathbf\{X\}\; \backslash mid\; \backslash tau,\backslash mu)\; \backslash pi(\backslash tau,\backslash mu),$

where $\backslash mathbf\{L\}$ is the likelihood of the parameters given the data.

Since the data are i.i.d, the likelihood of the entire dataset is equal to the product of the likelihoods of the individual data samples:

:$$

\mathbf{L}(\mathbf{X} \mid \tau, \mu) = \prod_{i=1}^n \mathbf{L}(x_i \mid \tau, \mu).

This expression can be simplified as follows:

:$$

\begin{align}

\mathbf{L}(\mathbf{X} \mid \tau, \mu) & \propto \prod_{i=1}^n \tau^{1/2} \exp\left[\frac{-\tau}{2}(x_i-\mu)^2\right] \\[5pt]

& \propto \tau^{n/2} \exp\left[\frac{-\tau}{2}\sum_{i=1}^n(x_i-\mu)^2\right] \\[5pt]

& \propto \tau^{n/2} \exp\left[\frac{-\tau}{2} \sum_{i=1}^n(x_i-\bar{x} +\bar{x} -\mu)^2 \right] \\[5pt]

& \propto \tau^{n/2} \exp\left[\frac{-\tau} 2 \sum_{i=1}^n \left((x_i-\bar{x})^2 + (\bar{x} -\mu)^2 \right)\right] \\[5pt]

& \propto \tau^{n/2} \exp\left[\frac{-\tau}{2}\left(n s + n(\bar{x} -\mu)^2\right)\right],

\end{align}

where $\backslash bar\{x\}=\; \backslash frac\{1\}\{n\}\backslash sum\_\{i=1\}^n\; x\_i$, the mean of the data samples, and $s=\; \backslash frac\{1\}\{n\}\; \backslash sum\_\{i=1\}^n(x\_i-\backslash bar\{x\})^2$, the sample variance.

The posterior distribution of the parameters is proportional to the prior times the likelihood.

:$$

\begin{align}

\mathbf{P}(\tau, \mu \mid \mathbf{X}) &\propto \mathbf{L}(\mathbf{X} \mid \tau,\mu) \pi(\tau,\mu) \\

&\propto \tau^{n/2} \exp \left[ \frac{-\tau}{2}\left(n s + n(\bar{x} -\mu)^2\right) \right]

\tau^{\alpha_0-\frac{1}{2}}\,\exp[{-\beta_0\tau}]\,\exp\left[-\frac{\lambda_0\tau(\mu-\mu_0)^2}{2}\right] \\

&\propto \tau^{\frac{n}{2} + \alpha_0 - \frac{1}{2}}\exp\left[-\tau \left( \frac{1}{2} n s + \beta_0 \right) \right] \exp\left[- \frac{\tau}{2}\left(\lambda_0(\mu-\mu_0)^2 + n(\bar{x} -\mu)^2\right)\right]

\end{align}

The final exponential term is simplified by completing the square.

:$$

\begin{align}

\lambda_0(\mu-\mu_0)^2 + n(\bar{x} -\mu)^2&=\lambda_0 \mu^2 - 2 \lambda_0 \mu \mu_0 + \lambda_0 \mu_0^2 + n \mu^2 - 2 n \bar{x} \mu + n \bar{x}^2 \\

&= (\lambda_0 + n) \mu^2 - 2(\lambda_0 \mu_0 + n \bar{x}) \mu + \lambda_0 \mu_0^2 +n \bar{x}^2 \\

&= (\lambda_0 + n)( \mu^2 - 2 \frac{\lambda_0 \mu_0 + n \bar{x}}{\lambda_0 + n} \mu ) + \lambda_0 \mu_0^2 +n \bar{x}^2 \\

&= (\lambda_0 + n)\left(\mu - \frac{\lambda_0 \mu_0 + n \bar{x}}{\lambda_0 + n} \right) ^2 + \lambda_0 \mu_0^2 +n \bar{x}^2 - \frac{\left(\lambda_0 \mu_0 +n \bar{x}\right)^2} {\lambda_0 + n} \\

&= (\lambda_0 + n)\left(\mu - \frac{\lambda_0 \mu_0 + n \bar{x}}{\lambda_0 + n} \right) ^2 + \frac{\lambda_0 n (\bar{x} - \mu_0 )^2}{\lambda_0 +n}

\end{align}

On inserting this back into the expression above,

:$$

\begin{align}

\mathbf{P}(\tau, \mu \mid \mathbf{X}) & \propto \tau^{\frac{n}{2} + \alpha_0 - \frac{1}{2}} \exp \left[-\tau \left( \frac{1}{2} n s + \beta_0 \right) \right] \exp \left[- \frac{\tau}{2} \left( \left(\lambda_0 + n \right) \left(\mu- \frac{\lambda_0 \mu_0 + n \bar{x}}{\lambda_0 + n} \right)^2 + \frac{\lambda_0 n (\bar{x} - \mu_0 )^2}{\lambda_0 +n} \right) \right]\\

& \propto \tau^{\frac{n}{2} + \alpha_0 - \frac{1}{2}} \exp \left[-\tau \left( \frac{1}{2} n s + \beta_0 + \frac{\lambda_0 n (\bar{x} - \mu_0 )^2}{2(\lambda_0 +n)} \right) \right] \exp \left[- \frac{\tau}{2} \left(\lambda_0 + n \right) \left(\mu- \frac{\lambda_0 \mu_0 + n \bar{x}}{\lambda_0 + n} \right)^2 \right]

\end{align}

This final expression is in exactly the same form as a Normal-Gamma distribution, i.e.,

:$$

\mathbf{P}(\tau, \mu \mid \mathbf{X}) = \text{NormalGamma}\left(\frac{\lambda_0 \mu_0 + n \bar{x}}{\lambda_0 + n}, \lambda_0 + n, \alpha_0+\frac{n}{2}, \beta_0+ \frac{1}{2}\left(n s + \frac{\lambda_0 n (\bar{x} - \mu_0 )^2}{\lambda_0 +n} \right) \right)

The interpretation of parameters in terms of pseudo-observations is as follows:

• The new mean takes a weighted average of the old pseudo-mean and the observed mean, weighted by the number of associated (pseudo-)observations.
• The precision was estimated from $2\backslash alpha$ pseudo-observations (i.e. possibly a different number of pseudo-observations, to allow the variance of the mean and precision to be controlled separately) with sample mean $\backslash mu$ and sample variance $\backslash frac\{\backslash beta\}\{\backslash alpha\}$ (i.e. with sum of squared deviations $2\backslash beta$).
• The posterior updates the number of pseudo-observations ($\backslash lambda\_\{0\}$) simply by adding the corresponding number of new observations ($n$).
• The new sum of squared deviations is computed by adding the previous respective sums of squared deviations. However, a third "interaction term" is needed because the two sets of squared deviations were computed with respect to different means, and hence the sum of the two underestimates the actual total squared deviation.

As a consequence, if one has a prior mean of $\backslash mu\_0$ from $n\_\backslash mu$ samples and a prior precision of $\backslash tau\_0$ from $n\_\backslash tau$ samples, the prior distribution over $\backslash mu$ and $\backslash tau$ is

:$$

\mathbf{P}(\tau,\mu \mid \mathbf{X}) = \operatorname{NormalGamma} \left(\mu_0, n_\mu , \frac{n_\tau}{2}, \frac{n_\tau}{2 \tau_0}\right)

and after observing $n$ samples with mean $\backslash mu$ and variance $s$, the posterior probability is

:$$

\mathbf{P}(\tau,\mu \mid \mathbf{X}) = \text{NormalGamma}\left( \frac{n_\mu \mu_0 + n \mu}{n_\mu +n}, n_\mu +n ,\frac{1}{2}(n_\tau+n), \frac{1}{2}\left(\frac{n_\tau}{\tau_0} + n s + \frac{n_\mu n (\mu-\mu_0)^2}{n_\mu+n}\right) \right)

Note that in some programming languages, such as Matlab, the gamma distribution is implemented with the inverse definition of $\backslash beta$, so the fourth argument of the Normal-Gamma distribution is $2\; \backslash tau\_0\; /n\_\backslash tau$.

Generation of random variates is straightforward:

1. Sample $\backslash tau$ from a gamma distribution with parameters $\backslash alpha$ and $\backslash beta$
2. Sample $x$ from a normal distribution with mean $\backslash mu$ and variance $1/(\backslash lambda\; \backslash tau)$

• The normal-inverse-gamma distribution is the same distribution parameterized by variance rather than precision
• The normal-exponential-gamma distribution

{{Reflist}}

• Bernardo, J.M.; Smith, A.F.M. (1993) Bayesian Theory, Wiley. {{ISBN|0-471-49464-X}}
• Dearden et al. [http://www.aaai.org/Papers/AAAI/1998/AAAI98-108.pdf "Bayesian Q-learning"], Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98), July 26–30, 1998, Madison, Wisconsin, USA.

{{ProbDistributions|multivariate}}

{{DEFAULTSORT:Normal-gamma distribution}}

Category:Multivariate continuous distributions

Category:Conjugate prior distributions

Category:Normal distribution