Gaussian process

A time continuous stochastic process $\backslash left\backslash \{X\_t\; ;\; t\backslash in\; T\backslash right\backslash \}$ is Gaussian if and only if for every finite set of indices $t\_1,\backslash ldots,t\_k$ in the index set $T$

$$\backslash mathbf\{X\}\_\{t\_1,\; \backslash ldots,\; t\_k\}\; =\; (X\_\{t\_1\},\; \backslash ldots,\; X\_\{t\_k\})$$

is a multivariate Gaussian random variable.{{cite book| last=MacKay | first=David J.C.| author-link=David J.C. MacKay| title=Information Theory, Inference, and Learning Algorithms |year=2003 | pages=540|publisher=Cambridge University Press| isbn=9780521642989|url=http://www.inference.phy.cam.ac.uk/itprnn/book.pdf | quote=The probability distribution of a function $y(\backslash mathbf\{x\})$ is a Gaussian processes if for any finite selection of points $\backslash mathbf\{x\}^\{(1)\},\backslash mathbf\{x\}^\{(2)\},\backslash ldots,\backslash mathbf\{x\}^\{(N)\}$, the density $P(y(\backslash mathbf\{x\}^\{(1)\}),y(\backslash mathbf\{x\}^\{(2)\}),\backslash ldots,y(\backslash mathbf\{x\}^\{(N)\}))$ is a Gaussian}}

As the sum of independent and Gaussian distributed random variables is again Gaussian distributed, that is the same as saying every linear combination of $(X\_\{t\_1\},\; \backslash ldots,\; X\_\{t\_k\})$ has a univariate Gaussian (or normal) distribution.

Using characteristic functions of random variables with $i$ denoting the imaginary unit such that $i^2\; =-1$, the Gaussian property can be formulated as follows: $\backslash left\backslash \{X\_t\; ;\; t\backslash in\; T\backslash right\backslash \}$ is Gaussian if and only if, for every finite set of indices $t\_1,\backslash ldots,t\_k$, there are real-valued $\backslash sigma\_\{\backslash ell\; j\}$, $\backslash mu\_\backslash ell$ with $\backslash sigma\_\{jj\}\; >\; 0$ such that the following equality holds for all $s\_1,s\_2,\backslash ldots,s\_k\backslash in\backslash mathbb\{R\}$,

$$\{\; \backslash mathbb\; E\; \}\backslash left[\backslash exp\backslash left(i\; \backslash sum\_\{\backslash ell=1\}^k\; s\_\backslash ell\; \backslash ,\; \backslash mathbf\{X\}\_\{t\_\backslash ell\}\backslash right)\backslash right]\; =\; \backslash exp\; \backslash left(-\backslash tfrac\{1\}\{2\}\; \backslash sum\_\{\backslash ell,\; j\}\; \backslash sigma\_\{\backslash ell\; j\}\; s\_\backslash ell\; s\_j\; +\; i\; \backslash sum\_\backslash ell\; \backslash mu\_\backslash ell\; s\_\backslash ell\backslash right),$$

or $\{\; \backslash mathbb\; E\; \}\; \backslash left[\; \{\backslash mathrm\; e\}^\{\; i\backslash ,\; \backslash mathbf\{s\}\backslash ,\; (\backslash mathbf\{X\}\_t\; -\; \backslash mathbf\{\backslash mu\})\; \}\; \backslash right]\; =\; \{\backslash mathrm\; e\}^\{\; -\; \backslash mathbf\{s\}\backslash ,\; \backslash sigma\backslash ,\; \backslash mathbf\{s\}/2\; \}$.

The numbers $\backslash sigma\_\{\backslash ell\; j\}$ and $\backslash mu\_\backslash ell$ can be shown to be the covariances and means of the variables in the process.{{cite book |last=Dudley |first=R.M. |title=Real Analysis and Probability |year=1989 |publisher=Wadsworth and Brooks/Cole |isbn=0-534-10050-3 }}

The variance of a Gaussian process is finite at any time $t$, formally{{cite book|author=Amos Lapidoth|title=A Foundation in Digital Communication|url=https://books.google.com/books?id=6oTuDQAAQBAJ&q=independence|date=8 February 2017|publisher=Cambridge University Press|isbn=978-1-107-17732-1}}{{rp|p. 515}}

For general stochastic processes strict-sense stationarity implies wide-sense stationarity but not every wide-sense stationary stochastic process is strict-sense stationary. However, for a Gaussian stochastic process the two concepts are equivalent.{{rp|p. 518}}

A Gaussian stochastic process is strict-sense stationary if and only if it is wide-sense stationary.

There is an explicit representation for stationary Gaussian processes.{{cite journal |last1= Kac| first1= M.|last2 = Siegert|first2 = A.J.F|year= 1947|title= An Explicit Representation of a Stationary Gaussian Process|journal= The Annals of Mathematical Statistics|volume=18|issue=3|pages= 438–442|doi= 10.1214/aoms/1177730391 |doi-access= free}} A simple example of this representation is

$$X\_t\; =\; \backslash cos(at)\backslash ,\; \backslash xi\_1\; +\; \backslash sin(at)\backslash ,\; \backslash xi\_2$$

where $\backslash xi\_1$ and $\backslash xi\_2$ are independent random variables with the standard normal distribution.

{{main|Covariance function}}

{{further|Variogram}}

A key fact of Gaussian processes is that they can be completely defined by their second-order statistics.{{cite book |last=Bishop |first=C.M. |title= Pattern Recognition and Machine Learning |year=2006 |publisher=Springer |isbn=978-0-387-31073-2}} Thus, if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour. Importantly the non-negative definiteness of this function enables its spectral decomposition using the Karhunen–Loève expansion. Basic aspects that can be defined through the covariance function are the process' stationarity, isotropy, smoothness and periodicity.{{cite book |last=Barber |first=David |title=Bayesian Reasoning and Machine Learning |url=http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage |year=2012 |publisher=Cambridge University Press |isbn=978-0-521-51814-7}}{{cite book |last=Rasmussen |first=C.E. |author2=Williams, C.K.I |title=Gaussian Processes for Machine Learning |url=http://www.gaussianprocess.org/gpml/ |year=2006 |publisher=MIT Press |isbn=978-0-262-18253-9}}

Stationarity refers to the process' behaviour regarding the separation of any two points $x$ and $x\text{'}$. If the process is stationary, the covariance function depends only on $x-x\text{'}$. For example, the Ornstein–Uhlenbeck process is stationary.

If the process depends only on $|x-x\text{'}|$, the Euclidean distance (not the direction) between $x$ and $x\text{'}$, then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be homogeneous;{{cite book |last=Grimmett |first=Geoffrey |author2=David Stirzaker|title= Probability and Random Processes| year=2001 |publisher=Oxford University Press |isbn=978-0198572220}} in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

Ultimately Gaussian processes translate as taking priors on functions and the smoothness of these priors can be induced by the covariance function. If we expect that for "near-by" input points $x$ and $x\text{'}$ their corresponding output points $y$ and $y\text{'}$ to be "near-by" also, then the assumption of continuity is present. If we wish to allow for significant displacement then we might choose a rougher covariance function. Extreme examples of the behaviour is the Ornstein–Uhlenbeck covariance function and the squared exponential where the former is never differentiable and the latter infinitely differentiable.

Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input $x$ to a two dimensional vector $u(x)\; =\; \backslash left(\; \backslash cos(x),\; \backslash sin(x)\; \backslash right)$.

File:Gaussian process draws from prior distribution.png

There are a number of common covariance functions:

• Constant : $K\_\backslash operatorname\{C\}(x,x\text{'})\; =\; C$
• Linear: $K\_\backslash operatorname\{L\}(x,x\text{'})\; =\; x^\backslash mathsf\{T\}\; x\text{'}$
• white Gaussian noise: $K\_\backslash operatorname\{GN\}(x,x\text{'})\; =\; \backslash sigma^2\; \backslash delta\_\{x,x\text{'}\}$
• Squared exponential: $K\_\backslash operatorname\{SE\}(x,x\text{'})\; =\; \backslash exp\; \backslash left(-\backslash tfrac\{d^2\}\{2\backslash ell^2\}\; \backslash right)$
• Ornstein–Uhlenbeck: $K\_\backslash operatorname\{OU\}(x,x\text{'})\; =\; \backslash exp\; \backslash left(-\backslash tfrac\{d\}\; \backslash ell\; \backslash right)$
• Matérn: $K\_\backslash operatorname\{Matern\}(x,x\text{'})\; =\; \backslash tfrac\{2^\{1-\backslash nu\}\}\{\backslash Gamma(\backslash nu)\}\; \backslash left(\backslash tfrac\{\backslash sqrt\{2\backslash nu\}d\}\{\backslash ell\}\; \backslash right)^\backslash nu\; K\_\backslash nu\; \backslash left(\backslash tfrac\{\backslash sqrt\{2\backslash nu\}d\}\{\backslash ell\}\; \backslash right)$
• Periodic: $K\_\backslash operatorname\{P\}(x,x\text{'})\; =\; \backslash exp\backslash left(-\backslash tfrac\{2\}\{\backslash ell^2\}\; \backslash sin^2\; (d/2)\; \backslash right)$
• Rational quadratic: $K\_\backslash operatorname\{RQ\}(x,x\text{'})\; =\; \backslash left(1+d^2\backslash right)^\{-\backslash alpha\},\; \backslash quad\; \backslash alpha\; \backslash geq\; 0$

Here $d\; =\; |x-\; x\text{'}|$. The parameter $\backslash ell$ is the characteristic length-scale of the process (practically, "how close" two points $x$ and $x\text{'}$ have to be to influence each other significantly), $\backslash delta$ is the Kronecker delta and $\backslash sigma$ the standard deviation of the noise fluctuations. Moreover, $K\_\backslash nu$ is the modified Bessel function of order $\backslash nu$ and $\backslash Gamma(\backslash nu)$ is the gamma function evaluated at $\backslash nu$. Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the data-set at hand.

The inferential results are dependent on the values of the hyperparameters $\backslash theta$ (e.g. $\backslash ell$ and $\backslash sigma$) defining the model's behaviour. A popular choice for $\backslash theta$ is to provide maximum a posteriori (MAP) estimates of it with some chosen prior. If the prior is very near uniform, this is the same as maximizing the marginal likelihood of the process; the marginalization being done over the observed process values $y$. This approach is also known as maximum likelihood II, evidence maximization, or empirical Bayes.{{cite journal |last1= Seeger| first1= Matthias |year= 2004 |title= Gaussian Processes for Machine Learning|journal= International Journal of Neural Systems|volume= 14|issue= 2|pages= 69–104 |doi=10.1142/s0129065704001899 | pmid= 15112367 | citeseerx= 10.1.1.71.1079 | s2cid= 52807317 }}

For a Gaussian process, continuity in probability is equivalent to mean-square continuity{{cite book |chapter-url=https://www.mathunion.org/fileadmin/ICM/Proceedings/ICM1974.2/ICM1974.2.ocr.pdf |title=Proceedings of the International Congress of Mathematicians |first=R. M. |last=Dudley |author-link=Richard M. Dudley |year=1975 |volume=2 |pages=143–146 |chapter=The Gaussian process and how to approach it}}{{rp|145}}{{rp|91 "Gaussian processes are discontinuous at fixed points."}}{{cite journal | first1 = Sudipto | last1 = Banerjee | first2 = Alan E. | last2 = Gelfand | title = On smoothness properties of spatial processes | journal = Journal of Multivariate Analysis | year = 2003 | volume = 84 | issue = 1 | pages = 85–100 | doi = 10.1016/S0047-259X(02)00016-7 | url = https://doi.org/10.1016/S0047-259X(02)00016-7}}

and continuity with probability one is equivalent to sample continuity.{{cite book |first=R. M. |last=Dudley |title=Selected Works of R.M. Dudley |chapter=Sample Functions of the Gaussian Process |date=2010 |author-link=Richard M. Dudley |journal=Annals of Probability |volume=1 |issue=1 |pages=66–103 |doi=10.1007/978-1-4419-5821-1_13 |isbn=978-1-4419-5820-4 |chapter-url=http://projecteuclid.org/euclid.aop/1176997026 }}

The latter implies, but is not implied by, continuity in probability.

Continuity in probability holds if and only if the mean and autocovariance are continuous functions. In contrast, sample continuity was challenging even for stationary Gaussian processes (as probably noted first by Andrey Kolmogorov), and more challenging for more general processes.{{cite book |last=Talagrand |first=Michel |author-link=Michel Talagrand |title=Upper and lower bounds for stochastic processes: modern methods and classical problems |year=2014 |publisher=Springer, Heidelberg |isbn=978-3-642-54074-5 |url=https://www.springer.com/gp/book/9783642540745|series=Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge / A Series of Modern Surveys in Mathematics }}{{rp|Sect. 2.8}}

{{citation

| last = Ledoux | first = Michel

| editor1-last = Dobrushin | editor1-first = Roland

| editor2-last = Groeneboom | editor2-first = Piet

| editor3-last = Ledoux | editor3-first = Michel

| contribution = Isoperimetry and Gaussian analysis

| doi = 10.1007/BFb0095676

| location = Berlin

| mr = 1600888

| pages = 165–294

| publisher = Springer

| series = Lecture Notes in Mathematics

| title = Lectures on Probability Theory and Statistics: Ecole d'Eté de Probabilités de Saint-Flour XXIV–1994

| volume = 1648

| isbn = 978-3-540-62055-6

| year = 1996}}{{rp|69,81}}

{{cite book

| last = Adler | first = Robert J.

| title = An Introduction to Continuity, Extrema, and Related Topics for General Gaussian Processes

| journal = Lecture Notes-Monograph Series

| isbn = 0-940600-17-X

| jstor = 4355563

| location = Hayward, California

| mr = 1088478

| publisher = Institute of Mathematical Statistics

| volume = 12

| year = 1990}}{{rp|80}}

{{cite journal |title=Review of: Adler 1990 'An introduction to continuity...' |first=Simeon M. |last=Berman |date=1992 |journal=Mathematical Reviews|mr = 1088478}}

As usual, by a sample continuous process one means a process that admits a sample continuous modification.

{{cite journal |first=R. M. |last=Dudley |author-link=Richard M. Dudley |title=The sizes of compact subsets of Hilbert space and continuity of Gaussian processes |journal=Journal of Functional Analysis |volume=1 |issue= 3|pages=290–330 |year=1967|doi=10.1016/0022-1236(67)90017-1 |doi-access=free }}{{rp|292}}

{{cite book |chapter-url=https://projecteuclid.org/euclid.bsmsp/1200514231 |title=Proceedings of the sixth Berkeley symposium on mathematical statistics and probability, vol. II: probability theory |first1=M.B. |last1=Marcus |first2=Lawrence A. |last2=Shepp |author-link2=Lawrence Shepp |year=1972 |publisher=Univ. California, Berkeley |pages=423–441 |chapter=Sample behavior of Gaussian processes|volume=6 |issue=2 }}{{rp|424}}

For a stationary Gaussian process $X=(X\_t)\_\{t\backslash in\backslash R\},$ some conditions on its spectrum are sufficient for sample continuity, but fail to be necessary. A necessary and sufficient condition, sometimes called Dudley–Fernique theorem, involves the function $\backslash sigma$ defined by

$$\backslash sigma(h)\; =\; \backslash sqrt\{\; \{\backslash mathbb\; E\}\; \backslash big[\; X(t+h)\; -\; X(t)\; \backslash big]^2\; \}$$

(the right-hand side does not depend on $t$ due to stationarity). Continuity of $X$ in probability is equivalent to continuity of $\backslash sigma$ at $0.$ When convergence of $\backslash sigma(h)$ to $0$ (as $h\backslash to\; 0$) is too slow, sample continuity of $X$ may fail. Convergence of the following integrals matters:

$$I(\backslash sigma)\; =\; \backslash int\_0^1\; \backslash frac\{\; \backslash sigma(h)\; \}\{\; h\; \backslash sqrt\{\; \backslash log(1/h)\; \}\; \}\; \backslash ,\; dh\; =\; \backslash int\_0^\backslash infty\; 2\backslash sigma(\; e^\{-x^2\})\; \backslash ,\; dx\; ,$$

these two integrals being equal according to integration by substitution $h\; =\; e^\{-x^2\},$ $x\; =\; \backslash sqrt\{\backslash log(1/h)\}\; .$ The first integrand need not be bounded as $h\backslash to\; 0+,$ thus the integral may converge () or diverge ($I(\backslash sigma)=\backslash infty$). Taking for example $\backslash sigma(\; e^\{-x^2\})\; =\; \backslash tfrac\{1\}\{x^a\}$ for large $x,$ that is, $\backslash sigma(h)\; =\; (\backslash log(1/h))^\{-a/2\}$ for small $h,$ one obtains when $a>1,$ and $I(\backslash sigma)=\backslash infty$ when

In these two cases the function $\backslash sigma$ is increasing on $[0,\backslash infty),$ but generally it is not. Moreover, the condition

{{block indent | em = 1.5 | text ={{vanchor|(*)}}   there exists $\backslash varepsilon\; >\; 0$ such that $\backslash sigma$ is monotone on $[0,\backslash varepsilon]$}}

does not follow from continuity of $\backslash sigma$ and the evident relations $\backslash sigma(h)\; \backslash ge\; 0$ (for all $h$) and $\backslash sigma(0)\; =\; 0.$

{{math theorem | name = Theorem 1 | math_statement = Let $\backslash sigma$ be continuous and satisfy (*). Then the condition is necessary and sufficient for sample continuity of $X.$}}

Some history.{{rp|424}}

Sufficiency was announced by Xavier Fernique in 1964, but the first proof was published by Richard M. Dudley in 1967.{{rp|Theorem 7.1}}

Necessity was proved by Michael B. Marcus and Lawrence Shepp in 1970.{{cite journal |first1=Michael B. |last1=Marcus |first2=Lawrence A. |last2=Shepp |author-link2=Lawrence Shepp |title=Continuity of Gaussian processes |journal=Transactions of the American Mathematical Society |volume=151 |issue= 2|pages=377–391 |year=1970 |doi=10.1090/s0002-9947-1970-0264749-1 |jstor=1995502 |doi-access=free }}{{rp|380}}

There exist sample continuous processes $X$ such that $I(\backslash sigma)=\backslash infty;$ they violate condition (*). An example found by Marcus and Shepp {{rp|387}} is a random lacunary Fourier series

$$X\_t\; =\; \backslash sum\_\{n=1\}^\backslash infty\; c\_n\; (\; \backslash xi\_n\; \backslash cos\; \backslash lambda\_n\; t\; +\; \backslash eta\_n\; \backslash sin\; \backslash lambda\_n\; t\; )\; ,$$

where $\backslash xi\_1,\backslash eta\_1,\backslash xi\_2,\backslash eta\_2,\backslash dots$ are independent random variables with standard normal distribution; frequencies are a fast growing sequence; and coefficients $c\_n>0$ satisfy The latter relation implies

whence almost surely, which ensures uniform convergence of the Fourier series almost surely, and sample continuity of $X.$

File:Autocorrelation of a random lacunary Fourier series.svg

Its autocovariation function

$$\{\backslash mathbb\; E\}[X\_t\; X\_\{t+h\}]\; =\; \backslash sum\_\{n=1\}^\backslash infty\; c\_n^2\; \backslash cos\; \backslash lambda\_n\; h$$

is nowhere monotone (see the picture), as well as the corresponding function $\backslash sigma,$

$$\backslash sigma(h)\; =\; \backslash sqrt\{\; 2\; \{\backslash mathbb\; E\}[X\_t\; X\_t]\; -\; 2\; \{\backslash mathbb\; E\}[X\_t\; X\_\{t+h\}]\; \}\; =\; 2\; \backslash sqrt\{\; \backslash sum\_\{n=1\}^\backslash infty\; c\_n^2\; \backslash sin^2\; \backslash frac\{\backslash lambda\_n\; h\}2\; \}\; .$$

A Wiener process (also known as Brownian motion) is the integral of a white noise generalized Gaussian process. It is not stationary, but it has stationary increments.

The Ornstein–Uhlenbeck process is a stationary Gaussian process.

The Brownian bridge is (like the Ornstein–Uhlenbeck process) an example of a Gaussian process whose increments are not independent.

The fractional Brownian motion is a Gaussian process whose covariance function is a generalisation of that of the Wiener process.

Let $f$ be a mean-zero Gaussian process $\backslash left\backslash \{X\_t\; ;\; t\backslash in\; T\backslash right\backslash \}$ with a non-negative definite covariance function $K$ and let $R$ be a symmetric and positive semidefinite function. Then, there exists a Gaussian process $X$ which has the covariance $R$. Moreover, the reproducing kernel Hilbert space (RKHS) associated to $R$ coincides with the Cameron–Martin theorem associated space $R(H)$ of $X$, and all the spaces $R(H)$, $H\_X$, and $\backslash mathcal\{H\}(K)$ are isometric.{{cite journal|last1=Azmoodeh|first1= Ehsan | last2=Sottinen | first2= Tommi | last3= Viitasaari | first3 =Lauri |last4= Yazigi| first4= Adil |title=Necessary and sufficient conditions for Hölder continuity of Gaussian processes|journal= Statistics & Probability Letters | volume=94 |year=2014|pages=230–235|doi=10.1016/j.spl.2014.07.030|arxiv=1403.2215 }} From now on, let $\backslash mathcal\{H\}(R)$ be a reproducing kernel Hilbert space with positive definite kernel $R$.

Driscoll's zero-one law is a result characterizing the sample functions generated by a Gaussian process:

where $K\_n$ and $R\_n$ are the covariance matrices of all possible pairs of $n$ points, implies

$$\backslash Pr[f\; \backslash in\; \backslash mathcal\{H\}(R)]\; =\; 1.$$

Moreover,

$$\backslash lim\_\{n\backslash to\backslash infty\}\; \backslash operatorname\{tr\}[K\_n\; R\_n^\{-1\}]\; =\; \backslash infty$$

implies {{cite journal|last1=Driscoll|first1=Michael F.|title=The reproducing kernel Hilbert space structure of the sample paths of a Gaussian process|journal=Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete | volume=26|issue=4|year=1973|pages=309–316|issn=0044-3719|doi=10.1007/BF00534894|s2cid=123348980|doi-access=free}}

$$\backslash Pr[f\; \backslash in\; \backslash mathcal\{H\}(R)]\; =\; 0.$$

This has significant implications when $K\; =\; R$, as

$$\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash operatorname\{tr\}[R\_n\; R\_n^\{-1\}]\; =\; \backslash lim\_\{n\backslash to\backslash infty\}\backslash operatorname\{tr\}[I]\; =\; \backslash lim\_\{n\; \backslash to\; \backslash infty\}\; n\; =\; \backslash infty.$$

As such, almost all sample paths of a mean-zero Gaussian process with positive definite kernel $K$ will lie outside of the Hilbert space $\backslash mathcal\{H\}(K)$.

For many applications of interest some pre-existing knowledge about the system at hand is already given. Consider e.g. the case where the output of the Gaussian process corresponds to a magnetic field; here, the real magnetic field is bound by Maxwell's equations and a way to incorporate this constraint into the Gaussian process formalism would be desirable as this would likely improve the accuracy of the algorithm.

A method on how to incorporate linear constraints into Gaussian processes already exists:{{cite arXiv| last1=Jidling|first1=Carl| last2=Wahlström|first2=Niklas|last3=Wills|first3=Adrian|last4=Schön|first4=Thomas B.| date=2017-09-19| title=Linearly constrained Gaussian processes|class=stat.ML|eprint=1703.00787}}

Consider the (vector valued) output function $f(x)$ which is known to obey the linear constraint (i.e. $\backslash mathcal\{F\}\_X$ is a linear operator)

$$\backslash mathcal\{F\}\_X(f(x))\; =\; 0.$$

Then the constraint $\backslash mathcal\{F\}\_X$ can be fulfilled by choosing $f(x)\; =\; \backslash mathcal\{G\}\_X(g(x))$, where $g(x)\; \backslash sim\; \backslash mathcal\{GP\}(\backslash mu\_g,\; K\_g)$ is modelled as a Gaussian process, and finding $\backslash mathcal\{G\}\_X$ such that

$$\backslash mathcal\{F\}\_X(\backslash mathcal\{G\}\_X(g))\; =\; 0\; \backslash qquad\; \backslash forall\; g.$$

Given $\backslash mathcal\{G\}\_X$ and using the fact that Gaussian processes are closed under linear transformations, the Gaussian process for $f$ obeying constraint $\backslash mathcal\{F\}\_X$ becomes

$$f(x)\; =\; \backslash mathcal\{G\}\_X\; g\; \backslash sim\; \backslash mathcal\{GP\}\; (\; \backslash mathcal\{G\}\_X\; \backslash mu\_g,\; \backslash mathcal\{G\}\_X\; K\_g\; \backslash mathcal\{G\}\_\{X\text{'}\}^\backslash mathsf\{T\}\; ).$$

Hence, linear constraints can be encoded into the mean and covariance function of a Gaussian process.

Image:Regressions sine demo.svg also has similar [http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_compare_gpr_krr.html examples].]]

A Gaussian process can be used as a prior probability distribution over functions in Bayesian inference.{{cite book |last=Liu |first=W. |author2=Principe, J.C. |author3=Haykin, S. |title=Kernel Adaptive Filtering: A Comprehensive Introduction |url=http://www.cnel.ufl.edu/~weifeng/publication.htm |year=2010 |publisher=John Wiley |isbn=978-0-470-44753-6 |access-date=2010-03-26 |archive-url=https://web.archive.org/web/20160304042652/http://www.cnel.ufl.edu/~weifeng/publication.htm |archive-date=2016-03-04 |url-status=dead }} Given any set of N points in the desired domain of your functions, take a multivariate Gaussian whose covariance matrix parameter is the Gram matrix of your N points with some desired kernel, and sample from that Gaussian. For solution of the multi-output prediction problem, Gaussian process regression for vector-valued function was developed. In this method, a 'big' covariance is constructed, which describes the correlations between all the input and output variables taken in N points in the desired domain.{{cite journal |last1= Álvarez|first1= Mauricio A.|last2= Rosasco | first2= Lorenzo |last3= Lawrence|first3=Neil D.|year= 2012 |title= Kernels for vector-valued functions: A review|journal= Foundations and Trends in Machine Learning|volume= 4|issue= 3|pages= 195–266 |doi=10.1561/2200000036|s2cid= 456491|url= http://eprints.whiterose.ac.uk/114503/1/1106.6251v2.pdf}} This approach was elaborated in detail for the matrix-valued Gaussian processes and generalised to processes with 'heavier tails' like Student-t processes.{{cite journal |last1= Chen| first1= Zexun |last2= Wang| first2= Bo|last3= Gorban|first3=Alexander N.|year= 2019 |title= Multivariate Gaussian and Student-t process regression for multi-output prediction|journal= Neural Computing and Applications|volume=32|issue=8|pages= 3005–3028 | doi=10.1007/s00521-019-04687-8|doi-access= free| arxiv= 1703.04455 }}

Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging.{{cite book |last=Stein |first=M.L. |title=Interpolation of Spatial Data: Some Theory for Kriging |year=1999 |publisher = Springer}} Gaussian processes are thus useful as a powerful non-linear multivariate interpolation tool. Kriging is also used to extend Gaussian process in the case of mixed integer inputs.{{Cite journal | doi=10.1016/j.neucom.2023.126472 | title=A mixed-categorical correlation kernel for Gaussian process| journal=Neurocomputing| volume=550| pages=126472| year=2023| last1=Saves| first1=Paul | last2=Diouane| first2=Youssef | last3=Bartoli| first3=Nathalie | last4=Lefebvre| first4=Thierry | last5=Morlier| first5=Joseph | arxiv=2211.08262}}

Gaussian processes are also commonly used to tackle numerical analysis problems such as numerical integration, solving differential equations, or optimisation in the field of probabilistic numerics.

Gaussian processes can also be used in the context of mixture of experts models, for example.{{Cite journal |doi = 10.1109/TPAMI.2013.183|pmid = 26353224|title = Gaussian Process-Mixture Conditional Heteroscedasticity|journal = IEEE Transactions on Pattern Analysis and Machine Intelligence|volume = 36|issue = 5|pages = 888–900|year = 2014|last1 = Platanios |first1 = Emmanouil A.|last2 = Chatzis|first2 = Sotirios P.|s2cid = 10424638}}{{Cite journal | doi=10.1016/j.neucom.2013.04.029| title=A latent variable Gaussian process model with Pitman–Yor process priors for multiclass classification| journal=Neurocomputing| volume=120| pages=482–489| year=2013| last1=Chatzis| first1=Sotirios P.}} The underlying rationale of such a learning framework consists in the assumption that a given mapping cannot be well captured by a single Gaussian process model. Instead, the observation space is divided into subsets, each of which is characterized by a different mapping function; each of these is learned via a different Gaussian process component in the postulated mixture.

In the natural sciences, Gaussian processes have found use as probabilistic models of astronomical time series and as predictors of molecular properties.{{Cite thesis |doi = 10.17863/CAM.93643|title = Applications of Gaussian Processes at Extreme Lengthscales: From Molecules to Black Holes|degree = PhD| publisher = University of Cambridge|year = 2022 |first=Ryan-Rhys |last=Griffiths| arxiv=2303.14291 }} They are also being increasingly used as surrogate models for force field optimization.{{cite journal |last1=Shanks |first1=B. L. |last2=Sullivan |first2=H. W. |last3=Shazed |first3=A. R. |last4=Hoepfner |first4=M. P. |title=Accelerated Bayesian Inference for Molecular Simulations using Local Gaussian Process Surrogate Models |journal=Journal of Chemical Theory and Computation |date=2024 |volume=20 |issue=9 |pages=3798–3808 |doi=10.1021/acs.jctc.3c01358 |pmid=38551198 |url=https://pubs.acs.org/doi/full/10.1021/acs.jctc.3c01358|arxiv=2310.19108 }}

{{further|Kriging}}

File:Gaussian Process Regression.png

When concerned with a general Gaussian process regression problem (Kriging), it is assumed that for a Gaussian process $f$ observed at coordinates $x$, the vector of values {{tmath|f(x)}} is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates {{tmath|n}}. Therefore, under the assumption of a zero-mean distribution, {{tmath|f (x') \sim N (0, K(\theta,x,x'))}}, where {{tmath|K(\theta,x,x')}} is the covariance matrix between all possible pairs {{tmath|(x,x')}} for a given set of hyperparameters θ.

As such the log marginal likelihood is:

$$\backslash log\; p(f(x\text{'})\backslash mid\backslash theta,x)\; =\; -\backslash frac\{1\}\{2\}\; \backslash left(f(x)^\backslash mathsf\{T\}\; K(\backslash theta,x,x\text{'})^\{-1\}\; f(x\text{'})\; +\; \backslash log\; \backslash det(K(\backslash theta,x,x\text{'}))\; +\; n\; \backslash log\; 2\backslash pi\; \backslash right)$$

and maximizing this marginal likelihood towards {{mvar|θ}} provides the complete specification of the Gaussian process {{math|f}}. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified {{mvar|θ}}, making predictions about unobserved values {{tmath|f(x^*)}} at coordinates {{math|x*}} is then only a matter of drawing samples from the predictive distribution $p(y^*\backslash mid\; x^*,f(x),x)\; =\; N(y^*\backslash mid\; A,B)$ where the posterior mean estimate {{mvar|A}} is defined as

$$A\; =\; K(\backslash theta,x^*,x)\; K(\backslash theta,x,x\text{'})^\{-1\}\; f(x)$$

and the posterior variance estimate B is defined as:

$$B\; =\; K(\backslash theta,x^*,x^*)\; -\; K(\backslash theta,x^*,x)\; K(\backslash theta,x,x\text{'})^\{-1\}\; K(\backslash theta,x^*,x)^\backslash mathsf\{T\}$$

where {{tmath|K(\theta,x^*,x)}} is the covariance between the new coordinate of estimation x* and all other observed coordinates x for a given hyperparameter vector {{mvar|θ}}, {{tmath|K(\theta,x,x')}} and {{tmath|f(x)}} are defined as before and {{tmath|K(\theta,x^*,x^*)}} is the variance at point {{math|x*}} as dictated by {{mvar|θ}}. It is important to note that practically the posterior mean estimate of {{tmath|f(x^*)}} (the "point estimate") is just a linear combination of the observations {{tmath|f(x)}}; in a similar manner the variance of {{tmath|f(x^*)}} is actually independent of the observations {{tmath|f(x)}}. A known bottleneck in Gaussian process prediction is that the computational complexity of inference and likelihood evaluation is cubic in the number of points |x|, and as such can become unfeasible for larger data sets.{{Cite journal |last1 = Banerjee| first1 = Sudipto | title= High-dimensional Bayesian Geostatistics |journal= Bayesian Analysis | year = 2017 | volume = 12 | issue = 2 | pages=583–614| doi= 10.1214/17-BA1056R | url=https://doi.org/10.1214/17-BA1056R | pmid = 29391920 | pmc = 5790125 }} Works on sparse Gaussian processes, that usually are based on the idea of building a representative set for the given process f, try to circumvent this issue. {{cite journal |last1= Smola| first1= A.J.| last2=Schoellkopf | first2= B. |year= 2000 |title= Sparse greedy matrix approximation for machine learning |journal= Proceedings of the Seventeenth International Conference on Machine Learning| pages=911–918| citeseerx= 10.1.1.43.3153}}{{cite journal |last1= Csato| first1=L.| last2=Opper | first2= M. |year= 2002 |title= Sparse on-line Gaussian processes |journal= Neural Computation |number=3| volume= 14 | pages=641–668 | doi=10.1162/089976602317250933| pmid=11860686| citeseerx=10.1.1.335.9713| s2cid=11375333}}{{Cite journal |last1 = Banerjee| first1 = Sudipto | last2=Gelfand | first2 = Alan E.| last3 = Finley | first3 = Andrew O. | last4 = Sang | first4 = Huiyan | title= Gaussian Predictive Process Models for large spatial datasets |journal= Journal of the Royal Statistical Society, Series B (Statistical Methodology) | year = 2008 | volume = 70 | issue = 4 | pages=825–848| doi=10.1111/j.1467-9868.2008.00663.x | url=https://doi.org/10.1111/j.1467-9868.2008.00663.x | pmid = 19750209 | pmc = 2741335}} The kriging method can be used in the latent level of a nonlinear mixed-effects model for a spatial functional prediction: this technique is called the latent kriging.{{Cite journal |last1=Lee|first1=Se Yoon |first2=Bani|last2=Mallick| title = Bayesian Hierarchical Modeling: Application Towards Production Results in the Eagle Ford Shale of South Texas|journal=Sankhya B|year=2021|volume=84 |pages=1–43 |doi=10.1007/s13571-020-00245-8|doi-access=free}} Other classes of scalable Gaussian process for analyzing massive datasets have emerged from the Vecchia approximation and Nearest Neighbor Gaussian Processes (NNGP).{{cite journal|last1=Datta|first1=Abhirup|last2=Banerjee|first2=Sudipto|last3=Finley|first3=Andrew|last4=Gelfand|first4=Alan|title=Hierarchical Nearest-Neighbor Gaussian Process Models for Large Spatial Data|journal=Journal of the American Statistical Association|year=2016|volume=111|issue=514|pages=800–812|doi=10.1080/01621459.2015.1044091|pmid=29720777 |pmc=5927603 }}

Often, the covariance has the form $K(\backslash theta,\; x,x\text{'})\; =\; \backslash frac\{1\}\{\backslash sigma^2\}\; \backslash tilde\{K\}(\backslash theta,x,x\text{'})$, where $\backslash sigma^2$ is a scaling parameter. Examples are the Matérn class covariance functions. If this scaling parameter $\backslash sigma^2$ is either known or unknown (i.e. must be marginalized), then the posterior probability, $p(\backslash theta\; \backslash mid\; D)$, i.e. the probability for the hyperparameters $\backslash theta$ given a set of data pairs $D$ of observations of $x$ and $f(x)$, admits an analytical expression.{{Cite journal| last1=Ranftl|first1=Sascha|last2=Melito|first2=Gian Marco|last3=Badeli|first3=Vahid|last4=Reinbacher-Köstinger|first4=Alice| last5=Ellermann|first5=Katrin|last6=von der Linden|first6=Wolfgang|date=2019-12-31|title=Bayesian Uncertainty Quantification with Multi-Fidelity Data and Gaussian Processes for Impedance Cardiography of Aortic Dissection|journal=Entropy| volume=22|issue=1| pages=58|doi=10.3390/e22010058|issn=1099-4300|pmc=7516489|pmid=33285833|bibcode=2019Entrp..22...58R |doi-access=free}}

{{further|Neural network Gaussian process}}

Bayesian neural networks are a particular type of Bayesian network that results from treating deep learning and artificial neural network models probabilistically, and assigning a prior distribution to their parameters. Computation in artificial neural networks is usually organized into sequential layers of artificial neurons. The number of neurons in a layer is called the layer width. As layer width grows large, many Bayesian neural networks reduce to a Gaussian process with a closed form compositional kernel. This Gaussian process is called the Neural Network Gaussian Process (NNGP) (not to be confused with the Nearest Neighbor Gaussian Process ).{{cite journal |last1=Novak |first1=Roman |last2=Xiao |first2=Lechao |last3=Hron |first3=Jiri |last4=Lee |first4=Jaehoon |last5=Alemi |first5=Alexander A. |last6=Sohl-Dickstein |first6=Jascha |last7=Schoenholz |first7=Samuel S. |title=Neural Tangents: Fast and Easy Infinite Neural Networks in Python |journal=International Conference on Learning Representations |date=2020|arxiv=1912.02803 }}{{Cite book|last=Neal|first=Radford M.|title=Bayesian Learning for Neural Networks|publisher=Springer Science and Business Media| year=2012}} It allows predictions from Bayesian neural networks to be more efficiently evaluated, and provides an analytic tool to understand deep learning models.

{{See also|Gaussian process approximations}}

In practical applications, Gaussian process models are often evaluated on a grid leading to multivariate normal distributions. Using these models for prediction or parameter estimation using maximum likelihood requires evaluating a multivariate Gaussian density, which involves calculating the determinant and the inverse of the covariance matrix. Both of these operations have cubic computational complexity which means that even for grids of modest sizes, both operations can have a prohibitive computational cost. This drawback led to the development of multiple approximation methods.

• Bayes linear statistics
• Bayesian interpretation of regularization
• Kriging
• Gaussian free field
• Gauss–Markov process
• Gradient-enhanced kriging (GEK)
• Student's t-process

{{Reflist}}

{{wikibooks|Gaussian process}}

• [https://gaussianprocess.org The Gaussian Processes Web Site, including the text of Rasmussen and Williams' Gaussian Processes for Machine Learning]
• {{cite arXiv|eprint=1505.02965 |last1=Ebden |first1=Mark |title=Gaussian Processes: A Quick Introduction |year=2015 |class=math.ST }}
• [http://publications.nr.no/917_Rapport.pdf A Review of Gaussian Random Fields and Correlation Functions]
• [https://web.archive.org/web/20180826005000/https://pdfs.semanticscholar.org/c9f2/1b84149991f4d547b3f0f625f710750ad8d9.pdf Efficient Reinforcement Learning using Gaussian Processes]

{{further|Comparison of Gaussian process software}}

• [http://www.gaussianprocess.org/gpml/code/matlab/doc/ GPML: A comprehensive Matlab toolbox for GP regression and classification]
• [http://sourceforge.net/projects/kriging STK: a Small (Matlab/Octave) Toolbox for Kriging and GP modeling]
• [http://www.uqlab.com/ Kriging module in UQLab framework (Matlab)]
• [http://codes.arizona.edu/toolbox/ CODES Toolbox: implementations of Kriging, variational kriging and multi-fidelity models (Matlab)]
• [http://au.mathworks.com/matlabcentral/fileexchange/38880: Matlab/Octave function for stationary Gaussian fields]
• [https://github.com/Yelp/MOE Yelp MOE – A black box optimization engine using Gaussian process learning]
• [http://www.sumo.intec.ugent.be/ooDACE ooDACE] {{Webarchive|url=https://web.archive.org/web/20200809021046/http://sumo.intec.ugent.be/ooDACE |date=2020-08-09 }} – A flexible object-oriented Kriging Matlab toolbox.
• [https://web.archive.org/web/20141009045756/http://becs.aalto.fi/en/research/bayes/gpstuff/ GPstuff – Gaussian process toolbox for Matlab and Octave]
• [https://github.com/SheffieldML/GPy GPy – A Gaussian processes framework in Python]
• [https://github.com/GeoStat-Framework/GSTools GSTools - A geostatistical toolbox, including Gaussian process regression, written in Python]
• [http://www.tmpl.fi/gp/ Interactive Gaussian process regression demo]
• [https://github.com/ChristophJud/GPR Basic Gaussian process library written in C++11]
• [http://scikit-learn.org scikit-learn] – A machine learning library for Python which includes Gaussian process regression and classification
• [https://sambo-optimization.github.io SAMBO Optimization] library for Python supports sequential optimization driven by Gaussian process regressor from scikit-learn.
• [https://github.com/modsim/KriKit] - The Kriging toolKit (KriKit) is developed at the Institute of Bio- and Geosciences 1 (IBG-1) of Forschungszentrum Jülich (FZJ)

• [http://videolectures.net/gpip06_mackay_gpb Gaussian Process Basics by David MacKay]
• [http://videolectures.net/epsrcws08_rasmussen_lgp Learning with Gaussian Processes by Carl Edward Rasmussen]
• [http://videolectures.net/mlss07_rasmussen_bigp Bayesian inference and Gaussian processes by Carl Edward Rasmussen]

{{Stochastic processes}}

{{Authority control}}

{{DEFAULTSORT:Gaussian Process}}

Category:Stochastic processes

Category:Kernel methods for machine learning

Category:Nonparametric Bayesian statistics

Category:Normal distribution