Reproducing kernel Hilbert space

Let $X$ be an arbitrary set and $H$ a Hilbert space of real-valued functions on $X$, equipped with pointwise addition and pointwise scalar multiplication. The evaluation functional over the Hilbert space of functions $H$ is a linear functional that evaluates each function at a point $x$,

:$L\_\{x\}\; :\; f\; \backslash mapsto\; f(x)\; \backslash text\{\; \}\; \backslash forall\; f\; \backslash in\; H.$

We say that H is a reproducing kernel Hilbert space if, for all $x$ in $X$, $L\_x$ is continuous at every $f$ in $H$ or, equivalently, if $L\_x$ is a bounded operator on $H$, i.e. there exists some $M\_x>0$ such that

{{NumBlk|:|$|L\_x(f)|\; :=\; |f(x)|\; \backslash le\; M\_x\backslash ,\; \backslash |f\backslash |\_H\; \backslash qquad\; \backslash forall\; f\; \backslash in\; H.\; \backslash ,$|{{EquationRef|1}}}}

Although is assumed for all $x\; \backslash in\; X$, it might still be the case that $\backslash sup\_x\; M\_x\; =\; \backslash infty$.

While property ({{EquationNote|1}}) is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in $H$ at every point in the domain, it does not lend itself to easy application in practice. A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of $f$ with a function $K\_x$ in $H$. This function is the so-called reproducing kernel{{Citation needed|date=September 2022}} for the Hilbert space $H$ from which the RKHS takes its name. More formally, the Riesz representation theorem implies that for all $x$ in $X$ there exists a unique element $K\_x$ of $H$ with the reproducing property,

{{NumBlk|:|$f(x)\; =\; L\_x(f)\; =\; \backslash langle\; f,\backslash \; K\_x\; \backslash rangle\_H\; \backslash quad\; \backslash forall\; f\; \backslash in\; H.$|{{EquationRef|2}}}}

Since $K\_x$ is itself a function defined on $X$ with values in the field $\backslash mathbb\{R\}$ (or $\backslash mathbb\{C\}$ in the case of complex Hilbert spaces) and as $K\_x$ is in $H$ we have that

:$K\_x(y)\; =\; L\_y(K\_x)=\; \backslash langle\; K\_x,\backslash \; K\_y\; \backslash rangle\_H,$

where $K\_y\backslash in\; H$ is the element in $H$ associated to $L\_y$.

This allows us to define the reproducing kernel of $H$ as a function $K:\; X\; \backslash times\; X\; \backslash to\; \backslash mathbb\{R\}$ (or $\backslash mathbb\{C\}$ in the complex case) by

:$K(x,y)\; =\; \backslash langle\; K\_x,\backslash \; K\_y\; \backslash rangle\_H.$

From this definition it is easy to see that $K:\; X\; \backslash times\; X\; \backslash to\; \backslash mathbb\{R\}$ (or $\backslash mathbb\{C\}$ in the complex case) is both symmetric (resp. conjugate symmetric) and positive definite, i.e.

:$\backslash sum\_\{i,j\; =1\}^n\; c\_i\; c\_j\; K(x\_i,\; x\_j)=$

\sum_{i=1}^n c_i \left\langle K_{x_i} , \sum_{j=1}^n c_j K_{x_j} \right\rangle_{H} =

\left\langle \sum_{i=1}^n c_i K_{x_i} , \sum_{j=1}^n c_j K_{x_j} \right\rangle_{H} =

\left\|\sum_{i=1}^nc_iK_{x_i}\right\|_H^2 \ge 0

for every $n\; \backslash in\; \backslash mathbb\{N\},\; x\_1,\; \backslash dots,\; x\_n\; \backslash in\; X,\; \backslash text\{\; and\; \}\; c\_1,\; \backslash dots,\; c\_n\; \backslash in\; \backslash mathbb\{R\}.$Durrett The Moore–Aronszajn theorem (see below) is a sort of converse to this: if a function $K$ satisfies these conditions then there is a Hilbert space of functions on $X$ for which it is a reproducing kernel.

The simplest example of a reproducing kernel Hilbert space is the space $L^2(X,\backslash mu)$ where $X$ is a set and $\backslash mu$ is the counting measure on $X$. For $x\backslash in\; X$, the reproducing kernel $K\_x$ is the indicator function of the one point set $\backslash \{x\backslash \}\backslash subset\; X$.

Nontrivial reproducing kernel Hilbert spaces often involve analytic functions, as we now illustrate by example. Consider the Hilbert space of bandlimited continuous functions $H$. Fix some cutoff frequency

:$H\; =\; \backslash \{\; f\; \backslash in\; L^2(\backslash mathbb\{R\})\; \backslash mid\; \backslash operatorname\{supp\}(F)\; \backslash subset\; [-a,a]\; \backslash \}$

where $L^2(\backslash mathbb\{R\})$ is the set of square integrable functions, and $F(\backslash omega)\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; f(t)\; e^\{-i\backslash omega\; t\}\; \backslash ,\; dt$ is the Fourier transform of $f$. As the inner product, we use

: $\backslash langle\; f,\; g\backslash rangle\_\{L^2\}\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; f(x)\; \backslash cdot\; \backslash overline\{g(x)\}\; \backslash ,\; dx.$

Since this is a closed subspace of $L^2(\backslash mathbb\; R)$, it is a Hilbert space. Moreover, the elements of $H$ are smooth functions on $\backslash mathbb\; R$ that tend to zero at infinity, essentially by the Riemann-Lebesgue lemma. In fact, the elements of $H$ are the restrictions to $\backslash mathbb\; R$ of entire holomorphic functions, by the Paley–Wiener theorem.

From the Fourier inversion theorem, we have

:$f(x)\; =\; \backslash frac\{1\}\{2\; \backslash pi\}\; \backslash int\_\{-a\}^a\; F(\backslash omega)\; e^\{ix\; \backslash omega\}\; \backslash ,\; d\backslash omega\; .$

It then follows by the Cauchy–Schwarz inequality and Plancherel's theorem that, for all $x$,

:$|f(x)|\; \backslash le$

\frac{1}{2 \pi} \sqrt{ 2a\int_{-a}^a |F(\omega)|^2 \, d\omega}

=\frac{ \sqrt{2a} }{2\pi}\sqrt{\int_{-\infty}^\infty |F(\omega)|^2 \, d\omega}

= \sqrt{\frac{a}{\pi}} \|f\|_{L^2}.

This inequality shows that the evaluation functional is bounded, proving that $H$ is indeed a RKHS.

The kernel function $K\_x$ in this case is given by

:$K\_x(y)\; =\; \backslash frac\{a\}\{\backslash pi\}\; \backslash operatorname\{sinc\}\backslash left\; (\; \backslash frac\{a\}\{\backslash pi\}\; (y-x)\; \backslash right\; )=\backslash frac\{\backslash sin(a(y-x))\}\{\backslash pi(y-x)\}.$

The Fourier transform of $K\_x(y)$ defined above is given by

:$\backslash int\_\{-\backslash infty\}^\backslash infty\; K\_x(y)e^\{-i\; \backslash omega\; y\}\; \backslash ,\; dy\; =$

\begin{cases}

e^{-i \omega x} &\text{if } \omega \in [-a, a], \\

0 &\textrm{otherwise},

\end{cases}

which is a consequence of the time-shifting property of the Fourier transform. Consequently, using Plancherel's theorem, we have

:$\backslash langle\; f,\; K\_x\backslash rangle\_\{L^2\}\; =\; \backslash int\_\{-\backslash infty\}^\backslash infty\; f(y)\; \backslash cdot\; \backslash overline\{K\_x(y)\}\; \backslash ,\; dy$

= \frac{1}{2\pi} \int_{-a}^a F(\omega) \cdot e^{i\omega x} \, d\omega = f(x) .

Thus we obtain the reproducing property of the kernel.

$K\_x$ in this case is the "bandlimited version" of the Dirac delta function, and that $K\_x(y)$ converges to $\backslash delta(y-x)$ in the weak sense as the cutoff frequency $a$ tends to infinity.

We have seen how a reproducing kernel Hilbert space defines a reproducing kernel function that is both symmetric and positive definite. The Moore–Aronszajn theorem goes in the other direction; it states that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's Theory of Reproducing Kernels, although he attributes it to E. H. Moore.

:Theorem. Suppose K is a symmetric, positive definite kernel on a set X. Then there is a unique Hilbert space of functions on X for which K is a reproducing kernel.

Proof. For all x in X, define K_x = K(x, ⋅ ). Let H₀ be the linear span of {K_x : x ∈ X}. Define an inner product on H₀ by

:$\backslash left\backslash langle\; \backslash sum\_\{j=1\}^n\; b\_j\; K\_\{y\_j\},\; \backslash sum\_\{i=1\}^m\; a\_i\; K\_\{x\_i\}\; \backslash right\; \backslash rangle\_\{H\_0\}\; =\; \backslash sum\_\{i=1\}^m\; \backslash sum\_\{j=1\}^n\; \{a\_i\}\; b\_j\; K(y\_j,\; x\_i),$

which implies $K(x,y)=\backslash left\backslash langle\; K\_\{x\},\; K\_\{y\}\; \backslash right\backslash rangle\_\{H\_0\}$.

The symmetry of this inner product follows from the symmetry of K and the non-degeneracy follows from the fact that K is positive definite.

Let H be the completion of H₀ with respect to this inner product. Then H consists of functions of the form

:$f(x)\; =\; \backslash sum\_\{i=1\}^\backslash infty\; a\_i\; K\_\{x\_i\}\; (x)\; \backslash quad\; \backslash text\{where\}\; \backslash quad\; \backslash lim\_\{n\; \backslash to\; \backslash infty\}\backslash sup\_\{p\backslash geq0\}\backslash left\backslash |\backslash sum\_\{i=n\}^\{n+p\}\; a\_i\; K\_\{x\_i\}\backslash right\backslash |\_\{H\_0\}\; =\; 0.$

Now we can check the reproducing property ({{EquationNote|2}}):

:$\backslash langle\; f,\; K\_x\; \backslash rangle\_H\; =\; \backslash sum\_\{i=1\}^\backslash infty\; a\_i\backslash left\; \backslash langle\; K\_\{x\_i\},\; K\_x\; \backslash right\; \backslash rangle\_\{H\_0\}=\; \backslash sum\_\{i=1\}^\backslash infty\; a\_i\; K\; (x\_i,\; x)\; =\; f(x).$

To prove uniqueness, let G be another Hilbert space of functions for which K is a reproducing kernel. For every x and y in X, ({{EquationNote|2}}) implies that

:$\backslash langle\; K\_x,\; K\_y\; \backslash rangle\_H\; =\; K(x,\; y)\; =\; \backslash langle\; K\_x,\; K\_y\; \backslash rangle\_G.$

By linearity, $\backslash langle\; \backslash cdot,\; \backslash cdot\; \backslash rangle\_H\; =\; \backslash langle\; \backslash cdot,\; \backslash cdot\; \backslash rangle\_G$ on the span of $\backslash \{K\_x\; :\; x\; \backslash in\; X\backslash \}$. Then $H\; \backslash subset\; G$ because G is complete and contains H₀ and hence contains its completion.

Now we need to prove that every element of G is in H. Let $f$ be an element of G. Since H is a closed subspace of G, we can write $f=f\_H\; +\; f\_\{H^\backslash bot\}$ where $f\_H\; \backslash in\; H$ and $f\_\{H^\backslash bot\}\; \backslash in\; H^\backslash bot$. Now if $x\; \backslash in\; X$ then, since K is a reproducing kernel of G and H:

:$f(x)\; =\; \backslash langle\; K\_x\; ,\; f\; \backslash rangle\_G\; =\; \backslash langle\; K\_x,\; f\_H\; \backslash rangle\_G\; +\; \backslash langle\; K\_x,\; f\_\{H^\backslash bot\}\; \backslash rangle\_G\; =\; \backslash langle\; K\_x\; ,\; f\_H\; \backslash rangle\_G\; =\; \backslash langle\; K\_x\; ,\; f\_H\; \backslash rangle\_H\; =\; f\_H(x),$

where we have used the fact that $K\_x$ belongs to H so that its inner product with $f\_\{H^\backslash bot\}$ in G is zero.

This shows that $f\; =\; f\_H$ in G and concludes the proof.

We may characterize a symmetric positive definite kernel $K$ via the integral operator using Mercer's theorem and obtain an additional view of the RKHS. Let $X$ be a compact space equipped with a strictly positive finite Borel measure $\backslash mu$ and $K:\; X\; \backslash times\; X\; \backslash to\; \backslash R$ a continuous, symmetric, and positive definite function. Define the integral operator $T\_K:\; L\_2(X)\; \backslash to\; L\_2(X)$ as

:$[T\_K\; f](\backslash cdot)\; =\backslash int\_X\; K(\{\}\backslash cdot\{\},t)\; f(t)\backslash ,\; d\backslash mu(t)$

where $L\_2(X)$ is the space of square integrable functions with respect to $\backslash mu$.

Mercer's theorem states that the spectral decomposition of the integral operator $T\_K$ of $K$ yields a series representation of $K$ in terms of the eigenvalues and eigenfunctions of $T\_K$. This then implies that $K$ is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions. We provide the details below.

Under these assumptions $T\_K$ is a compact, continuous, self-adjoint, and positive operator. The spectral theorem for self-adjoint operators implies that there is an at most countable decreasing sequence $(\backslash sigma\_i)\_\{i\; \backslash geq\; 0\}$ such that $\backslash lim\_\{i\; \backslash to\; \backslash infty\}\backslash sigma\_i\; =\; 0$ and

$T\_K\backslash varphi\_i(x)\; =\; \backslash sigma\_i\backslash varphi\_i(x)$, where the $\backslash \{\backslash varphi\_i\backslash \}$ form an orthonormal basis of $L\_2(X)$. By the positivity of $T\_K,\; \backslash sigma\_i\; >\; 0$ for all $i.$ One can also show that $T\_K$ maps continuously into the space of continuous functions $C(X)$ and therefore we may choose continuous functions as the eigenvectors, that is, $\backslash varphi\_i\; \backslash in\; C(X)$ for all $i.$ Then by Mercer's theorem $K$ may be written in terms of the eigenvalues and continuous eigenfunctions as

:$K(x,y)\; =\; \backslash sum\_\{j=1\}^\backslash infty\; \backslash sigma\_j\; \backslash ,\; \backslash varphi\_j(x)\; \backslash ,\; \backslash varphi\_j(y)$

for all $x,\; y\; \backslash in\; X$ such that

:$\backslash lim\_\{n\; \backslash to\; \backslash infty\}\backslash sup\_\{u,v\}\; \backslash left\; |K(u,v)\; -\; \backslash sum\_\{j=1\}^n\; \backslash sigma\_j\; \backslash ,\; \backslash varphi\_j(u)\; \backslash ,\; \backslash varphi\_j(v)\; \backslash right\; |\; =\; 0.$

This above series representation is referred to as a Mercer kernel or Mercer representation of $K$.

Furthermore, it can be shown that the RKHS $H$ of $K$ is given by

:

where the inner product of $H$ given by

:$\backslash left\backslash langle\; f,g\; \backslash right\backslash rangle\_H\; =\; \backslash sum\_\{i=1\}^\backslash infty\; \backslash frac\{\backslash left\backslash langle\; f,\backslash varphi\_i\; \backslash right\backslash rangle\_\{L\_2\}\backslash left\backslash langle\; g,\backslash varphi\_i\; \backslash right\backslash rangle\_\{L\_2\}\}\{\backslash sigma\_i\}.$

This representation of the RKHS has application in probability and statistics, for example to the Karhunen–Loève representation for stochastic processes and kernel PCA.

A feature map is a map $\backslash varphi\backslash colon\; X\; \backslash rightarrow\; F$, where $F$ is a Hilbert space which we will call the feature space. The first sections presented the connection between bounded/continuous evaluation functions, positive definite functions, and integral operators and in this section we provide another representation of the RKHS in terms of feature maps.

Every feature map defines a kernel via

{{NumBlk|:|$K(x,y)\; =\; \backslash langle\; \backslash varphi(x),\; \backslash varphi(y)\; \backslash rangle\_F.$ |{{EquationRef|3}}}}

Clearly $K$ is symmetric and positive definiteness follows from the properties of inner product in $F$. Conversely, every positive definite function and corresponding reproducing kernel Hilbert space has infinitely many associated feature maps such that ({{EquationNote|3}}) holds.

For example, we can trivially take $F\; =\; H$ and $\backslash varphi(x)\; =\; K\_x$ for all $x\; \backslash in\; X$. Then ({{EquationNote|3}}) is satisfied by the reproducing property. Another classical example of a feature map relates to the previous section regarding integral operators by taking $F\; =\; \backslash ell^2$ and $\backslash varphi(x)\; =\; (\backslash sqrt\{\backslash sigma\_i\}\; \backslash varphi\_i(x))\_i$.

This connection between kernels and feature maps provides us with a new way to understand positive definite functions and hence reproducing kernels as inner products in $H$. Moreover, every feature map can naturally define a RKHS by means of the definition of a positive definite function.

Lastly, feature maps allow us to construct function spaces that reveal another perspective on the RKHS. Consider the linear space

:$H\_\backslash varphi\; =\; \backslash \{\; f:\; X\; \backslash to\; \backslash mathbb\{R\}\; \backslash mid\; \backslash exists\; w\; \backslash in\; F,\; f(x)\; =\; \backslash langle\; w,\; \backslash varphi(x)\; \backslash rangle\_\{F\},\; \backslash forall\; \backslash text\{\; \}\; x\; \backslash in\; X\; \backslash \}\; .$

We can define a norm on $H\_\backslash varphi$ by

:$\backslash |f\backslash |\_\backslash varphi\; =\; \backslash inf\; \backslash \{\backslash |w\backslash |\_F\; :\; w\; \backslash in\; F,\; f(x)\; =\; \backslash langle\; w,\; \backslash varphi(x)\backslash rangle\_F,\; \backslash forall\; \backslash text\{\; \}\; x\; \backslash in\; X\; \backslash \}\; .$

It can be shown that $H\_\{\backslash varphi\}$ is a RKHS with kernel defined by $K(x,y)\; =\; \backslash langle\backslash varphi(x),\; \backslash varphi(y)\backslash rangle\_F$. This representation implies that the elements of the RKHS are inner products of elements in the feature space and can accordingly be seen as hyperplanes. This view of the RKHS is related to the kernel trick in machine learning.Rosasco

Useful properties of RKHSs:

• Let $(X\_i)\_\{i=1\}^p$ be a sequence of sets and $(K\_i)\_\{i=1\}^p$ be a collection of corresponding positive definite functions on $(X\_i)\_\{i=1\}^p.$ It then follows that
• ::$K((x\_1,\backslash ldots\; ,x\_p),(y\_1,\backslash ldots,y\_p))\; =\; K\_1(x\_1,y\_1)\backslash cdots\; K\_p(x\_p,y\_p)$
• :is a kernel on $X\; =\; X\_1\; \backslash times\; \backslash dots\; \backslash times\; X\_p.$
• Let $X\_0\; \backslash subset\; X,$ then the restriction of $K$ to $X\_0\; \backslash times\; X\_0$ is also a reproducing kernel.
• Consider a normalized kernel $K$ such that $K(x,\; x)\; =\; 1$ for all $x\; \backslash in\; X$. Define a pseudo-metric on X as
• ::$d\_K(x,y)\; =\; \backslash |K\_x\; -\; K\_y\backslash |\_H^2\; =\; 2(1-K(x,y))\; \backslash qquad\; \backslash forall\; x\; \backslash in\; X\; .$
• :By the Cauchy–Schwarz inequality,
• ::$K(x,y)^2\; \backslash le\; K(x,\; x)K(y,\; y)=1\; \backslash qquad\; \backslash forall\; x,y\; \backslash in\; X.$
• :This inequality allows us to view $K$ as a measure of similarity between inputs. If $x,\; y\; \backslash in\; X$ are similar then $K(x,y)$ will be closer to 1 while if $x,y\; \backslash in\; X$ are dissimilar then $K(x,y)$ will be closer to 0.

• The closure of the span of $\backslash \{\; K\_x\; \backslash mid\; x\; \backslash in\; X\; \backslash \}$ coincides with $H$.Rosasco

:$K(x,y)\; =\; \backslash langle\; x,y\backslash rangle$

The RKHS $H$ corresponding to this kernel is the dual space, consisting of functions $f(x)\; =\; \backslash langle\; x,\backslash beta\backslash rangle$ satisfying $\backslash |f\backslash |\_H^2=\backslash |\backslash beta\backslash |^2$.

:$K(x,y)\; =\; (\backslash alpha\backslash langle\; x,y\; \backslash rangle\; +\; 1)^d,\; \backslash qquad\; \backslash alpha\; \backslash in\; \backslash R,\; d\; \backslash in\; \backslash N$

These are another common class of kernels which satisfy $K(x,y)\; =\; K(\backslash |x\; -\; y\backslash |)$. Some examples include:

• Gaussian or squared exponential kernel:
• ::$K(x,y)\; =\; e^\{-\backslash frac\{\backslash |x\; -\; y\backslash |^2\}\{2\backslash sigma^2\}\},\; \backslash qquad\; \backslash sigma\; >\; 0$
• Laplacian kernel:
• ::$K(x,y)\; =\; e^\{-\backslash frac\{\backslash |x\; -\; y\backslash |\}\{\backslash sigma\}\},\; \backslash qquad\; \backslash sigma\; >\; 0$
• :The squared norm of a function $f$ in the RKHS $H$ with this kernel is:Berlinet, Alain and Thomas, Christine. [https://books.google.com/books?id=bX3TBwAAQBAJ&dq=%22Reproducing+kernel+Hilbert+spaces+in+Probability+and+Statistics%22&pg=PP11 Reproducing kernel Hilbert spaces in Probability and Statistics], Kluwer Academic Publishers, 2004Thomas-Agnan C . Computing a family of reproducing kernels for statistical applications. Numerical Algorithms, 13, pp. 21-32 (1996)
• ::$\backslash |f\backslash |\_H^2=\backslash int\_\{\backslash mathbb\; R\}\backslash Big(\; \backslash frac1\{\backslash sigma\}\; f(x)^2\; +\; \backslash sigma\; f\text{'}(x)^2\backslash Big)\; \backslash mathrm\; d\; x.$

We also provide examples of Bergman kernels. Let X be finite and let H consist of all complex-valued functions on X. Then an element of H can be represented as an array of complex numbers. If the usual inner product is used, then K_x is the function whose value is 1 at x and 0 everywhere else, and $K(x,y)$ can be thought of as an identity matrix since

:

In this case, H is isomorphic to $\backslash Complex^n$.

The case of $X=\; \backslash mathbb\{D\}$ (where $\backslash mathbb\{D\}$ denotes the unit disc) is more sophisticated. Here the Bergman space $A^2(\backslash mathbb\{D\})$ is the space of square-integrable holomorphic functions on $\backslash mathbb\{D\}$. It can be shown that the reproducing kernel for $A^2(\backslash mathbb\{D\})$ is

:$K(x,y)=\backslash frac\{1\}\{\backslash pi\}\backslash frac\{1\}\{(1-x\backslash overline\{y\})^2\}.$

Lastly, the space of band limited functions in $L^2(\backslash R)$ with bandwidth $2a$ is a RKHS with reproducing kernel

:$K(x,y)=\backslash frac\{\backslash sin\; a\; (x\; -\; y)\}\{\backslash pi\; (x-y)\}.$

In this section we extend the definition of the RKHS to spaces of vector-valued functions as this extension is particularly important in multi-task learning and manifold regularization. The main difference is that the reproducing kernel $\backslash Gamma$ is a symmetric function that is now a positive semi-definite matrix for every $x,y$ in $X$. More formally, we define a vector-valued RKHS (vvRKHS) as a Hilbert space of functions $f:\; X\; \backslash to\; \backslash mathbb\{R\}^T$ such that for all $c\; \backslash in\; \backslash mathbb\{R\}^T$ and $x\; \backslash in\; X$

:$\backslash Gamma\_xc(y)\; =\; \backslash Gamma(x,\; y)c\; \backslash in\; H\; \backslash text\{\; for\; \}\; y\; \backslash in\; X$

and

:$\backslash langle\; f,\; \backslash Gamma\_x\; c\; \backslash rangle\_H\; =\; f(x)^\backslash intercal\; c.$

This second property parallels the reproducing property for the scalar-valued case. This definition can also be connected to integral operators, bounded evaluation functions, and feature maps as we saw for the scalar-valued RKHS. We can equivalently define the vvRKHS as a vector-valued Hilbert space with a bounded evaluation functional and show that this implies the existence of a unique reproducing kernel by the Riesz Representation theorem. Mercer's theorem can also be extended to address the vector-valued setting and we can therefore obtain a feature map view of the vvRKHS. Lastly, it can also be shown that the closure of the span of $\backslash \{\; \backslash Gamma\_xc\; :\; x\; \backslash in\; X,\; c\; \backslash in\; \backslash mathbb\{R\}^T\; \backslash \}$ coincides with $H$, another property similar to the scalar-valued case.

We can gain intuition for the vvRKHS by taking a component-wise perspective on these spaces. In particular, we find that every vvRKHS is isometrically isomorphic to a scalar-valued RKHS on a particular input space. Let $\backslash Lambda\; =\; \backslash \{1,\; \backslash dots,\; T\; \backslash \}$. Consider the space $X\; \backslash times\; \backslash Lambda$ and the corresponding reproducing kernel

{{NumBlk|:|$\backslash gamma:\; X\; \backslash times\; \backslash Lambda\; \backslash times\; X\; \backslash times\; \backslash Lambda\; \backslash to\; \backslash mathbb\{R\}.$|{{EquationRef|4}}}}

As noted above, the RKHS associated to this reproducing kernel is given by the closure of the span of $\backslash \{\; \backslash gamma\_\{(x,t)\}\; :\; x\; \backslash in\; X,\; t\; \backslash in\; \backslash Lambda\; \backslash \}$ where

$\backslash gamma\_\{(x,t)\}\; (y,s)\; =\; \backslash gamma(\; (x,t),\; (y,s))$ for every set of pairs {{nowrap|$(x,t),\; (y,s)\; \backslash in\; X\; \backslash times\; \backslash Lambda$.}}

The connection to the scalar-valued RKHS can then be made by the fact that every matrix-valued kernel can be identified with a kernel of the form of ({{EquationNote|4}}) via

:$\backslash Gamma(x,y)\_\{(t,s)\}\; =\; \backslash gamma((x,t),\; (y,s)).$

Moreover, every kernel with the form of ({{EquationNote|4}}) defines a matrix-valued kernel with the above expression. Now letting the map $D:\; H\_\backslash Gamma\; \backslash to\; H\_\backslash gamma$ be defined as

:$(Df)(x,t)\; =\; \backslash langle\; f(x),\; e\_t\; \backslash rangle\_\{\backslash mathbb\{R\}^T\}$

where $e\_t$ is the $t^\backslash text\{th\}$ component of the canonical basis for $\backslash mathbb\{R\}^T$, one can show that $D$ is bijective and an isometry between $H\_\backslash Gamma$ and $H\_\backslash gamma$.

While this view of the vvRKHS can be useful in multi-task learning, this isometry does not reduce the study of the vector-valued case to that of the scalar-valued case. In fact, this isometry procedure can make both the scalar-valued kernel and the input space too difficult to work with in practice as properties of the original kernels are often lost.De VitoZhangAlvarez

An important class of matrix-valued reproducing kernels are separable kernels which can factorized as the product of a scalar valued kernel and a $T$-dimensional symmetric positive semi-definite matrix. In light of our previous discussion these kernels are of the form

:$\backslash gamma((x,t),(y,s))\; =\; K(x,y)\; K\_T(t,s)$

for all $x,y$ in $X$ and $t,s$ in $T$. As the scalar-valued kernel encodes dependencies between the inputs, we can observe that the matrix-valued kernel encodes dependencies among both the inputs and the outputs.

We lastly remark that the above theory can be further extended to spaces of functions with values in function spaces but obtaining kernels for these spaces is a more difficult task.Rosasco

The ReLU function is commonly defined as $f(x)=\backslash max\; \backslash \{0,\; x\backslash \}$ and is a mainstay in the architecture of neural networks where it is used as an activation function. One can construct a ReLU-like nonlinear function using the theory of reproducing kernel Hilbert spaces. Below, we derive this construction and show how it implies the representation power of neural networks with ReLU activations.

We will work with the Hilbert space $\backslash mathcal\{H\}=L^1\_2(0)[0,\; \backslash infty)$ of absolutely continuous functions with $f(0)\; =\; 0$ and square integrable (i.e. $L\_2$) derivative. It has the inner product

: $\backslash langle\; f,g\; \backslash rangle\_\{\backslash mathcal\{H\}\}\; =\; \backslash int\_0^\backslash infty\; f\text{'}(x)g\text{'}(x)\; \backslash ,\; dx\; .$

To construct the reproducing kernel it suffices to consider a dense subspace, so let $f\backslash in\; C^1[0,\; \backslash infty)$ and $f(0)=0$.

The Fundamental Theorem of Calculus then gives

: $f(y)=\; \backslash int\_0^y\; f\text{'}(x)\; \backslash ,\; dx\; =\; \backslash int\_0^\backslash infty\; G(x,y)\; f\text{'}(x)\; \backslash ,\; dx\; =\; \backslash langle\; K\_y,f\; \backslash rangle$

where

:$G(x,y)=$

\begin{cases} 1, & x < y\\

0, & \text{otherwise}

\end{cases}

and $K\_y\text{'}(x)=\; G(x,y),\backslash \; K\_y(0)\; =\; 0$ i.e.

:$K(x,\; y)=K\_y(x)=\backslash int\_0^x\; G(z,\; y)\; \backslash ,\; dz=$

\begin{cases}

x, & 0\leq x

y, & \text{otherwise.}

\end{cases}=\min(x, y)

This implies $K\_y=K(\backslash cdot,\; y)$ reproduces $f$.

Moreover the minimum function on $X\backslash times\; X\; =\; [0,\backslash infty)\backslash times\; [0,\backslash infty)$ has the following representations with the ReLu function:

: $\backslash min(x,y)\; =\; x\; -\backslash operatorname\{ReLU\}(x-y)\; =\; y\; -\; \backslash operatorname\{ReLU\}(y-x).$

Using this formulation, we can apply the representer theorem to the RKHS, letting one prove the optimality of using ReLU activations in neural network settings.{{Citation needed|date=January 2022|reason=Optimal in what sense?}}

• Positive definite kernel
• Mercer's theorem
• Kernel trick
• Kernel embedding of distributions
• Representer theorem

{{Reflist}}

• Alvarez, Mauricio, Rosasco, Lorenzo and Lawrence, Neil, “Kernels for Vector-Valued Functions: a Review,” https://arxiv.org/abs/1106.6251, June 2011.
• {{cite journal

|first=Nachman |last=Aronszajn |author-link=Nachman Aronszajn

|title=Theory of Reproducing Kernels

|journal=Transactions of the American Mathematical Society

|volume=68 |issue=3 |pages=337–404 |year=1950

|jstor=1990404

|mr=51437

|doi=10.1090/S0002-9947-1950-0051437-7

|doi-access=free}}

• Berlinet, Alain and Thomas, Christine. Reproducing kernel Hilbert spaces in Probability and Statistics, Kluwer Academic Publishers, 2004.
• {{cite journal

|first1=Felipe |last1=Cucker

|first2=Steve |last2=Smale

|title=On the Mathematical Foundations of Learning

|journal=Bulletin of the American Mathematical Society

|volume=39 |issue=1 |pages=1–49 |year=2002

|doi=10.1090/S0273-0979-01-00923-5

|mr=1864085

|doi-access=free}}

• De Vito, Ernest, Umanita, Veronica, and Villa, Silvia. "An extension of Mercer theorem to vector-valued measurable kernels," {{ArXiv|1110.4017}}, June 2013.
• Durrett, Greg. 9.520 Course Notes, Massachusetts Institute of Technology, https://www.mit.edu/~9.520/scribe-notes/class03_gdurett.pdf, February 2010.
• {{cite journal

|first1=George |last1=Kimeldorf

|first2=Grace |last2=Wahba |author-link2=Grace Wahba

|url=http://www.stat.wisc.edu/~wahba/ftp1/oldie/kw71.pdf

|title=Some results on Tchebycheffian Spline Functions

|journal=Journal of Mathematical Analysis and Applications

|volume=33 |issue=1 |year=1971 |pages=82–95 |doi=10.1016/0022-247X(71)90184-3

|mr=290013

|doi-access=free}}

• Okutmustur, Baver. “Reproducing Kernel Hilbert Spaces,” M.S. dissertation, Bilkent University, https://users.metu.edu.tr/baver/MS.Thesis.pdf, August 2005.
• Paulsen, Vern. “An introduction to the theory of reproducing kernel Hilbert spaces,” https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=440218056738e05b5ab43679f932a9f33fccee87.
• {{cite journal

|first1=Ingo |last1=Steinwart

|first2=Clint |last2=Scovel

|title= Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs

|journal=Constr. Approx.

|volume=35 |issue=3|year=2012 |pages=363–417

|mr=2914365 |doi=10.1007/s00365-012-9153-3

|s2cid=253885172

}}

• Rosasco, Lorenzo and Poggio, Thomas. "A Regularization Tour of Machine Learning – MIT 9.520 Lecture Notes" Manuscript, Dec. 2014.
• Wahba, Grace, Spline Models for Observational Data, [http://www.siam.org/books/ SIAM], 1990.
• {{cite journal | last1 = Zhang | first1 = Haizhang | last2 = Xu | first2 = Yuesheng | last3 = Zhang | first3 = Qinghui | year = 2012 | title = Refinement of Operator-valued Reproducing Kernels | journal = Journal of Machine Learning Research | volume = 13 | pages = 91–136 |url=http://www.jmlr.org/papers/volume13/zhang12a/zhang12a.pdf}}

Category:Hilbert spaces