radial basis function kernel

{{Short description|Machine learning kernel function}}

In machine learning, the radial basis function kernel, or RBF kernel, is a popular kernel function used in various kernelized learning algorithms. In particular, it is commonly used in support vector machine classification.{{cite journal | last1 = Chang | first1 = Yin-Wen | last2 = Hsieh | first2 = Cho-Jui | last3 = Chang | first3 = Kai-Wei | last4 = Ringgaard | first4 = Michael | last5 = Lin | first5 = Chih-Jen | year = 2010 | title = Training and testing low-degree polynomial data mappings via linear SVM | url = https://jmlr.org/papers/v11/chang10a.html | journal = Journal of Machine Learning Research | volume = 11 | pages = 1471–1490 }}

The RBF kernel on two samples $\backslash mathbf\{x\}\backslash in\; \backslash mathbb\{R\}^\{k\}$ and $\backslash mathbf\{x\text{'}\}$, represented as feature vectors in some input space, is defined asJean-Philippe Vert, Koji Tsuda, and Bernhard Schölkopf (2004). [https://cbio.ensmp.fr/~jvert/publi/04kmcbbook/kernelprimer.pdf "A primer on kernel methods".] Kernel Methods in Computational Biology.

:$K(\backslash mathbf\{x\},\; \backslash mathbf\{x\text{'}\})\; =\; \backslash exp\backslash left(-\backslash frac\{\backslash |\backslash mathbf\{x\}\; -\; \backslash mathbf\{x\text{'}\}\backslash |^2\}\{2\backslash sigma^2\}\backslash right)$

$\backslash textstyle\backslash |\backslash mathbf\{x\}\; -\; \backslash mathbf\{x\text{'}\}\backslash |^2$ may be recognized as the squared Euclidean distance between the two feature vectors. $\backslash sigma$ is a free parameter. An equivalent definition involves a parameter $\backslash textstyle\backslash gamma\; =\; \backslash tfrac\{1\}\{2\backslash sigma^2\}$:

:$K(\backslash mathbf\{x\},\; \backslash mathbf\{x\text{'}\})\; =\; \backslash exp(-\backslash gamma\backslash |\backslash mathbf\{x\}\; -\; \backslash mathbf\{x\text{'}\}\backslash |^2)$

Since the value of the RBF kernel decreases with distance and ranges between zero (in the infinite-distance limit) and one (when {{math|x {{=}} x'}}), it has a ready interpretation as a similarity measure.

The feature space of the kernel has an infinite number of dimensions; for $\backslash sigma\; =\; 1$, its expansion using the multinomial theorem is:{{cite arXiv

|last=Shashua

|first=Amnon

|eprint=0904.3664v1

|title=Introduction to Machine Learning: Class Notes 67577

|class=cs.LG

|year=2009

}}

:$$

\begin{alignat}{2}

\exp\left(-\frac{1}{2}\|\mathbf{x} - \mathbf{x'}\|^2\right)

&= \exp(\frac{2}{2}\mathbf{x}^\top \mathbf{x'} - \frac{1}{2}\|\mathbf{x}\|^2 - \frac{1}{2}\|\mathbf{x'}\|^2)\\[5pt]

&= \exp(\mathbf{x}^\top \mathbf{x'}) \exp( - \frac{1}{2}\|\mathbf{x}\|^2) \exp( - \frac{1}{2}\|\mathbf{x'}\|^2) \\[5pt]

&= \sum_{j=0}^\infty \frac{(\mathbf{x}^\top \mathbf{x'})^j}{j!} \exp\left(-\frac{1}{2}\|\mathbf{x}\|^2\right) \exp\left(-\frac{1}{2}\|\mathbf{x'}\|^2\right)\\[5pt]

&= \sum_{j=0}^\infty \quad \sum_{n_1+n_2+\dots +n_k=j}

\exp\left(-\frac{1}{2}\|\mathbf{x}\|^2\right)

\frac{x_1^{n_1}\cdots x_k^{n_k} }{\sqrt{n_1! \cdots n_k! }}

\exp\left(-\frac{1}{2}\|\mathbf{x'}\|^2\right)

\frac{{x'}_1^{n_1}\cdots {x'}_k^{n_k} }{\sqrt{n_1! \cdots n_k! }} \\[5pt]

&=\langle \varphi(\mathbf{x}), \varphi(\mathbf{x'}) \rangle

\end{alignat}

:$$

\varphi(\mathbf{x})

=

\exp\left(-\frac{1}{2}\|\mathbf{x}\|^2\right)

\left(a^{(0)}_{\ell_0},a^{(1)}_1,\dots,a^{(1)}_{\ell_1},\dots,a^{(j)}_1,\dots,a^{(j)}_{\ell_j},\dots \right )

where $\backslash ell\_j=\backslash tbinom\; \{k+j-1\}\{j\}$,

:$$

a^{(j)}_{\ell}=\frac{x_1^{n_1}\cdots x_k^{n_k} }{\sqrt{n_1! \cdots n_k! }} \quad|\quad n_1+n_2+\dots+n_k = j \wedge 1\leq \ell\leq \ell_j