Restricted Boltzmann machine

The standard type of RBM has binary-valued (Boolean) hidden and visible units, and consists of a matrix of weights $W$ of size $m\backslash times\; n$. Each weight element $(w\_\{i,j\})$ of the matrix is associated with the connection between the visible (input) unit $v\_i$ and the hidden unit $h\_j$. In addition, there are bias weights (offsets) $a\_i$ for $v\_i$ and $b\_j$ for $h\_j$. Given the weights and biases, the energy of a configuration (pair of Boolean vectors) {{math|(v,h)}} is defined as

:$E(v,h)\; =\; -\backslash sum\_i\; a\_i\; v\_i\; -\; \backslash sum\_j\; b\_j\; h\_j\; -\backslash sum\_i\; \backslash sum\_j\; v\_i\; w\_\{i,j\}\; h\_j$

or, in matrix notation,

:$E(v,h)\; =\; -a^\{\backslash mathrm\{T\}\}\; v\; -\; b^\{\backslash mathrm\{T\}\}\; h\; -v^\{\backslash mathrm\{T\}\}\; W\; h.$

This energy function is analogous to that of a Hopfield network. As with general Boltzmann machines, the joint probability distribution for the visible and hidden vectors is defined in terms of the energy function as follows,Geoffrey Hinton (2010). [http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf A Practical Guide to Training Restricted Boltzmann Machines]. UTML TR 2010–003, University of Toronto.

:$P(v,h)\; =\; \backslash frac\{1\}\{Z\}\; e^\{-E(v,h)\}$

where $Z$ is a partition function defined as the sum of $e^\{-E(v,h)\}$ over all possible configurations, which can be interpreted as a normalizing constant to ensure that the probabilities sum to 1. The marginal probability of a visible vector is the sum of $P(v,h)$ over all possible hidden layer configurations,

:$P(v)\; =\; \backslash frac\{1\}\{Z\}\; \backslash sum\_\{\backslash \{h\backslash \}\}\; e^\{-E(v,h)\}$,

and vice versa. Since the underlying graph structure of the RBM is bipartite (meaning there are no intra-layer connections), the hidden unit activations are mutually independent given the visible unit activations. Conversely, the visible unit activations are mutually independent given the hidden unit activations. That is, for m visible units and n hidden units, the conditional probability of a configuration of the visible units {{mvar|v}}, given a configuration of the hidden units {{mvar|h}}, is

:$P(v|h)\; =\; \backslash prod\_\{i=1\}^m\; P(v\_i|h)$.

Conversely, the conditional probability of {{mvar|h}} given {{mvar|v}} is

:$P(h|v)\; =\; \backslash prod\_\{j=1\}^n\; P(h\_j|v)$.

The individual activation probabilities are given by

:$P(h\_j=1|v)\; =\; \backslash sigma\; \backslash left(b\_j\; +\; \backslash sum\_\{i=1\}^m\; w\_\{i,j\}\; v\_i\; \backslash right)$ and $\backslash ,P(v\_i=1|h)\; =\; \backslash sigma\; \backslash left(a\_i\; +\; \backslash sum\_\{j=1\}^n\; w\_\{i,j\}\; h\_j\; \backslash right)$

where $\backslash sigma$ denotes the logistic sigmoid.

The visible units of Restricted Boltzmann Machine can be multinomial, although the hidden units are Bernoulli.{{clarify|date=April 2021|reason=What is meant by Bernoulli in this context?}} In this case, the logistic function for visible units is replaced by the softmax function

:$P(v\_i^k\; =\; 1|h)\; =\; \backslash frac\{\backslash exp(a\_i^k\; +\; \backslash Sigma\_j\; W\_\{ij\}^k\; h\_j)\}\; \{\backslash Sigma\_\{k\text{'}=1\}^K\; \backslash exp(a\_i^\{k\text{'}\}\; +\; \backslash Sigma\_j\; W\_\{ij\}^\{k\text{'}\}\; h\_j)\}$

where K is the number of discrete values that the visible values have. They are applied in topic modeling, and recommender systems.

Restricted Boltzmann machines are a special case of Boltzmann machines and Markov random fields.{{cite journal | first1 = Ilya | last1 = Sutskever | first2 = Tijmen | last2 = Tieleman | year = 2010 | title = On the convergence properties of contrastive divergence | journal = Proc. 13th Int'l Conf. On AI and Statistics (AISTATS) | url = http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2010_SutskeverT10.pdf | url-status = dead | archive-url = https://web.archive.org/web/20150610230811/http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2010_SutskeverT10.pdf | archive-date = 2015-06-10 }}Asja Fischer and Christian Igel. [http://image.diku.dk/igel/paper/TRBMAI.pdf Training Restricted Boltzmann Machines: An Introduction] {{Webarchive|url=https://web.archive.org/web/20150610230447/http://image.diku.dk/igel/paper/TRBMAI.pdf# |date=2015-06-10 }}. Pattern Recognition 47, pp. 25-39, 2014

The graphical model of RBMs corresponds to that of factor analysis.{{cite journal |author1=María Angélica Cueto |author2=Jason Morton |author3=Bernd Sturmfels |year=2010 |title=Geometry of the restricted Boltzmann machine |journal=Algebraic Methods in Statistics and Probability |volume=516 |publisher=American Mathematical Society |arxiv=0908.4425 |bibcode=2009arXiv0908.4425A }}

{{Anchor|Contrastive divergence}}

Restricted Boltzmann machines are trained to maximize the product of probabilities assigned to some training set $V$ (a matrix, each row of which is treated as a visible vector $v$),

:$\backslash arg\backslash max\_W\; \backslash prod\_\{v\; \backslash in\; V\}\; P(v)$

or equivalently, to maximize the expected log probability of a training sample $v$ selected randomly from $V$:

:$\backslash arg\backslash max\_W\; \backslash mathbb\{E\}\; \backslash left[\; \backslash log\; P(v)\backslash right]$

The algorithm most often used to train RBMs, that is, to optimize the weight matrix $W$, is the contrastive divergence (CD) algorithm due to Hinton, originally developed to train PoE (product of experts) models.Geoffrey Hinton (1999). [http://www.gatsby.ucl.ac.uk/publications/papers/06-1999.pdf Products of Experts]. ICANN 1999.{{Cite journal | last1 = Hinton | first1 = G. E. | title = Training Products of Experts by Minimizing Contrastive Divergence | doi = 10.1162/089976602760128018 | journal = Neural Computation | volume = 14 | issue = 8 | pages = 1771–1800 | year = 2002 | pmid = 12180402| s2cid = 207596505 | url = http://www.cs.toronto.edu/~hinton/absps/tr00-004.pdf}}

The algorithm performs Gibbs sampling and is used inside a gradient descent procedure (similar to the way backpropagation is used inside such a procedure when training feedforward neural nets) to compute weight update.

The basic, single-step contrastive divergence (CD-1) procedure for a single sample can be summarized as follows:

1. Take a training sample {{mvar|v}}, compute the probabilities of the hidden units and sample a hidden activation vector {{mvar|h}} from this probability distribution.
2. Compute the outer product of {{mvar|v}} and {{mvar|h}} and call this the positive gradient.
3. From {{mvar|h}}, sample a reconstruction {{mvar|v'}} of the visible units, then resample the hidden activations {{mvar|h'}} from this. (Gibbs sampling step)
4. Compute the outer product of {{mvar|v'}} and {{mvar|h'}} and call this the negative gradient.
5. Let the update to the weight matrix $W$ be the positive gradient minus the negative gradient, times some learning rate: $\backslash Delta\; W\; =\; \backslash epsilon\; (vh^\backslash mathsf\{T\}\; -\; v\text{'}h\text{'}^\backslash mathsf\{T\})$.
6. Update the biases {{mvar|a}} and {{mvar|b}} analogously: $\backslash Delta\; a\; =\; \backslash epsilon\; (v\; -\; v\text{'})$, $\backslash Delta\; b\; =\; \backslash epsilon\; (h\; -\; h\text{'})$.

A Practical Guide to Training RBMs written by Hinton can be found on his homepage.

{{Technical|section|date=August 2023}}

{{More citations needed section|date=August 2023}}

{{See also|Deep belief network}}

• The difference between the Stacked Restricted Boltzmann Machines and RBM is that RBM has lateral connections within a layer that are prohibited to make analysis tractable. On the other hand, the Stacked Boltzmann consists of a combination of an unsupervised three-layer network with symmetric weights and a supervised fine-tuned top layer for recognizing three classes.
• The usage of Stacked Boltzmann is to understand Natural languages, retrieve documents, image generation, and classification. These functions are trained with unsupervised pre-training and/or supervised fine-tuning. Unlike the undirected symmetric top layer, with a two-way unsymmetric layer for connection for RBM. The restricted Boltzmann's connection is three-layers with asymmetric weights, and two networks are combined into one.
• Stacked Boltzmann does share similarities with RBM, the neuron for Stacked Boltzmann is a stochastic binary Hopfield neuron, which is the same as the Restricted Boltzmann Machine. The energy from both Restricted Boltzmann and RBM is given by Gibb's probability measure: $E\; =\; -\backslash frac12\backslash sum\_\{i,j\}\{w\_\{ij\}\{s\_i\}\{s\_j\}\}+\backslash sum\_i\{\backslash theta\_i\}\{s\_i\}$. The training process of Restricted Boltzmann is similar to RBM. Restricted Boltzmann train one layer at a time and approximate equilibrium state with a 3-segment pass, not performing back propagation. Restricted Boltzmann uses both supervised and unsupervised on different RBM for pre-training for classification and recognition. The training uses contrastive divergence with Gibbs sampling: Δw_ij = e*(p_ij - p'_ij)
• The restricted Boltzmann's strength is it performs a non-linear transformation so it's easy to expand, and can give a hierarchical layer of features. The Weakness is that it has complicated calculations of integer and real-valued neurons. It does not follow the gradient of any function, so the approximation of Contrastive divergence to maximum likelihood is improvised.

• {{Citation|last1=Fischer|first1=Asja|title=An Introduction to Restricted Boltzmann Machines|date=2012|work=Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications|pages=14–36|place=Berlin, Heidelberg|publisher=Springer Berlin Heidelberg|last2=Igel|first2=Christian|series=Lecture Notes in Computer Science |volume=7441 |doi=10.1007/978-3-642-33275-3_2 |isbn=978-3-642-33274-6 |doi-access=free}}

• Autoencoder
• Helmholtz machine

{{Reflist|1}}

• {{cite web |url=http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ |title=Introduction to Restricted Boltzmann Machines |first=Edwin |last=Chen |work=Edwin Chen's blog |date=2011-07-18 }}
• {{Cite web |url=https://deeplearning4j.org/restrictedboltzmannmachine.html |title=A Beginner's Tutorial for Restricted Boltzmann Machines |access-date=2018-11-15 |archive-url=https://web.archive.org/web/20170211042953/https://deeplearning4j.org/restrictedboltzmannmachine.html |archive-date=2017-02-11 |url-status=bot: unknown |work=Deeplearning4j Documentation |first1=Chris |last1=Nicholson |first2=Adam |last2=Gibson }}
• {{Cite web |url=http://deeplearning4j.org/understandingRBMs.html |title=Understanding RBMs |access-date=2014-12-29 |archive-url=https://web.archive.org/web/20160920122139/http://deeplearning4j.org/understandingRBMs.html |archive-date=2016-09-20 |url-status=dead |work=Deeplearning4j Documentation |first1=Chris |last1=Nicholson |first2=Adam |last2=Gibson }}

• Python [https://github.com/AmazaspShumik/sklearn-bayes/blob/master/skbayes/decomposition_models/rbm.py implementation] of Bernoulli RBM and [https://github.com/AmazaspShumik/sklearn-bayes/blob/master/ipython_notebooks_tutorials/decomposition_models/rbm_demo.ipynb tutorial]
• [https://github.com/swirepe/SimpleRBM SimpleRBM] is a very small RBM code (24kB) useful for you to learn about how RBMs learn and work.
• Julia implementation of Restricted Boltzmann machines: https://github.com/cossio/RestrictedBoltzmannMachines.jl

Category:Neural network architectures

Category:Stochastic models

Category:Supervised learning

Category:Unsupervised learning