flow-based generative model

File:Normalizing-flow.svg

Let $z\_0$ be a (possibly multivariate) random variable with distribution $p\_0(z\_0)$.

For $i\; =\; 1,\; ...,\; K$, let $z\_i\; =\; f\_i(z\_\{i-1\})$ be a sequence of random variables transformed from $z\_0$. The functions $f\_1,\; ...,\; f\_K$ should be invertible, i.e. the inverse function $f^\{-1\}\_i$ exists. The final output $z\_K$ models the target distribution.

The log likelihood of $z\_K$ is (see derivation):

: $\backslash log\; p\_K(z\_K)\; =\; \backslash log\; p\_0(z\_0)\; -\; \backslash sum\_\{i=1\}^\{K\}\; \backslash log\; \backslash left|\backslash det\; \backslash frac\{df\_i(z\_\{i-1\})\}\{dz\_\{i-1\}\}\backslash right|$

To efficiently compute the log likelihood, the functions $f\_1,\; ...,\; f\_K$ should be easily invertible, and the determinants of their Jacobians should be simple to compute. In practice, the functions $f\_1,\; ...,\; f\_K$ are modeled using deep neural networks, and are trained to minimize the negative log-likelihood of data samples from the target distribution. These architectures are usually designed such that only the forward pass of the neural network is required in both the inverse and the Jacobian determinant calculations. Examples of such architectures include NICE,{{cite arXiv | eprint=1410.8516| last1=Dinh| first1=Laurent| last2=Krueger| first2=David| last3=Bengio| first3=Yoshua| title=NICE: Non-linear Independent Components Estimation| year=2014| class=cs.LG}} RealNVP,{{cite arXiv | eprint=1605.08803| last1=Dinh| first1=Laurent| last2=Sohl-Dickstein| first2=Jascha| last3=Bengio| first3=Samy| title=Density estimation using Real NVP| year=2016| class=cs.LG}} and Glow.{{cite arXiv | eprint=1807.03039| last1=Kingma| first1=Diederik P.| last2=Dhariwal| first2=Prafulla| title=Glow: Generative Flow with Invertible 1x1 Convolutions| year=2018| class=stat.ML}}

Consider $z\_1$ and $z\_0$. Note that $z\_0\; =\; f^\{-1\}\_1(z\_1)$.

By the change of variable formula, the distribution of $z\_1$ is:

: $p\_1(z\_1)\; =\; p\_0(z\_0)\backslash left|\backslash det\; \backslash frac\{df\_1^\{-1\}(z\_1)\}\{dz\_1\}\backslash right|$

Where $\backslash det\; \backslash frac\{df\_1^\{-1\}(z\_1)\}\{dz\_1\}$ is the determinant of the Jacobian matrix of $f^\{-1\}\_1$.

By the inverse function theorem:

: $p\_1(z\_1)\; =\; p\_0(z\_0)\backslash left|\backslash det\; \backslash left(\backslash frac\{df\_1(z\_0)\}\{dz\_0\}\backslash right)^\{-1\}\backslash right|$

By the identity $\backslash det(A^\{-1\})\; =\; \backslash det(A)^\{-1\}$ (where $A$ is an invertible matrix), we have:

: $p\_1(z\_1)\; =\; p\_0(z\_0)\backslash left|\backslash det\; \backslash frac\{df\_1(z\_0)\}\{dz\_0\}\backslash right|^\{-1\}$

The log likelihood is thus:

: $\backslash log\; p\_1(z\_1)\; =\; \backslash log\; p\_0(z\_0)\; -\; \backslash log\; \backslash left|\backslash det\; \backslash frac\{df\_1(z\_0)\}\{dz\_0\}\backslash right|$

In general, the above applies to any $z\_i$ and $z\_\{i-1\}$. Since $\backslash log\; p\_i(z\_i)$ is equal to $\backslash log\; p\_\{i-1\}(z\_\{i-1\})$ subtracted by a non-recursive term, we can infer by induction that:

: $\backslash log\; p\_K(z\_K)\; =\; \backslash log\; p\_0(z\_0)\; -\; \backslash sum\_\{i=1\}^\{K\}\; \backslash log\; \backslash left|\backslash det\; \backslash frac\{df\_i(z\_\{i-1\})\}\{dz\_\{i-1\}\}\backslash right|$

As is generally done when training a deep learning model, the goal with normalizing flows is to minimize the Kullback–Leibler divergence between the model's likelihood and the target distribution to be estimated. Denoting $p\_\backslash theta$ the model's likelihood and $p^*$ the target distribution to learn, the (forward) KL-divergence is:

: $D\_\{\backslash text\{KL\}\}[p^\{*\}(x)\; \backslash |\; p\_\{\backslash theta\}(x)]\; =\; -\backslash mathop\{\backslash mathbb\{E\}\}\_\{p^\{*\}(x)\}[\backslash log\; p\_\{\backslash theta\}(x)]\; +\; \backslash mathop\{\backslash mathbb\{E\}\}\_\{p^\{*\}(x)\}[\backslash log\; p^\{*\}(x)\; ]$

The second term on the right-hand side of the equation corresponds to the entropy of the target distribution and is independent of the parameter $\backslash theta$ we want the model to learn, which only leaves the expectation of the negative log-likelihood to minimize under the target distribution. This intractable term can be approximated with a Monte-Carlo method by importance sampling. Indeed, if we have a dataset $\backslash \{x\_\{i\}\backslash \}\_\{i=1\}^N$ of samples each independently drawn from the target distribution $p^*(x)$, then this term can be estimated as:

: $-\backslash hat\{\backslash mathop\{\backslash mathbb\{E\}\}\}\_\{p^\{*\}(x)\}[\backslash log\; p\_\{\backslash theta\}(x)]\; =\; -\backslash frac\{1\}\{N\}\; \backslash sum\_\{i=0\}^\{N\}\; \backslash log\; p\_\{\backslash theta\}(x\_\{i\})$

Therefore, the learning objective

: $\backslash underset\{\backslash theta\}\{\backslash operatorname\{arg\backslash ,min\}\}\backslash \; D\_\{\backslash text\{KL\}\}[p^\{*\}(x)\; \backslash |\; p\_\{\backslash theta\}(x)]$

is replaced by

: $\backslash underset\{\backslash theta\}\{\backslash operatorname\{arg\backslash ,max\}\}\backslash \; \backslash sum\_\{i=0\}^\{N\}\; \backslash log\; p\_\{\backslash theta\}(x\_\{i\})$

In other words, minimizing the Kullback–Leibler divergence between the model's likelihood and the target distribution is equivalent to maximizing the model likelihood under observed samples of the target distribution.{{Cite journal |last1=Papamakarios |first1=George |last2=Nalisnick |first2=Eric |last3=Rezende |first3=Danilo Jimenez |last4=Shakir |first4=Mohamed |last5=Balaji |first5=Lakshminarayanan |date=March 2021 |title=Normalizing Flows for Probabilistic Modeling and Inference |journal=Journal of Machine Learning Research |url=https://jmlr.org/papers/v22/19-1028.html |volume=22 |issue=57 |pages=1–64 |arxiv=1912.02762}}

A pseudocode for training normalizing flows is as follows:{{Cite journal |last1=Kobyzev |first1=Ivan |last2=Prince |first2=Simon J.D. |last3=Brubaker |first3=Marcus A. |date=November 2021 |title=Normalizing Flows: An Introduction and Review of Current Methods |url=https://ieeexplore.ieee.org/document/9089305 |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=43 |issue=11 |pages=3964–3979 |doi=10.1109/TPAMI.2020.2992934 |pmid=32396070 |arxiv=1908.09257 |s2cid=208910764 |issn=1939-3539}}

• INPUT. dataset $x\_\{1:n\}$, normalizing flow model $f\_\backslash theta(\backslash cdot),\; p\_0$.
• SOLVE. $\backslash max\_\backslash theta\; \backslash sum\_j\; \backslash ln\; p\_\backslash theta(x\_j)$ by gradient descent
• RETURN. $\backslash hat\backslash theta$

The earliest example.{{cite arXiv | eprint=1505.05770| author1=Danilo Jimenez Rezende| last2=Mohamed| first2=Shakir| title=Variational Inference with Normalizing Flows| year=2015| class=stat.ML}} Fix some activation function $h$, and let $\backslash theta\; =\; (u,\; w,\; b)$ with the appropriate dimensions, then$$x\; =\; f\_\backslash theta(z)\; =\; z\; +\; u\; h(\backslash langle\; w,\; z\; \backslash rangle\; +\; b)$$The inverse $f\_\backslash theta^\{-1\}$ has no closed-form solution in general.

The Jacobian is $|\backslash det\; (I\; +\; h\text{'}(\backslash langle\; w,\; z\; \backslash rangle\; +\; b)\; uw^T)|\; =\; |1\; +\; h\text{'}(\backslash langle\; w,\; z\; \backslash rangle\; +\; b)\; \backslash langle\; u,\; w\backslash rangle|$.

For it to be invertible everywhere, it must be nonzero everywhere. For example, $h\; =\; \backslash tanh$ and $\backslash langle\; u,\; w\; \backslash rangle\; >\; -1$ satisfies the requirement.

Let $x,\; z\backslash in\; \backslash R^\{2n\}$ be even-dimensional, and split them in the middle. Then the normalizing flow functions are$$x\; =\; \backslash begin\{bmatrix\}$$

x_1 \\ x_2

\end{bmatrix}= f_\theta(z) = \begin{bmatrix}

z_1 \\z_2

\end{bmatrix} + \begin{bmatrix}

0 \\ m_\theta(z_1)

\end{bmatrix}where $m\_\backslash theta$ is any neural network with weights $\backslash theta$.

$f\_\backslash theta^\{-1\}$ is just $z\_1\; =\; x\_1,\; z\_2\; =\; x\_2\; -\; m\_\backslash theta(x\_1)$, and the Jacobian is just 1, that is, the flow is volume-preserving.

When $n=1$, this is seen as a curvy shearing along the $x\_2$ direction.

The Real Non-Volume Preserving model generalizes NICE model by:$$x\; =\; \backslash begin\{bmatrix\}$$

x_1 \\ x_2

\end{bmatrix}= f_\theta(z) = \begin{bmatrix}

z_1 \\ e^{s_\theta(z_1)} \odot z_2

\end{bmatrix} + \begin{bmatrix}

0 \\ m_\theta(z_1)

\end{bmatrix}

Its inverse is $z\_1\; =\; x\_1,\; z\_2\; =\; e^\{-s\_\backslash theta\; (x\_1)\}\backslash odot\; (x\_2\; -\; m\_\backslash theta\; (x\_1))$, and its Jacobian is $\backslash prod^n\_\{i=1\}\; e^\{s\_\backslash theta(z\_\{1,\; \})\}$. The NICE model is recovered by setting $s\_\backslash theta\; =\; 0$.

Since the Real NVP map keeps the first and second halves of the vector $x$ separate, it's usually required to add a permutation $(x\_1,\; x\_2)\; \backslash mapsto\; (x\_2,\; x\_1)$ after every Real NVP layer.

In generative flow model, each layer has 3 parts:

• channel-wise affine transform$$y\_\{cij\}\; =\; s\_c(x\_\{cij\}\; +\; b\_c)$$with Jacobian $\backslash prod\_c\; s\_c^\{HW\}$.
• invertible 1x1 convolution$$z\_\{cij\}\; =\; \backslash sum\_\{c\text{'}\}\; K\_\{cc\text{'}\}\; y\_\{cij\}$$with Jacobian $\backslash det(K)^\{HW\}$. Here $K$ is any invertible matrix.
• Real NVP, with Jacobian as described in Real NVP.

The idea of using the invertible 1x1 convolution is to permute all layers in general, instead of merely permuting the first and second half, as in Real NVP.

An autoregressive model of a distribution on $\backslash R^n$ is defined as the following stochastic process:{{Cite journal |last1=Papamakarios |first1=George |last2=Pavlakou |first2=Theo |last3=Murray |first3=Iain |date=2017 |title=Masked Autoregressive Flow for Density Estimation |url=https://proceedings.neurips.cc/paper/2017/hash/6c1da886822c67822bcf3679d04369fa-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=30|arxiv=1705.07057 }}

$$\backslash begin\{align\}$$

x_1 \sim& N(\mu_1, \sigma_1^2)\\

x_2 \sim& N(\mu_2(x_1), \sigma_2(x_1)^2)\\

&\cdots \\

x_n \sim& N(\mu_n(x_{1:n-1}), \sigma_n(x_{1:n-1})^2)\\

\end{align}where $\backslash mu\_i:\; \backslash R^\{i-1\}\; \backslash to\; \backslash R$ and $\backslash sigma\_i:\; \backslash R^\{i-1\}\; \backslash to\; (0,\; \backslash infty)$ are fixed functions that define the autoregressive model.

By the reparameterization trick, the autoregressive model is generalized to a normalizing flow:$$\backslash begin\{align\}$$

x_1 =& \mu_1 + \sigma_1 z_1\\

x_2 =& \mu_2(x_1) + \sigma_2(x_1) z_2\\

&\cdots \\

x_n =& \mu_n(x_{1:n-1}) + \sigma_n(x_{1:n-1}) z_n\\

\end{align}The autoregressive model is recovered by setting $z\; \backslash sim\; N(0,\; I\_\{n\})$.

The forward mapping is slow (because it's sequential), but the backward mapping is fast (because it's parallel).

The Jacobian matrix is lower-diagonal, so the Jacobian is $\backslash sigma\_1\; \backslash sigma\_2(x\_1)\backslash cdots\; \backslash sigma\_n(x\_\{1:n-1\})$.

Reversing the two maps $f\_\backslash theta$ and $f\_\backslash theta^\{-1\}$ of MAF results in Inverse Autoregressive Flow (IAF), which has fast forward mapping and slow backward mapping.{{Cite journal |last1=Kingma |first1=Durk P |last2=Salimans |first2=Tim |last3=Jozefowicz |first3=Rafal |last4=Chen |first4=Xi |last5=Sutskever |first5=Ilya |last6=Welling |first6=Max |date=2016 |title=Improved Variational Inference with Inverse Autoregressive Flow |url=https://proceedings.neurips.cc/paper/2016/hash/ddeebdeefdb7e7e7a697e1c3e3d8ef54-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=29|arxiv=1606.04934 }}

Instead of constructing flow by function composition, another approach is to formulate the flow as a continuous-time dynamic.{{cite arXiv | eprint=1810.01367| last1=Grathwohl| first1=Will| last2=Chen| first2=Ricky T. Q.| last3=Bettencourt| first3=Jesse| last4=Sutskever| first4=Ilya| last5=Duvenaud| first5=David| title=FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models| year=2018| class=cs.LG}}{{Cite arXiv |last1=Lipman |first1=Yaron |last2=Chen |first2=Ricky T. Q. |last3=Ben-Hamu |first3=Heli |last4=Nickel |first4=Maximilian |last5=Le |first5=Matt |date=2022-10-01 |title=Flow Matching for Generative Modeling |class=cs.LG | eprint=2210.02747}} Let $z\_0$ be the latent variable with distribution $p(z\_0)$. Map this latent variable to data space with the following flow function:

: $x\; =\; F(z\_0)\; =\; z\_T\; =\; z\_0\; +\; \backslash int\_0^T\; f(z\_t,\; t)\; dt$

where $f$ is an arbitrary function and can be modeled with e.g. neural networks.

The inverse function is then naturally:

: $z\_0\; =\; F^\{-1\}(x)\; =\; z\_T\; +\; \backslash int\_T^0\; f(z\_t,\; t)\; dt\; =\; z\_T\; -\; \backslash int\_0^T\; f(z\_t,t)\; dt$

And the log-likelihood of $x$ can be found as:

: $\backslash log(p(x))\; =\; \backslash log(p(z\_0))\; -\; \backslash int\_0^T\; \backslash text\{Tr\}\backslash left[\backslash frac\{\backslash partial\; f\}\{\backslash partial\; z\_t\}\; \backslash right]\; dt$

Since the trace depends only on the diagonal of the Jacobian $\backslash partial\_\{z\_t\}\; f$, this allows "free-form" Jacobian.{{cite arXiv |last1=Grathwohl |first1=Will |last2=Chen |first2=Ricky T. Q. |last3=Bettencourt |first3=Jesse |last4=Sutskever |first4=Ilya |last5=Duvenaud |first5=David |date=2018-10-22 |title=FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models |class=cs.LG |eprint=1810.01367 }} Here, "free-form" means that there is no restriction on the Jacobian's form. It is contrasted with previous discrete models of normalizing flow, where the Jacobian is carefully designed to be only upper- or lower-diagonal, so that the Jacobian can be evaluated efficiently.

The trace can be estimated by "Hutchinson's trick":{{Cite journal |last1=Finlay |first1=Chris |last2=Jacobsen |first2=Joern-Henrik |last3=Nurbekyan |first3=Levon |last4=Oberman |first4=Adam |date=2020-11-21 |title=How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization |url=https://proceedings.mlr.press/v119/finlay20a.html |journal=International Conference on Machine Learning |language=en |publisher=PMLR |pages=3154–3164|arxiv=2002.02798 }}{{Cite journal |last=Hutchinson |first=M.F. |date=January 1989 |title=A Stochastic Estimator of the Trace of the Influence Matrix for Laplacian Smoothing Splines |url=http://www.tandfonline.com/doi/abs/10.1080/03610918908812806 |journal=Communications in Statistics - Simulation and Computation |language=en |volume=18 |issue=3 |pages=1059–1076 |doi=10.1080/03610918908812806 |issn=0361-0918|url-access=subscription }}

Given any matrix $W\backslash in\; \backslash R^\{n\backslash times\; n\}$, and any random $u\backslash in\; \backslash R^n$ with $E[uu^T]\; =\; I$, we have $E[u^T\; W\; u]\; =\; tr(W)$. (Proof: expand the expectation directly.)

Usually, the random vector is sampled from $N(0,\; I)$ (normal distribution) or $\backslash \{\backslash pm\; n^\{-1/2\}\backslash \}^n$ (Radamacher distribution).

When $f$ is implemented as a neural network, neural ODE methods{{cite conference |last1=Chen |first1=Ricky T. Q. |last2=Rubanova |first2=Yulia |last3=Bettencourt |first3=Jesse |last4=Duvenaud |first4=David K. |year=2018 |editor1-last=Bengio |editor1-first=S. |editor2-last=Wallach |editor2-first=H. |editor3-last=Larochelle |editor3-first=H. |editor4-last=Grauman |editor4-first=K. |editor5-last=Cesa-Bianchi |editor5-first=N. |editor6-last=Garnett |editor6-first=R. |title=Neural Ordinary Differential Equations |url=https://proceedings.neurips.cc/paper_files/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf |conference= |publisher=Curran Associates, Inc. |volume=31 |arxiv=1806.07366 |book-title=Advances in Neural Information Processing Systems}} would be needed. Indeed, CNF was first proposed in the same paper that proposed neural ODE.

There are two main deficiencies of CNF, one is that a continuous flow must be a homeomorphism, thus preserve orientation and ambient isotopy (for example, it's impossible to flip a left-hand to a right-hand by continuous deforming of space, and it's impossible to turn a sphere inside out, or undo a knot), and the other is that the learned flow $f$ might be ill-behaved, due to degeneracy (that is, there are an infinite number of possible $f$ that all solve the same problem).

By adding extra dimensions, the CNF gains enough freedom to reverse orientation and go beyond ambient isotopy (just like how one can pick up a polygon from a desk and flip it around in 3-space, or unknot a knot in 4-space), yielding the "augmented neural ODE".{{Cite journal |last1=Dupont |first1=Emilien |last2=Doucet |first2=Arnaud |last3=Teh |first3=Yee Whye |date=2019 |title=Augmented Neural ODEs |url=https://proceedings.neurips.cc/paper/2019/hash/21be9a4bd4f81549a9d1d241981cec3c-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=32}}

Any homeomorphism of $\backslash R^n$ can be approximated by a neural ODE operating on $\backslash R^\{2n+1\}$, proved by combining Whitney embedding theorem for manifolds and the universal approximation theorem for neural networks.{{cite arXiv |last1=Zhang |first1=Han |last2=Gao |first2=Xi |last3=Unterman |first3=Jacob |last4=Arodz |first4=Tom |date=2019-07-30 |title=Approximation Capabilities of Neural ODEs and Invertible Residual Networks |class=cs.LG |language=en |eprint=1907.12998}}

To regularize the flow $f$, one can impose regularization losses. The paper proposed the following regularization loss based on optimal transport theory:$$\backslash lambda\_\{K\}\; \backslash int\_\{0\}^\{T\}\backslash left\backslash |f(z\_t,\; t)\backslash right\backslash |^\{2\}\; dt$$

+\lambda_{J} \int_{0}^{T}\left\|\nabla_z f(z_t, t)\right\|_F^{2} dt

where $\backslash lambda\_K,\; \backslash lambda\_J\; >0$

are hyperparameters. The first term punishes the model for oscillating the flow field over time, and the second term punishes it for oscillating the flow field over space. Both terms together guide the model into a flow that is smooth (not "bumpy") over space and time.

When a probabilistic flow transforms a distribution on an $m$-dimensional manifold embedded in $\backslash R^n$, where not given by the naive computation of the determinant of the $n\backslash text\{-by-\}n$ Jacobian (which is zero), but instead by the determinant(s) of one or more suitably defined $m\backslash text\{-by-\}m$ matrices. This section is an interpretation of the tutorial in the appendix of Sorrenson et al.(2023),{{Cite arXiv

|last1=Sorrenson

|first1=Peter

|last2=Draxler

|first2=Felix

|last3=Rousselot

|first3=Armand

|last4=Hummerich

|first4=Sander

|last5=Köthe

|first5=Ullrich

|title=Learning Distributions on Manifolds with Free-Form Flows

|eprint=2312.09852

|year=2023

}} where the more general case of non-isometrically embedded Riemann manifolds is also treated. Here we restrict attention to isometrically embedded manifolds.

As running examples of manifolds with smooth, isometric embedding in $\backslash R^n$ we shall use:

• The unit hypersphere: $\backslash mathbb\; S^\{n-1\}=\backslash \{\backslash mathbf\; x\backslash in\backslash R^n:\backslash mathbf\; x\text{'}\backslash mathbf\; x=1\backslash \}$, where flows can be used to generalize e.g. Von Mises-Fisher or uniform spherical distributions.
• The simplex interior: $\backslash Delta^\{n-1\}=\backslash \{\backslash mathbf\; p=(p\_1,\backslash dots,p\_n)\backslash in\backslash R^n:p\_i>0,\; \backslash sum\_ip\_i=1\backslash \}$, where $n$-way categorical distributions live; and where flows can be used to generalize e.g. Dirichlet, or uniform simplex distributions.

As a first example of a spherical manifold flow transform, consider the normalized linear transform, which radially projects onto the unitsphere the output of an invertible linear transform, parametrized by the $n\backslash text\{-by-\}n$ invertible matrix $\backslash mathbf\; M$:

:$$

f_\text{lin}(\mathbf x; \mathbf M) = \frac{\mathbf{Mx}}{\lVert\mathbf{Mx}\rVert}

In full Euclidean space, $f\_\backslash text\{lin\}:\backslash R^n\backslash to\backslash R^n$ is not invertible, but if we restrict the domain and co-domain to the unitsphere, then $f\_\backslash text\{lin\}:\backslash mathbb\; S^\{n-1\}\backslash to\backslash mathbb\; S^\{n-1\}$ is invertible (more specifically it is a bijection and a homeomorphism and a diffeomorphism), with inverse $f\_\backslash text\{lin\}(\backslash cdot\backslash ,;\backslash mathbf\; M^\{-1\})$

. The Jacobian of $f\_\backslash text\{lin\}:\backslash R^n\backslash to\backslash R^n$, at $\backslash mathbf\; y=f\_\backslash text\{lin\}(\backslash mathbf\; x;\backslash mathbf\; M)$ is $\backslash lVert\backslash mathbf\{Mx\}\backslash rVert^\{-1\}(\backslash mathbf\; I\_n\; -\backslash mathbf\{yy\}\text{'})\backslash mathbf\; M$, which has rank $n-1$ and determinant of zero; while as explained here, the factor (see subsection below) relating source and transformed densities is: $\backslash lVert\backslash mathbf\{Mx\}\backslash rVert^\{-n\}\backslash left|\backslash operatorname\{det\}\backslash mathbf\; M\backslash right|$.

For

:$$

P_X(\mathbf x)\operatorname{volume}(U)\approx P_Y(\mathbf y)\operatorname{volume}(V)

where volume (for very small regions) is given by Lebesgue measure in $m$-dimensional tangent space. By making the regions infinitessimally small, the factor relating the two densities is the ratio of volumes, which we term the differential volume ratio.

To obtain concrete formulas for volume, we let $U$ be an $m$-dimensional rectangle, or a linearly transformed rectange, which is a parallelotope (multidimensional generalization of a parallelogram). For very small regions, $f$ becomes essentially linear, so that the image, $V=f(U)$ is also a parallelotope. In $\backslash R^m$, we can represent an $m$-dimensional parallelotope with an $m\backslash text\{-by-\}m$ matrix whose colum-vectors are a set of edges (meeting at a common vertex) that span the paralellotope. The volume is given by the absolute value of the determinant of this matrix. If more generally, the same paralellotope is embedded in $\backslash R^n$, it can be represented with a (tall) $n\backslash text\{-by-\}m$ matrix, say $\backslash mathbf\; V$, whose volume is given by the square root of the Gram determinant, $\backslash sqrt\{\backslash left|\backslash operatorname\{det\}(\backslash mathbf\; V\text{'}\backslash mathbf\; V)\backslash right|\}$. In the sections below, we show various ways to use this volume formula to derive the differential volume ratio.

As a first example, we develop expressions for the differential volume ratio in the simplex case, when $\backslash mathcal\; M=\backslash Delta^\{n-1\}$. Define the embedding function, $e:\backslash tilde\backslash mathbf\; p=(p\_1\backslash dots,p\_\{n-1\})\backslash mapsto\backslash mathbf\; p=(p\_1\backslash dots,p\_\{n-1\},1-\backslash sum\_\{i=1\}^\{n-1\}p\_i)$

which maps an arbitrarily chosen, $m=(n-1)$-dimensional repesentation, $\backslash tilde\backslash mathbf\; p$, to the embedded manifold. The $n\backslash text\{-by-\}m$ Jacobian is

$\backslash mathbf\; E\; =\; \backslash begin\{bmatrix\}$

\mathbf{I}_{n-1} \\

-\boldsymbol{1}'

\end{bmatrix}

.

To define $U$ in $\backslash mathbf\; p$-space, start with a rectangle in $\backslash tilde\backslash mathbf\; p$-space, having (signed) differential side-lengths, $dp\_1,\; \backslash dots,\; dp\_\{n-1\}$ and use these to form the $(n-1)\backslash text\{-by-\}(n-1)$ diagonal matrix $\backslash mathbf\; D$, the columns of which span the rectangle. The paralleotope $U$ is spanned by the columns of the matrix $\backslash mathbf\{ED\}$.

File:Simplex measure pullback.svg from tangent space (coinciding with the simplex), via the embedding $p\_1\backslash mapsto(p\_1,1-p\_1)$, with Jacobian , a scaling factor of $\backslash sqrt\{\backslash mathbf\; E\text{'}\backslash mathbf\; E\}=\backslash sqrt2$ results.]]

The Gram determinant volume formula gives:

:$$

\operatorname{volume}(U) = \sqrt{\left|\operatorname{det}(\mathbf{DE}'\mathbf{ED})\right|}

= \sqrt{\left|\operatorname{det}(\mathbf E'\mathbf E)\right|}\,

\left|\operatorname{det}\mathbf D)\right|

=\sqrt n\prod_{i=1}^{n-1} \left|dp_i\right|

To understand the geometric interpretation of the factor $\backslash sqrt\{n\}$, see the example for the 1-simplex in the diagram at right.

To get the volume of $V=f(U)$, we use $\backslash mathbf\{F\_p\}$ for the $n\backslash text\{-by-\}n$ Jacobian of $f$ at $\backslash mathbf\; p=e(\backslash tilde\backslash mathbf\; p)$ to get:

:$$

\operatorname{volume}(V) =

\sqrt{\left|\operatorname{det}(\mathbf{DE}'\mathbf{ F_p}'\mathbf{F_pED})\right|}

= \sqrt{\left|\operatorname{det}(\mathbf E'\mathbf{F_p}'\mathbf{F_pE})\right|}\,

\left|\operatorname{det}\mathbf D)\right|

so that the factor $\backslash left|\backslash operatorname\{det\}\backslash mathbf\; D)\backslash right|$ cancels in the volume ratio, which can now already be numerically evaluated. It can however be rewritten in a sometimes more convenient form by also introducing the representation function, $r:\backslash mathbf\; p\backslash mapsto\backslash tilde\backslash mathbf\; p$, which simply extracts the first $(n-1)$ components. The Jacobian is . Observe that, since $e\backslash circ\; r\backslash circ\; f=f$, the chain rule for function composition gives: $\backslash mathbf\{ERF\_p\}=\backslash mathbf\{F\_p\}$. By plugging this expansion into the above Gram determinant and then refactoring it as a product of determinants of square matrices, we can extract the factor $\backslash sqrt\{\backslash left|\backslash operatorname\{det\}(\backslash mathbf\; E\text{'}\backslash mathbf\; E)\backslash right|\}=\backslash sqrt\; n$, which now also cancels in the ratio, which finally simpifies to the determinant of the Jacobian of the "sandwiched" flow transformation, $r\backslash circ\; f\backslash circ\; e$:

:$$

R^\Delta_f(\mathbf p)=\frac{\operatorname{volume}(V)}{\operatorname{volume}(U)}

= \left|\operatorname{det}(\mathbf{RF_pE})\right|

which, if $\backslash mathbf\; p\backslash sim\; P\_\backslash mathbf\; P$, can be used to derive the pushforward density after a change of variables, $\backslash mathbf\; q\; =\; f(\backslash mathbf\; p)$:

:$$

P_{\mathbf Q}(\mathbf q) = \frac{P_{\mathbf P}(\mathbf p)}{R^\Delta_f(\mathbf p)}\,,\;\text{where}\;\;

\mathbf p=f^{-1}(\mathbf q)

It should be noted that this formula is valid only because the simplex is flat and the Jacobian, $\backslash mathbf\; E$ is constant. The more general case is discussed below, after we present a concrete example of a simplex transform.

A calibration transform, which is sometimes used in machine learning for post-processing of the (class posterior) outputs of a probabilistic $n$-class classifier,{{Cite arXiv

|last1=Ferrer

|first1=Luciana

|last2=Ramos

|first2=Daniel

|title=Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration

|eprint=2408.02841

|year=2024

}} uses the softmax function to renormalize categorical distributions after scaling and translation of the input distributions in log-probability space. For $\backslash mathbf\; p,\; \backslash mathbf\; q\backslash in\backslash Delta^\{n-1\}$ and with parameters, $a>0$ and $\backslash mathbf\; c\backslash in\backslash R^n$ the transform can be specified as:

:$$

\mathbf q=f_\text{cal}(\mathbf p; a, \mathbf c) = \operatorname{softmax}(a^{-1}\log\mathbf p+\mathbf c)\;\iff\;

\mathbf p=f^{-1}_\text{cal}(\mathbf q; a, \mathbf c) = \operatorname{softmax}(a\log\mathbf q-a\mathbf c)

where the log is applied elementwise. After some algebra the differential volume ratio can be expressed as:

:$$

R^\Delta_\text{cal}(\mathbf p; a, \mathbf c) = \left|\operatorname{det}(\mathbf{RF_pE})\right| = a^{1-n}\prod_{i=1}^n\frac{q_i}{p_i}

While calibration transforms are most often trained as discriminative models, the reinterpretation here as a probabilistic flow allows also the design of generative calibration models.

For a flow, $\backslash mathbf\; y=f(\backslash mathbf\; x)$ on a curved manifold, for example $\backslash mathbb\; S^\{n-1\}$, equipped an with embedding function, $e$ that maps a set of $(n-1)$ angular spherical coordinates to $\backslash mathbb\; S^\{n-1\}$, the Jacobian of $e$ is non-constant and we have to evaluate it at both input ($\backslash mathbf\; \{E\_x\}$) and output ($\backslash mathbf\; \{E\_y\}$). The same applies to $r$, the represententation function that recovers spherical coordinates from points in $\backslash R^n$, where we will need the Jacobian at the output ($\backslash mathbf\{R\_y\}$). The differential volume ratio now generalizes to:

:$$

R_f(\mathbf x) = \left|\operatorname{det}(\mathbf{R_yF_xE_x})\right|\,\frac{\sqrt{\left|\operatorname{det}(\mathbf{E_y}'\mathbf{E_y})\right|}}{\sqrt{\left|\operatorname{det}(\mathbf{E_x}'\mathbf{E_x})\right|}}

For geometric insight, consider $\backslash mathbf\; S^2$, where we choose to work with spherical coordinates, with co-latitude $\backslash theta\backslash in[0,\backslash pi]$ and longitude $\backslash phi\backslash in[0,2\backslash pi)$, where at $\backslash mathbf\; x\; =\; e(\backslash theta,\backslash phi)$, we get $\backslash sqrt\{\backslash left|\backslash operatorname\{det\}(\backslash mathbf\{E\_x\}\text{'}\backslash mathbf\{E\_x\})\backslash right|\}=\backslash sin\backslash theta$, which gives the radius of the circle at that latitude (compare e.g. polar circle to equator).

We can however, more generally for isometrically embedded manifolds, simplify this formula by abandoning the fixed coordinate system as given by the fixed functions $e,r$. Instead at the input and output, we choose two different local coordinate systems, the embedding Jacobians of which are $n\backslash text\{-by-\}(n-1)$ matrices (call them $\backslash mathbf\{T\_x\},\; \backslash mathbf\{T\_y\}$) with orthonormal columns that span the local tangent spaces. Then we get $\backslash mathbf\{E\_x\}=\backslash mathbf\{T\_x\}$ and $\backslash mathbf\{R\_y\}=\backslash mathbf\{T\_y\}\text{'}$ and, since $\backslash mathbf\{T\_x\}\text{'}\backslash mathbf\{T\_x\}=\backslash mathbf\{T\_y\}\text{'}\backslash mathbf\{T\_y\}=\backslash mathbf\; I\_\{n-1\}$, the square root factors cancel, to give:

:$$

R_f(\mathbf x) = \left|\operatorname{det}(\mathbf{T_y}'\mathbf{F_xT_x})\right|

which agrees with the formula from Sorrenson et al.(2023), which they derived via a different argument. Moreover, in that paper they give a general method to construct the tangent matrices and how to efficently compute stochastic gradient approximations for the differential volume log-ratio during learnin g.

The tangent matrix at a point $\backslash mathbf\; x\backslash in\backslash mathcal\; M$ is not unique. If $\backslash mathbf\{T\_x\}$, $n\backslash text\{-by-\}m$, spans the local tangent space with orthonormal columns, then so does $\backslash mathbf\{T\_xQ\}$, if $\backslash mathbf\; Q$ is any $m\backslash text\{-by-\}m$ orthogonal matrix and it is easy to check that the above determinant formula is invariant to such reparametrization of the tangent space. For our running examples:

• The tangent space of the simplex, $\backslash Delta^\{n-1\}$, is the hyperplane that coincides with the simplex and it is the same everywhere.
• The local tangent space at $\backslash mathbf\; x\backslash in\backslash mathcal\; S^\{n-1\}$ is the hyperplane perpendicular to the radius, $\backslash mathbf\; x$ and it is the image (column space) of the orthogonal projection matrix, $\backslash mathbf\; I\_n-\backslash mathbf\{xx\}\text{'}$.

Despite normalizing flows success in estimating high-dimensional densities, some downsides still exist in their designs. First of all, their latent space where input data is projected onto is not a lower-dimensional space and therefore, flow-based models do not allow for compression of data by default and require a lot of computation. However, it is still possible to perform image compression with them.{{cite arXiv | eprint=2008.10486| last1=Helminger| first1=Leonhard| last2=Djelouah| first2=Abdelaziz| last3=Gross| first3=Markus| last4=Schroers| first4=Christopher| title=Lossy Image Compression with Normalizing Flows| year=2020| class=cs.CV}}

Flow-based models are also notorious for failing in estimating the likelihood of out-of-distribution samples (i.e.: samples that were not drawn from the same distribution as the training set).{{cite arXiv | eprint=1810.09136v3| last1=Nalisnick| first1=Eric| last2=Matsukawa| first2=Teh| last3=Zhao| first3=Yee Whye| last4=Song| first4=Zhao| title=Do Deep Generative Models Know What They Don't Know?| year=2018| class=stat.ML}} Some hypotheses were formulated to explain this phenomenon, among which the typical set hypothesis,{{cite arXiv | eprint=1906.02994| last1=Nalisnick| first1=Eric| last2=Matsukawa| first2=Teh| last3=Zhao| first3=Yee Whye| last4=Song| first4=Zhao| title=Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality| year=2019| class=stat.ML}} estimation issues when training models,{{Cite journal |last1=Zhang |first1=Lily |last2=Goldstein |first2=Mark |last3=Ranganath |first3=Rajesh |date=2021 |title=Understanding Failures in Out-of-Distribution Detection with Deep Generative Models |journal=Proceedings of Machine Learning Research |url=https://proceedings.mlr.press/v139/zhang21g.html |volume=139 |pages=12427–12436|pmid=35860036 |pmc=9295254 }} or fundamental issues due to the entropy of the data distributions.{{Cite arXiv|last1=Caterini |first1=Anthony L. |last2=Loaiza-Ganem |first2=Gabriel |date=2022 |title=Entropic Issues in Likelihood-Based OOD Detection |pages=21–26|class=stat.ML |eprint=2109.10794 }}

One of the most interesting properties of normalizing flows is the invertibility of their learned bijective map. This property is given by constraints in the design of the models (cf.: RealNVP, Glow) which guarantee theoretical invertibility. The integrity of the inverse is important in order to ensure the applicability of the change-of-variable theorem, the computation of the Jacobian of the map as well as sampling with the model. However, in practice this invertibility is violated and the inverse map explodes because of numerical imprecision.{{cite arXiv | eprint=2006.09347| last1=Behrmann| first1=Jens| last2=Vicol| first2=Paul| last3=Wang| first3=Kuan-Chieh| last4=Grosse| first4=Roger| last5=Jacobsen| first5=Jörn-Henrik| title=Understanding and Mitigating Exploding Inverses in Invertible Neural Networks| year=2020| class=cs.LG}}

Flow-based generative models have been applied on a variety of modeling tasks, including:

• Audio generation{{cite arXiv | eprint=1912.01219| last1=Ping| first1=Wei| last2=Peng| first2=Kainan| last3=Gorur| first3=Dilan| last4=Lakshminarayanan| first4=Balaji| title=WaveFlow: A Compact Flow-based Model for Raw Audio| year=2019| class=cs.SD}}
• Image generation
• Molecular graph generation{{cite arXiv | eprint=2001.09382| last1=Shi| first1=Chence| last2=Xu| first2=Minkai| last3=Zhu| first3=Zhaocheng| last4=Zhang| first4=Weinan| last5=Zhang| first5=Ming| last6=Tang| first6=Jian| title=GraphAF: A Flow-based Autoregressive Model for Molecular Graph Generation| year=2020| class=cs.LG}}
• Point-cloud modeling{{cite arXiv | eprint=1906.12320| last1=Yang| first1=Guandao| last2=Huang| first2=Xun| last3=Hao| first3=Zekun| last4=Liu| first4=Ming-Yu| last5=Belongie| first5=Serge| last6=Hariharan| first6=Bharath| title=PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows| year=2019| class=cs.CV}}
• Video generation{{cite arXiv | eprint=1903.01434| last1=Kumar| first1=Manoj| last2=Babaeizadeh| first2=Mohammad| last3=Erhan| first3=Dumitru| last4=Finn| first4=Chelsea| last5=Levine| first5=Sergey| last6=Dinh| first6=Laurent| last7=Kingma| first7=Durk| title=VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation| year=2019| class=cs.CV}}
• Lossy image compression
• Anomaly detection{{cite arXiv | eprint=2008.12577| last1=Rudolph| first1=Marco| last2=Wandt| first2=Bastian| last3=Rosenhahn| first3=Bodo| title=Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flows| year=2021| class=cs.CV}}

{{reflist}}

• [https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html Flow-based Deep Generative Models]
• [https://deepgenerativemodels.github.io/notes/flow/ Normalizing flow models]

Category:Machine learning

Category:Statistical models

Category:Probabilistic models