Diffusion model#Score-based generative model

Diffusion models were introduced in 2015 as a method to train a model that can sample from a highly complex probability distribution. They used techniques from non-equilibrium thermodynamics, especially diffusion.{{Cite journal |last1=Sohl-Dickstein |first1=Jascha |last2=Weiss |first2=Eric |last3=Maheswaranathan |first3=Niru |last4=Ganguli |first4=Surya |date=2015-06-01 |title=Deep Unsupervised Learning using Nonequilibrium Thermodynamics |url=http://proceedings.mlr.press/v37/sohl-dickstein15.pdf |journal=Proceedings of the 32nd International Conference on Machine Learning |language=en |publisher=PMLR |volume=37 |pages=2256–2265|arxiv=1503.03585 }}

Consider, for example, how one might model the distribution of all naturally-occurring photos. Each image is a point in the space of all images, and the distribution of naturally-occurring photos is a "cloud" in space, which, by repeatedly adding noise to the images, diffuses out to the rest of the image space, until the cloud becomes all but indistinguishable from a Gaussian distribution $\backslash mathcal\{N\}(0,\; I)$. A model that can approximately undo the diffusion can then be used to sample from the original distribution. This is studied in "non-equilibrium" thermodynamics, as the starting distribution is not in equilibrium, unlike the final distribution.

The equilibrium distribution is the Gaussian distribution $\backslash mathcal\{N\}(0,\; I)$, with pdf $\backslash rho(x)\; \backslash propto\; e^\{-\backslash frac\; 12\; \backslash |x\backslash |^2\}$. This is just the Maxwell–Boltzmann distribution of particles in a potential well $V(x)\; =\; \backslash frac\; 12\; \backslash |x\backslash |^2$ at temperature 1. The initial distribution, being very much out of equilibrium, would diffuse towards the equilibrium distribution, making biased random steps that are a sum of pure randomness (like a Brownian walker) and gradient descent down the potential well. The randomness is necessary: if the particles were to undergo only gradient descent, then they will all fall to the origin, collapsing the distribution.

The 2020 paper proposed the Denoising Diffusion Probabilistic Model (DDPM), which improves upon the previous method by variational inference.{{Cite journal |last1=Ho |first1=Jonathan |last2=Jain |first2=Ajay |last3=Abbeel |first3=Pieter |date=2020 |title=Denoising Diffusion Probabilistic Models |url=https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=33 |pages=6840–6851}}{{Citation |last=Ho |first=Jonathan |title=hojonathanho/diffusion |date=Jun 20, 2020 |url=https://github.com/hojonathanho/diffusion |access-date=2024-09-07}}

To present the model, we need some notation.

• $\backslash beta\_1,\; ...,\; \backslash beta\_T\; \backslash in\; (0,\; 1)$ are fixed constants.
• $\backslash alpha\_t\; :=\; 1-\backslash beta\_t$
• $\backslash bar\; \backslash alpha\_t\; :=\; \backslash alpha\_1\; \backslash cdots\; \backslash alpha\_t$
• $\backslash sigma\_t\; :=\; \backslash sqrt\{1\; -\backslash bar\; \backslash alpha\_t\}$
• $\backslash tilde\; \backslash sigma\_t\; :=\; \backslash frac\{\backslash sigma\_\{t-1\}\}\{\backslash sigma\_\{t\}\}\backslash sqrt\{\backslash beta\_t\}$
• $\backslash tilde\backslash mu\_t(x\_t,\; x\_0)\; :=\backslash frac\{\backslash sqrt\{\backslash alpha\_\{t\}\}(1-\backslash bar\; \backslash alpha\_\{t-1\})x\_t\; +\backslash sqrt\{\backslash bar\backslash alpha\_\{t-1\}\}(1-\backslash alpha\_\{t\})x\_0\}\{\backslash sigma\_\{t\}^2\}$
• $\backslash mathcal\{N\}(\backslash mu,\; \backslash Sigma)$ is the normal distribution with mean $\backslash mu$ and variance $\backslash Sigma$, and $\backslash mathcal\{N\}(x\; |\; \backslash mu,\; \backslash Sigma)$ is the probability density at $x$.
• A vertical bar denotes conditioning.

A forward diffusion process starts at some starting point $x\_0\; \backslash sim\; q$, where $q$ is the probability distribution to be learned, then repeatedly adds noise to it by$$x\_t\; =\; \backslash sqrt\{1-\backslash beta\_t\}\; x\_\{t-1\}\; +\; \backslash sqrt\{\backslash beta\_t\}\; z\_t$$where $z\_1,\; ...,\; z\_T$ are IID samples from $\backslash mathcal\{N\}(0,\; I)$. This is designed so that for any starting distribution of $x\_0$, we have $\backslash lim\_t\; x\_t|x\_0$ converging to $\backslash mathcal\{N\}(0,\; I)$.

The entire diffusion process then satisfies$$q(x\_\{0:T\})\; =\; q(x\_0)q(x\_1|x\_0)\; \backslash cdots\; q(x\_T|x\_\{T-1\})\; =\; q(x\_0)\; \backslash mathcal\{N\}(x\_1\; |\; \backslash sqrt\{\backslash alpha\_1\}\; x\_0,\; \backslash beta\_1\; I)\; \backslash cdots\; \backslash mathcal\{N\}(x\_T\; |\; \backslash sqrt\{\backslash alpha\_T\}\; x\_\{T-1\},\; \backslash beta\_T\; I)$$or$$\backslash ln\; q(x\_\{0:T\})\; =\; \backslash ln\; q(x\_0)\; -\; \backslash sum\_\{t=1\}^T\; \backslash frac\{1\}\{2\backslash beta\_t\}\; \backslash |\; x\_t\; -\; \backslash sqrt\{1-\backslash beta\_t\}x\_\{t-1\}\backslash |^2\; +\; C$$where $C$ is a normalization constant and often omitted. In particular, we note that $x\_\{1:T\}|x\_0$ is a gaussian process, which affords us considerable freedom in reparameterization. For example, by standard manipulation with gaussian process, $$x\_\{t\}|x\_0\; \backslash sim\; N\backslash left(\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\},\; \backslash sigma\_\{t\}^2\; I\; \backslash right)$$$$x\_\{t-1\}\; |\; x\_t,\; x\_0\; \backslash sim\; \backslash mathcal\{N\}(\backslash tilde\backslash mu\_t(x\_t,\; x\_0),\; \backslash tilde\backslash sigma\_t^2\; I)$$In particular, notice that for large $t$, the variable $x\_\{t\}|x\_0\; \backslash sim\; N\backslash left(\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\},\; \backslash sigma\_\{t\}^2\; I\; \backslash right)$ converges to $\backslash mathcal\{N\}(0,\; I)$. That is, after a long enough diffusion process, we end up with some $x\_T$ that is very close to $\backslash mathcal\{N\}(0,\; I)$, with all traces of the original $x\_0\; \backslash sim\; q$ gone.

For example, since$$x\_\{t\}|x\_0\; \backslash sim\; N\backslash left(\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\},\; \backslash sigma\_\{t\}^2\; I\; \backslash right)$$we can sample $x\_\{t\}|x\_0$ directly "in one step", instead of going through all the intermediate steps $x\_1,\; x\_2,\; ...,\; x\_\{t-1\}$.

{{Math proof|title=Derivation by reparameterization|proof=

We know $x\_\{t-1\}|x\_0$ is a gaussian, and $x\_t|x\_\{t-1\}$ is another gaussian. We also know that these are independent. Thus we can perform a reparameterization: $$x\_\{t-1\}\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_\{t-1\}\}\; x\_\{0\}\; +\; \backslash sqrt\{1\; -\; \backslash bar\backslash alpha\_\{t-1\}\}\; z$$ $$x\_t\; =\; \backslash sqrt\{\backslash alpha\_t\}\; x\_\{t-1\}\; +\; \backslash sqrt\{1-\backslash alpha\_t\}\; z\text{'}$$ where $z,\; z\text{'}$ are IID gaussians.

There are 5 variables $x\_0,\; x\_\{t-1\},\; x\_t,\; z,\; z\text{'}$ and two linear equations. The two sources of randomness are $z,\; z\text{'}$, which can be reparameterized by rotation, since the IID gaussian distribution is rotationally symmetric.

By plugging in the equations, we can solve for the first reparameterization: $$x\_t\; =\; \backslash sqrt\{\backslash bar\; \backslash alpha\_t\}x\_0\; +\; \backslash underbrace\{\backslash sqrt\{\backslash alpha\_t\; -\; \backslash bar\backslash alpha\_t\}z\; +\; \backslash sqrt\{1-\backslash alpha\_t\}z\text{'}\}\_\{=\; \backslash sigma\_t\; z$$} where $z$ is a gaussian with mean zero and variance one.

To find the second one, we complete the rotational matrix: $$\backslash begin\{bmatrix\}z$$ \\z'\end{bmatrix} =

\begin{bmatrix} \frac{\sqrt{\alpha_t - \bar\alpha_t}}{\sigma_t} & \frac{\sqrt{\beta_t}}{\sigma_t} \\?&?\end{bmatrix}

\begin{bmatrix} z\\z'\end{bmatrix}

Since rotational matrices are all of the form , we know the matrix must be $$\backslash begin\{bmatrix\}z$$ \\z'\end{bmatrix} =

\begin{bmatrix} \frac{\sqrt{\alpha_t - \bar\alpha_t}}{\sigma_t} & \frac{\sqrt{\beta_t}}{\sigma_t} \\- \frac{\sqrt{\beta_t}}{\sigma_t} & \frac{\sqrt{\alpha_t - \bar\alpha_t}}{\sigma_t}

\end{bmatrix}

\begin{bmatrix} z\\z'\end{bmatrix} and since the inverse of rotational matrix is its transpose,

$$\backslash begin\{bmatrix\}z\; \backslash \backslash z\text{'}\backslash end\{bmatrix\}\; =$$

\begin{bmatrix} \frac{\sqrt{\alpha_t - \bar\alpha_t}}{\sigma_t} & -\frac{\sqrt{\beta_t}}{\sigma_t} \\ \frac{\sqrt{\beta_t}}{\sigma_t} & \frac{\sqrt{\alpha_t - \bar\alpha_t}}{\sigma_t}

\end{bmatrix}

\begin{bmatrix} z\\z'\end{bmatrix}

Plugging back, and simplifying, we have $$x\_t\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}x\_0\; +\; \backslash sigma\_tz$$ $$x\_\{t-1\}\; =\; \backslash tilde\backslash mu\_t(x\_t,\; x\_0)\; -\; \backslash tilde\backslash sigma\_t\; z$$'

}}

The key idea of DDPM is to use a neural network parametrized by $\backslash theta$. The network takes in two arguments $x\_t,\; t$, and outputs a vector $\backslash mu\_\backslash theta(x\_t,\; t)$ and a matrix $\backslash Sigma\_\backslash theta(x\_t,\; t)$, such that each step in the forward diffusion process can be approximately undone by $x\_\{t-1\}\; \backslash sim\; \backslash mathcal\{N\}(\backslash mu\_\backslash theta(x\_t,\; t),\; \backslash Sigma\_\backslash theta(x\_t,\; t))$. This then gives us a backward diffusion process $p\_\backslash theta$ defined by$$p\_\backslash theta(x\_T)\; =\; \backslash mathcal\{N\}(x\_T\; |\; 0,\; I)$$$$p\_\backslash theta(x\_\{t-1\}\; |\; x\_t)\; =\; \backslash mathcal\{N\}(x\_\{t-1\}\; |\; \backslash mu\_\backslash theta(x\_t,\; t),\; \backslash Sigma\_\backslash theta(x\_t,\; t))$$The goal now is to learn the parameters such that $p\_\backslash theta(x\_0)$ is as close to $q(x\_0)$ as possible. To do that, we use maximum likelihood estimation with variational inference.

The ELBO inequality states that $\backslash ln\; p\_\backslash theta(x\_0)\; \backslash geq\; E\_\{x\_\{1:T\}\backslash sim\; q(\backslash cdot\; |\; x\_0)\}[\; \backslash ln\; p\_\backslash theta(x\_\{0:T\})\; -\; \backslash ln\; q(x\_\{1:T\}|x\_0)]$, and taking one more expectation, we get$$E\_\{x\_0\; \backslash sim\; q\}[\backslash ln\; p\_\backslash theta(x\_0)]\; \backslash geq\; E\_\{x\_\{0:T\}\backslash sim\; q\}[\; \backslash ln\; p\_\backslash theta(x\_\{0:T\})\; -\; \backslash ln\; q(x\_\{1:T\}|x\_0)]$$We see that maximizing the quantity on the right would give us a lower bound on the likelihood of observed data. This allows us to perform variational inference.

Define the loss function$$L(\backslash theta)\; :=\; -E\_\{x\_\{0:T\}\backslash sim\; q\}[\; \backslash ln\; p\_\backslash theta(x\_\{0:T\})\; -\; \backslash ln\; q(x\_\{1:T\}|x\_0)]$$and now the goal is to minimize the loss by stochastic gradient descent. The expression may be simplified to{{Cite web |last=Weng |first=Lilian |date=2021-07-11 |title=What are Diffusion Models? |url=https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ |access-date=2023-09-24 |website=lilianweng.github.io |language=en}}$$L(\backslash theta)\; =\; \backslash sum\_\{t=1\}^T\; E\_\{x\_\{t-1\},\; x\_t\backslash sim\; q\}[-\backslash ln\; p\_\backslash theta(x\_\{t-1\}\; |\; x\_t)]\; +\; E\_\{x\_0\; \backslash sim\; q\}[D\_\{KL\}(q(x\_T|x\_0)\; \backslash |\; p\_\backslash theta(x\_T))]\; +\; C$$where $C$ does not depend on the parameter, and thus can be ignored. Since $p\_\backslash theta(x\_T)\; =\; \backslash mathcal\{N\}(x\_T\; |\; 0,\; I)$ also does not depend on the parameter, the term $E\_\{x\_0\; \backslash sim\; q\}[D\_\{KL\}(q(x\_T|x\_0)\; \backslash |\; p\_\backslash theta(x\_T))]$ can also be ignored. This leaves just $L(\backslash theta\; )\; =\; \backslash sum\_\{t=1\}^T\; L\_t$ with $L\_t\; =\; E\_\{x\_\{t-1\},\; x\_t\backslash sim\; q\}[-\backslash ln\; p\_\backslash theta(x\_\{t-1\}\; |\; x\_t)]$ to be minimized.

Since $x\_\{t-1\}\; |\; x\_t,\; x\_0\; \backslash sim\; \backslash mathcal\{N\}(\backslash tilde\backslash mu\_t(x\_t,\; x\_0),\; \backslash tilde\backslash sigma\_t^2\; I)$, this suggests that we should use $\backslash mu\_\backslash theta(x\_t,\; t)\; =\; \backslash tilde\; \backslash mu\_t(x\_t,\; x\_0)$; however, the network does not have access to $x\_0$, and so it has to estimate it instead. Now, since $x\_\{t\}|x\_0\; \backslash sim\; N\backslash left(\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\},\; \backslash sigma\_\{t\}^2\; I\; \backslash right)$, we may write $x\_t\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\}\; +\; \backslash sigma\_t\; z$, where $z$ is some unknown gaussian noise. Now we see that estimating $x\_0$ is equivalent to estimating $z$.

Therefore, let the network output a noise vector $\backslash epsilon\_\backslash theta(x\_t,\; t)$, and let it predict$$\backslash mu\_\backslash theta(x\_t,\; t)\; =\backslash tilde\backslash mu\_t\backslash left(x\_t,\; \backslash frac\{x\_t\; -\; \backslash sigma\_t\; \backslash epsilon\_\backslash theta(x\_t,\; t)\}\{\backslash sqrt\{\backslash bar\backslash alpha\_t\}\}\backslash right)\; =\; \backslash frac\{x\_t\; -\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; \backslash beta\_t/\backslash sigma\_t\}\{\backslash sqrt\{\backslash alpha\_t\}\}$$It remains to design $\backslash Sigma\_\backslash theta(x\_t,\; t)$. The DDPM paper suggested not learning it (since it resulted in "unstable training and poorer sample quality"), but fixing it at some value $\backslash Sigma\_\backslash theta(x\_t,\; t)\; =\; \backslash zeta\_t^2\; I$, where either $\backslash zeta\_t^2\; =\; \backslash beta\_t\; \backslash text\{\; or\; \}\; \backslash tilde\backslash sigma\_t^2$ yielded similar performance.

With this, the loss simplifies to $$L\_t\; =\; \backslash frac\{\backslash beta\_t^2\}\{2\backslash alpha\_t\backslash sigma\_\{t\}^2\backslash zeta\_t^2\}\; E\_\{x\_0\backslash sim\; q;\; z\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)\}\backslash left[\; \backslash left\backslash |\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; -\; z\; \backslash right\backslash |^2\backslash right]\; +\; C$$which may be minimized by stochastic gradient descent. The paper noted empirically that an even simpler loss function$$L\_\{simple,\; t\}\; =\; E\_\{x\_0\backslash sim\; q;\; z\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)\}\backslash left[\; \backslash left\backslash |\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; -\; z\; \backslash right\backslash |^2\backslash right]$$resulted in better models.

After a noise prediction network is trained, it can be used for generating data points in the original distribution in a loop as follows:

1. Compute the noise estimate $\backslash epsilon\; \backslash leftarrow\; \backslash epsilon\_\backslash theta(x\_t,\; t)$
2. Compute the original data estimate $\backslash tilde\; x\_0\; \backslash leftarrow\; (x\_t\; -\; \backslash sigma\_t\; \backslash epsilon)\; /\; \backslash sqrt\{\backslash bar\; \backslash alpha\_t\}$
3. Sample the previous data $x\_\{t-1\}\; \backslash sim\; \backslash mathcal\{N\}(\backslash tilde\backslash mu\_t(x\_t,\; \backslash tilde\; x\_0),\; \backslash tilde\backslash sigma\_t^2\; I)$
4. Change time $t\; \backslash leftarrow\; t-1$

Score-based generative model is another formulation of diffusion modelling. They are also called noise conditional score network (NCSN) or score-matching with Langevin dynamics (SMLD).{{Cite web |title=Generative Modeling by Estimating Gradients of the Data Distribution {{!}} Yang Song |url=https://yang-song.net/blog/2021/score/ |access-date=2023-09-24 |website=yang-song.net}}{{Cite journal |last1=Song |first1=Yang |last2=Ermon |first2=Stefano |date=2019 |title=Generative Modeling by Estimating Gradients of the Data Distribution |url=https://proceedings.neurips.cc/paper/2019/hash/3001ef257407d5a371a96dcd947c7d93-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=32|arxiv=1907.05600 }}{{Cite arXiv |eprint=2011.13456 |class=cs.LG |first1=Yang |last1=Song |first2=Jascha |last2=Sohl-Dickstein |title=Score-Based Generative Modeling through Stochastic Differential Equations |date=2021-02-10 |last3=Kingma |first3=Diederik P. |last4=Kumar |first4=Abhishek |last5=Ermon |first5=Stefano |last6=Poole |first6=Ben}}{{Citation |title=ermongroup/ncsn |date=2019 |url=https://github.com/ermongroup/ncsn |access-date=2024-09-07 |publisher=ermongroup}}

Consider the problem of image generation. Let $x$ represent an image, and let $q(x)$ be the probability distribution over all possible images. If we have $q(x)$ itself, then we can say for certain how likely a certain image is. However, this is intractable in general.

Most often, we are uninterested in knowing the absolute probability of a certain image. Instead, we are usually only interested in knowing how likely a certain image is compared to its immediate neighbors — e.g. how much more likely is an image of cat compared to some small variants of it? Is it more likely if the image contains two whiskers, or three, or with some Gaussian noise added?

Consequently, we are actually quite uninterested in $q(x)$ itself, but rather, $\backslash nabla\_x\; \backslash ln\; q(x)$. This has two major effects:

• One, we no longer need to normalize $q(x)$, but can use any $\backslash tilde\; q(x)\; =\; Cq(x)$, where $C\; =\; \backslash int\; \backslash tilde\; q(x)\; dx\; >\; 0$ is any unknown constant that is of no concern to us.
• Two, we are comparing $q(x)$ neighbors $q(x\; +\; dx)$, by $\backslash frac\{q(x)\}\{q(x+dx)\}\; =e^\{-\backslash langle\; \backslash nabla\_x\; \backslash ln\; q,\; dx\; \backslash rangle\}$

Let the score function be $s(x)\; :=\; \backslash nabla\_x\; \backslash ln\; q(x)$; then consider what we can do with $s(x)$.

As it turns out, $s(x)$ allows us to sample from $q(x)$ using thermodynamics. Specifically, if we have a potential energy function $U(x)\; =\; -\backslash ln\; q(x)$, and a lot of particles in the potential well, then the distribution at thermodynamic equilibrium is the Boltzmann distribution $q\_U(x)\; \backslash propto\; e^\{-U(x)/k\_B\; T\}\; =\; q(x)^\{1/k\_BT\}$. At temperature $k\_BT=1$, the Boltzmann distribution is exactly $q(x)$.

Therefore, to model $q(x)$, we may start with a particle sampled at any convenient distribution (such as the standard gaussian distribution), then simulate the motion of the particle forwards according to the Langevin equation

$$dx\_\{t\}=\; -\backslash nabla\_\{x\_t\}U(x\_t)\; d\; t+d\; W\_t$$

and the Boltzmann distribution is, by Fokker-Planck equation, the unique thermodynamic equilibrium. So no matter what distribution $x\_0$ has, the distribution of $x\_t$ converges in distribution to $q$ as $t\backslash to\; \backslash infty$.

Given a density $q$, we wish to learn a score function approximation $f\_\backslash theta\; \backslash approx\; \backslash nabla\; \backslash ln\; q$. This is score matching.{{Cite web |title=Sliced Score Matching: A Scalable Approach to Density and Score Estimation {{!}} Yang Song |url=https://yang-song.net/blog/2019/ssm/ |access-date=2023-09-24 |website=yang-song.net}} Typically, score matching is formalized as minimizing Fisher divergence function $E\_q[\backslash |f\_\backslash theta(x)\; -\; \backslash nabla\; \backslash ln\; q(x)\backslash |^2]$. By expanding the integral, and performing an integration by parts, $$E\_q[\backslash |f\_\backslash theta(x)\; -\; \backslash nabla\; \backslash ln\; q(x)\backslash |^2]\; =\; E\_q[\backslash |f\_\backslash theta\backslash |^2\; +\; 2\backslash nabla\backslash cdot\; f\_\backslash theta]\; +\; C$$giving us a loss function, also known as the Hyvärinen scoring rule, that can be minimized by stochastic gradient descent.

Suppose we need to model the distribution of images, and we want $x\_0\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)$, a white-noise image. Now, most white-noise images do not look like real images, so $q(x\_0)\; \backslash approx\; 0$ for large swaths of $x\_0\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)$. This presents a problem for learning the score function, because if there are no samples around a certain point, then we can't learn the score function at that point. If we do not know the score function $\backslash nabla\_\{x\_t\}\backslash ln\; q(x\_t)$ at that point, then we cannot impose the time-evolution equation on a particle:$$dx\_\{t\}=\; \backslash nabla\_\{x\_t\}\backslash ln\; q(x\_t)\; d\; t+d\; W\_t$$To deal with this problem, we perform annealing. If $q$ is too different from a white-noise distribution, then progressively add noise until it is indistinguishable from one. That is, we perform a forward diffusion, then learn the score function, then use the score function to perform a backward diffusion.

Consider again the forward diffusion process, but this time in continuous time:$$x\_t\; =\; \backslash sqrt\{1-\backslash beta\_t\}\; x\_\{t-1\}\; +\; \backslash sqrt\{\backslash beta\_t\}\; z\_t$$By taking the $\backslash beta\_t\; \backslash to\; \backslash beta(t)dt,\; \backslash sqrt\{dt\}z\_t\; \backslash to\; dW\_t$ limit, we obtain a continuous diffusion process, in the form of a stochastic differential equation:$$dx\_t\; =\; -\backslash frac\; 12\; \backslash beta(t)\; x\_t\; dt\; +\; \backslash sqrt\{\backslash beta(t)\}\; dW\_t$$where $W\_t$ is a Wiener process (multidimensional Brownian motion).

Now, the equation is exactly a special case of the overdamped Langevin equation$$dx\_t\; =\; -\backslash frac\{D\}\{k\_BT\}\; (\backslash nabla\_x\; U)dt\; +\; \backslash sqrt\{2D\}dW\_t$$where $D$ is diffusion tensor, $T$ is temperature, and $U$ is potential energy field. If we substitute in $D=\; \backslash frac\; 12\; \backslash beta(t)I,\; k\_BT\; =\; 1,\; U\; =\; \backslash frac\; 12\; \backslash |x\backslash |^2$, we recover the above equation. This explains why the phrase "Langevin dynamics" is sometimes used in diffusion models.

Now the above equation is for the stochastic motion of a single particle. Suppose we have a cloud of particles distributed according to $q$ at time $t=0$, then after a long time, the cloud of particles would settle into the stable distribution of $\backslash mathcal\{N\}(0,\; I)$. Let $\backslash rho\_t$ be the density of the cloud of particles at time $t$, then we have$$\backslash rho\_0\; =\; q;\; \backslash quad\; \backslash rho\_T\; \backslash approx\; \backslash mathcal\{N\}(0,\; I)$$and the goal is to somehow reverse the process, so that we can start at the end and diffuse back to the beginning.

By Fokker-Planck equation, the density of the cloud evolves according to$$\backslash partial\_t\; \backslash ln\; \backslash rho\_t\; =\; \backslash frac\; 12\; \backslash beta(t)\; \backslash left($$

n + (x+ \nabla\ln\rho_t) \cdot \nabla \ln\rho_t + \Delta\ln\rho_t

\right)where $n$ is the dimension of space, and $\backslash Delta$ is the Laplace operator. Equivalently,$$\backslash partial\_t\; \backslash rho\_t\; =\; \backslash frac\; 12\; \backslash beta(t)\; (\; \backslash nabla\backslash cdot(x\backslash rho\_t)\; +\; \backslash Delta\; \backslash rho\_t)$$

If we have solved $\backslash rho\_t$ for time $t\backslash in\; [0,\; T]$, then we can exactly reverse the evolution of the cloud. Suppose we start with another cloud of particles with density $\backslash nu\_0\; =\; \backslash rho\_T$, and let the particles in the cloud evolve according to

$$dy\_t\; =\; \backslash frac\{1\}\{2\}\; \backslash beta(T-t)\; y\_\{t\}\; d\; t\; +\; \backslash beta(T-t)\; \backslash underbrace\{\backslash nabla\_\{y\_\{t\}\}\; \backslash ln\; \backslash rho\_\{T-t\}\backslash left(y\_\{t\}\backslash right)\}\_\{\backslash text\; \{score\; function\; \}\}\; d\; t+\backslash sqrt\{\backslash beta(T-t)\}\; d\; W\_t$$

then by plugging into the Fokker-Planck equation, we find that $\backslash partial\_t\; \backslash rho\_\{T-t\}\; =\; \backslash partial\_t\; \backslash nu\_t$. Thus this cloud of points is the original cloud, evolving backwards.{{Cite journal |last=Anderson |first=Brian D.O. |date=May 1982 |title=Reverse-time diffusion equation models |url=http://dx.doi.org/10.1016/0304-4149(82)90051-5 |journal=Stochastic Processes and Their Applications |volume=12 |issue=3 |pages=313–326 |doi=10.1016/0304-4149(82)90051-5 |issn=0304-4149|url-access=subscription }}

At the continuous limit,

$$\backslash bar\; \backslash alpha\_t\; =\; (1-\backslash beta\_1)\; \backslash cdots\; (1-\backslash beta\_t)\; =\; e^\{\backslash sum\_i\; \backslash ln(1-\backslash beta\_i)\}\; \backslash to\; e^\{-\backslash int\_0^t\; \backslash beta(t)dt\}$$

and so

$$x\_\{t\}|x\_0\; \backslash sim\; N\backslash left(e^\{-\backslash frac\; 12\backslash int\_0^t\; \backslash beta(t)dt\}\; x\_\{0\},\; \backslash left(1-\; e^\{-\backslash int\_0^t\; \backslash beta(t)dt\}\backslash right)\; I\; \backslash right)$$

In particular, we see that we can directly sample from any point in the continuous diffusion process without going through the intermediate steps, by first sampling $x\_0\; \backslash sim\; q,\; z\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)$, then get $x\_t\; =\; e^\{-\backslash frac\; 12\backslash int\_0^t\; \backslash beta(t)dt\}\; x\_\{0\}\; +\; \backslash left(1-\; e^\{-\backslash int\_0^t\; \backslash beta(t)dt\}\backslash right)\; z$. That is, we can quickly sample $x\_t\; \backslash sim\; \backslash rho\_t$ for any $t\; \backslash geq\; 0$.

Now, define a certain probability distribution $\backslash gamma$ over $[0,\; \backslash infty)$, then the score-matching loss function is defined as the expected Fisher divergence:

$$L(\backslash theta)\; =\; E\_\{t\backslash sim\; \backslash gamma,\; x\_t\; \backslash sim\; \backslash rho\_t\}[\backslash |f\_\backslash theta(x\_t,\; t)\backslash |^2\; +\; 2\backslash nabla\backslash cdot\; f\_\backslash theta(x\_t,\; t)]$$

After training, $f\_\backslash theta(x\_t,\; t)\; \backslash approx\; \backslash nabla\; \backslash ln\backslash rho\_t$, so we can perform the backwards diffusion process by first sampling $x\_T\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)$, then integrating the SDE from $t=T$ to $t=0$:

$$x\_\{t-dt\}=x\_t\; +\; \backslash frac\{1\}\{2\}\; \backslash beta(t)\; x\_\{t\}\; d\; t\; +\; \backslash beta(t)\; f\_\backslash theta(x\_t,\; t)\; d\; t+\backslash sqrt\{\backslash beta(t)\}\; d\; W\_t$$

This may be done by any SDE integration method, such as Euler–Maruyama method.

The name "noise conditional score network" is explained thus:

• "network", because $f\_\backslash theta$ is implemented as a neural network.
• "score", because the output of the network is interpreted as approximating the score function $\backslash nabla\backslash ln\backslash rho\_t$.
• "noise conditional", because $\backslash rho\_t$ is equal to $\backslash rho\_0$ blurred by an added gaussian noise that increases with time, and so the score function depends on the amount of noise added.

DDPM and score-based generative models are equivalent.{{Cite arXiv |eprint=2208.11970v1 |last=Luo |first=Calvin |date=2022 |title=Understanding Diffusion Models: A Unified Perspective|class=cs.LG }} This means that a network trained using DDPM can be used as a NCSN, and vice versa.

We know that $x\_\{t\}|x\_0\; \backslash sim\; N\backslash left(\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\},\; \backslash sigma\_\{t\}^2\; I\backslash right)$, so by Tweedie's formula, we have

$$\backslash nabla\_\{x\_t\}\backslash ln\; q(x\_t)\; =\; \backslash frac\{1\}\{\backslash sigma\_\{t\}^2\}(-x\_t\; +\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}\; E\_q[x\_0|x\_t])$$

As described previously, the DDPM loss function is $\backslash sum\_t\; L\_\{simple,\; t\}$ with

$$L\_\{simple,\; t\}\; =\; E\_\{x\_0\backslash sim\; q;\; z\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)\}\backslash left[\; \backslash left\backslash |\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; -\; z\; \backslash right\backslash |^2\backslash right]$$

where $x\_t\; =\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_\{0\}\; +\; \backslash sigma\_tz$

. By a change of variables,

$$L\_\{simple,\; t\}\; =\; E\_\{x\_0,\; x\_t\backslash sim\; q\}\backslash left[\; \backslash left\backslash |\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; -$$

\frac{x_t -\sqrt{\bar\alpha_t} x_{0}}{\sigma_t} \right\|^2\right] = E_{x_t\sim q, x_0\sim q(\cdot | x_t)}\left[ \left\| \epsilon_\theta(x_t, t) -

\frac{x_t -\sqrt{\bar\alpha_t} x_{0}}{\sigma_t} \right\|^2\right]

and the term inside becomes a least squares regression, so if the network actually reaches the global minimum of loss, then we have $\backslash epsilon\_\backslash theta(x\_t,\; t)\; =\; \backslash frac\{x\_t\; -\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; E\_q[x\_0|x\_t]\}\{\backslash sigma\_t\}\; =\; -\backslash sigma\_t\backslash nabla\_\{x\_t\}\backslash ln\; q(x\_t)$

Thus, a score-based network predicts noise, and can be used for denoising.

Conversely, the continuous limit $x\_\{t-1\}\; =\; x\_\{t-dt\},\; \backslash beta\_t\; =\; \backslash beta(t)\; dt,\; z\_t\backslash sqrt\{dt\}\; =\; dW\_t$ of the backward equation

$$x\_\{t-1\}\; =\; \backslash frac\{x\_t\}\{\backslash sqrt\{\backslash alpha\_t\}\}-\; \backslash frac\{\; \backslash beta\_t\}\{\backslash sigma\_\{t\}\backslash sqrt\{\backslash alpha\_t\; \}\}\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; +\; \backslash sqrt\{\backslash beta\_t\}\; z\_t;\; \backslash quad\; z\_t\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)$$

gives us precisely the same equation as score-based diffusion:

$$x\_\{t-dt\}\; =\; x\_t(1+\backslash beta(t)dt\; /\; 2)\; +\; \backslash beta(t)\; \backslash nabla\_\{x\_t\}\backslash ln\; q(x\_t)\; dt\; +\; \backslash sqrt\{\backslash beta(t)\}dW\_t$$Thus, at infinitesimal steps of DDPM, a denoising network performs score-based diffusion.

File:Linear diffusion noise scheduler.svg

In DDPM, the sequence of numbers is called a (discrete time) noise schedule. In general, consider a strictly increasing monotonic function $\backslash sigma$ of type $\backslash R\; \backslash to\; (0,\; 1)$, such as the sigmoid function. In that case, a noise schedule is a sequence of real numbers . It then defines a sequence of noises $\backslash sigma\_t\; :=\; \backslash sigma(\backslash lambda\_t)$, which then derives the other quantities $\backslash beta\_t\; =\; 1\; -\; \backslash frac\{1\; -\; \backslash sigma\_t^2\}\{1\; -\; \backslash sigma\_\{t-1\}^2\}$.

In order to use arbitrary noise schedules, instead of training a noise prediction model $\backslash epsilon\_\backslash theta(x\_t,\; t)$, one trains $\backslash epsilon\_\backslash theta(x\_t,\; \backslash sigma\_t)$.

Similarly, for the noise conditional score network, instead of training $f\_\backslash theta(x\_t,\; t)$, one trains $f\_\backslash theta(x\_t,\; \backslash sigma\_t)$.

The original DDPM method for generating images is slow, since the forward diffusion process usually takes $T\; \backslash sim\; 1000$ to make the distribution of $x\_T$ to appear close to gaussian. However this means the backward diffusion process also take 1000 steps. Unlike the forward diffusion process, which can skip steps as $x\_t\; |\; x\_0$ is gaussian for all $t\; \backslash geq\; 1$, the backward diffusion process does not allow skipping steps. For example, to sample $x\_\{t-2\}|x\_\{t-1\}\; \backslash sim\; \backslash mathcal\{N\}(\backslash mu\_\backslash theta(x\_\{t-1\},\; t-1),\; \backslash Sigma\_\backslash theta(x\_\{t-1\},\; t-1))$ requires the model to first sample $x\_\{t-1\}$. Attempting to directly sample $x\_\{t-2\}|x\_t$ would require us to marginalize out $x\_\{t-1\}$, which is generally intractable.

DDIM{{Cite arXiv |last1=Song |first1=Jiaming |last2=Meng |first2=Chenlin |last3=Ermon |first3=Stefano |date=3 Oct 2023 |title=Denoising Diffusion Implicit Models |class=cs.LG |eprint=2010.02502}} is a method to take any model trained on DDPM loss, and use it to sample with some steps skipped, sacrificing an adjustable amount of quality. If we generate the Markovian chain case in DDPM to non-Markovian case, DDIM corresponds to the case that the reverse process has variance equals to 0. In other words, the reverse process (and also the forward process) is deterministic. When using fewer sampling steps, DDIM outperforms DDPM.

In detail, the DDIM sampling method is as follows. Start with the forward diffusion process $x\_t\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_0\; +\; \backslash sigma\_t\; \backslash epsilon$. Then, during the backward denoising process, given $x\_t,\; \backslash epsilon\_\backslash theta(x\_t,\; t)$, the original data is estimated as $$x\_0\text{'}\; =\; \backslash frac\{x\_t\; -\; \backslash sigma\_t\; \backslash epsilon\_\backslash theta(x\_t,\; t)\}\{\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}\}$$then the backward diffusion process can jump to any step , and the next denoised sample is $$x\_\{s\}\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_\{s\}\}\; x\_0\text{'}$$

+ \sqrt{\sigma_{s}^2 - (\sigma'_s)^2} \epsilon_\theta(x_t, t)

+ \sigma_s' \epsilonwhere $\backslash sigma\_s\text{'}$ is an arbitrary real number within the range $[0,\; \backslash sigma\_s]$, and $\backslash epsilon\; \backslash sim\; \backslash mathcal\{N\}(0,\; I)$ is a newly sampled gaussian noise. If all $\backslash sigma\_s\text{'}\; =\; 0$, then the backward process becomes deterministic, and this special case of DDIM is also called "DDIM". The original paper noted that when the process is deterministic, samples generated with only 20 steps are already very similar to ones generated with 1000 steps on the high-level.

The original paper recommended defining a single "eta value" $\backslash eta\; \backslash in\; [0,\; 1]$, such that $\backslash sigma\_s\text{'}\; =\; \backslash eta\; \backslash tilde\backslash sigma\_s$. When $\backslash eta\; =\; 1$, this is the original DDPM. When $\backslash eta\; =\; 0$, this is the fully deterministic DDIM. For intermediate values, the process interpolates between them.

By the equivalence, the DDIM algorithm also applies for score-based diffusion models.

{{Main|Latent diffusion model}}

Since the diffusion model is a general method for modelling probability distributions, if one wants to model a distribution over images, one can first encode the images into a lower-dimensional space by an encoder, then use a diffusion model to model the distribution over encoded images. Then to generate an image, one can sample from the diffusion model, then use a decoder to decode it into an image.{{Cite arXiv|last1=Rombach |first1=Robin |last2=Blattmann |first2=Andreas |last3=Lorenz |first3=Dominik |last4=Esser |first4=Patrick |last5=Ommer |first5=Björn |date=13 April 2022 |title=High-Resolution Image Synthesis With Latent Diffusion Models |class=cs.CV |eprint=2112.10752 }}

The encoder-decoder pair is most often a variational autoencoder (VAE).

{{Cite journal |last1=Nichol |first1=Alexander Quinn |last2=Dhariwal |first2=Prafulla |date=2021-07-01 |title=Improved Denoising Diffusion Probabilistic Models |url=https://proceedings.mlr.press/v139/nichol21a.html |journal=Proceedings of the 38th International Conference on Machine Learning |language=en |publisher=PMLR |pages=8162–8171}} proposed various architectural improvements. For example, they proposed log-space interpolation during backward sampling. Instead of sampling from $x\_\{t-1\}\; \backslash sim\; \backslash mathcal\{N\}(\backslash tilde\backslash mu\_t(x\_t,\; \backslash tilde\; x\_0),\; \backslash tilde\backslash sigma\_t^2\; I)$, they recommended sampling from $\backslash mathcal\{N\}(\backslash tilde\backslash mu\_t(x\_t,\; \backslash tilde\; x\_0),\; (\backslash sigma\_t^v\; \backslash tilde\backslash sigma\_t^\{1-v\})^2\; I)$ for a learned parameter $v$.

In the v-prediction formalism, the noising formula $x\_t\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}\; x\_0\; +\; \backslash sqrt\{1\; -\; \backslash bar\backslash alpha\_t\}\; \backslash epsilon\_t$ is reparameterised by an angle $\backslash phi\_t$ such that $\backslash cos\; \backslash phi\_t\; =\; \backslash sqrt\{\backslash bar\backslash alpha\_t\}$ and a "velocity" defined by $\backslash cos\backslash phi\_t\; \backslash epsilon\_t\; -\; \backslash sin\backslash phi\_t\; x\_0$. The network is trained to predict the velocity $\backslash hat\; v\_\backslash theta$, and denoising is by $x\_\{\backslash phi\_t\; -\; \backslash delta\}\; =\; \backslash cos(\backslash delta)\backslash ;\; x\_\{\backslash phi\_t\}\; -\; \backslash sin(\backslash delta)\; \backslash hat\{v\}\_\{\backslash theta\}\backslash ;\; (x\_\{\backslash phi\_t\})$.{{Cite conference|conference=The Tenth International Conference on Learning Representations (ICLR 2022)|last1=Salimans|first1=Tim|last2=Ho|first2=Jonathan|date=2021-10-06|title=Progressive Distillation for Fast Sampling of Diffusion Models|url=https://openreview.net/forum?id=TIdIXIpzhoI|language=en}} This parameterization was found to improve performance, as the model can be trained to reach total noise (i.e. $\backslash phi\_t\; =\; 90^\backslash circ$) and then reverse it, whereas the standard parameterization never reaches total noise since $\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; >\; 0$ is always true.{{Cite conference|conference=IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)|last1=Lin |first1=Shanchuan |last2=Liu |first2=Bingchen |last3=Li |first3=Jiashi |last4=Yang |first4=Xiao |date=2024 |title=Common Diffusion Noise Schedules and Sample Steps Are Flawed |url=https://openaccess.thecvf.com/content/WACV2024/html/Lin_Common_Diffusion_Noise_Schedules_and_Sample_Steps_Are_Flawed_WACV_2024_paper.html |language=en |pages=5404–5411}}

Classifier guidance was proposed in 2021 to improve class-conditional generation by using a classifier. The original publication used CLIP text encoders to improve text-conditional image generation.

Suppose we wish to sample not from the entire distribution of images, but conditional on the image description. We don't want to sample a generic image, but an image that fits the description "black cat with red eyes". Generally, we want to sample from the distribution $p(x|y)$, where $x$ ranges over images, and $y$ ranges over classes of images (a description "black cat with red eyes" is just a very detailed class, and a class "cat" is just a very vague description).

Taking the perspective of the noisy channel model, we can understand the process as follows: To generate an image $x$ conditional on description $y$, we imagine that the requester really had in mind an image $x$, but the image is passed through a noisy channel and came out garbled, as $y$. Image generation is then nothing but inferring which $x$ the requester had in mind.

In other words, conditional image generation is simply "translating from a textual language into a pictorial language". Then, as in noisy-channel model, we use Bayes theorem to get

$$p(x|y)\; \backslash propto\; p(y|x)p(x)$$

in other words, if we have a good model of the space of all images, and a good image-to-class translator, we get a class-to-image translator "for free". In the equation for backward diffusion, the score $\backslash nabla\; \backslash ln\; p(x)$ can be replaced by

$$\backslash nabla\_x\; \backslash ln\; p(x|y)\; =\; \backslash underbrace\{\backslash nabla\_x\; \backslash ln\; p(x)\}\_\{\backslash text\{score\}\}\; +\; \backslash underbrace\{\backslash nabla\_x\; \backslash ln\; p(y|x)\}\_\{\backslash text\{classifier\; guidance\}\}$$

where $\backslash nabla\_x\; \backslash ln\; p(x)$ is the score function, trained as previously described, and $\backslash nabla\_x\; \backslash ln\; p(y|x)$ is found by using a differentiable image classifier.

During the diffusion process, we need to condition on the time, giving$$\backslash nabla\_\{x\_t\}\; \backslash ln\; p(x\_t|y,\; t)\; =\; \backslash nabla\_\{x\_t\}\; \backslash ln\; p(y|x\_t,\; t)\; +\; \backslash nabla\_\{x\_t\}\; \backslash ln\; p(x\_t|t)$$Although, usually the classifier model does not depend on time, in which case $p(y|x\_t,\; t)\; =\; p(y|x\_t)$.

Classifier guidance is defined for the gradient of score function, thus for score-based diffusion network, but as previously noted, score-based diffusion models are equivalent to denoising models by $\backslash epsilon\_\backslash theta(x\_t,\; t)\; =$

-\sigma_t\nabla_{x_t}\ln p(x_t|t), and similarly, $\backslash epsilon\_\backslash theta(x\_t,\; y,\; t)\; =$

-\sigma_t\nabla_{x_t}\ln p(x_t|y, t). Therefore, classifier guidance works for denoising diffusion as well, using the modified noise prediction:$$\backslash epsilon\_\backslash theta(x\_t,\; y,\; t)\; =\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; -\; \backslash underbrace\{\backslash sigma\_t\; \backslash nabla\_\{x\_t\}\; \backslash ln\; p(y|x\_t,\; t)\}\_\{\backslash text\{classifier\; guidance\}\}$$

The classifier-guided diffusion model samples from $p(x|y)$, which is concentrated around the maximum a posteriori estimate $\backslash arg\backslash max\_x\; p(x|y)$. If we want to force the model to move towards the maximum likelihood estimate $\backslash arg\backslash max\_x\; p(y|x)$, we can use

$$p\_\backslash gamma(x|y)\; \backslash propto\; p(y|x)^\backslash gamma\; p(x)$$

where $\backslash gamma\; >\; 0$ is interpretable as inverse temperature. In the context of diffusion models, it is usually called the guidance scale. A high $\backslash gamma$ would force the model to sample from a distribution concentrated around $\backslash arg\backslash max\_x\; p(y|x)$. This sometimes improves quality of generated images.{{Cite arXiv |last1=Dhariwal |first1=Prafulla |last2=Nichol |first2=Alex |date=2021-06-01 |title=Diffusion Models Beat GANs on Image Synthesis |class=cs.LG |eprint=2105.05233 }}

This gives a modification to the previous equation:$$\backslash nabla\_x\; \backslash ln\; p\_\backslash beta(x|y)\; =\; \backslash nabla\_x\; \backslash ln\; p(x)\; +\; \backslash gamma\; \backslash nabla\_x\; \backslash ln\; p(y|x)$$For denoising models, it corresponds to$$\backslash epsilon\_\backslash theta(x\_t,\; y,\; t)\; =\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; -\; \backslash gamma\; \backslash sigma\_t\; \backslash nabla\_\{x\_t\}\; \backslash ln\; p(y|x\_t,\; t)$$

If we do not have a classifier $p(y|x)$, we could still extract one out of the image model itself:{{Cite arXiv |last1=Ho |first1=Jonathan |last2=Salimans |first2=Tim |date=2022-07-25 |title=Classifier-Free Diffusion Guidance |class=cs.LG |eprint=2207.12598 }}

$$\backslash nabla\_x\; \backslash ln\; p\_\backslash gamma(x|y)\; =\; (1-\backslash gamma)\; \backslash nabla\_x\; \backslash ln\; p(x)\; +\; \backslash gamma\; \backslash nabla\_x\; \backslash ln\; p(x|y)$$

Such a model is usually trained by presenting it with both $(x,\; y)$ and $(x,\; \{\backslash rm\; None\})$, allowing it to model both $\backslash nabla\_x\backslash ln\; p(x|y)$ and $\backslash nabla\_x\backslash ln\; p(x)$.

Note that for CFG, the diffusion model cannot be merely a generative model of the entire data distribution $\backslash nabla\_x\; \backslash ln\; p(x)$. It must be a conditional generative model $\backslash nabla\_x\; \backslash ln\; p(x\; |\; y)$. For example, in stable diffusion, the diffusion backbone takes as input both a noisy model $x\_t$, a time $t$, and a conditioning vector $y$ (such as a vector encoding a text prompt), and produces a noise prediction $\backslash epsilon\_\backslash theta(x\_t,\; y,\; t)$.

For denoising models, it corresponds to$$\backslash epsilon\_\backslash theta(x\_t,\; y,\; t,\; \backslash gamma)\; =\; \backslash epsilon\_\backslash theta(x\_t,\; t)\; +\; \backslash gamma\; (\backslash epsilon\_\backslash theta(x\_t,\; y,\; t)\; -\; \backslash epsilon\_\backslash theta(x\_t,\; t))$$As sampled by DDIM, the algorithm can be written as{{cite arXiv |last1=Chung |first1=Hyungjin |title=CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models |date=2024-06-12 |eprint=2406.08070 |last2=Kim |first2=Jeongsol |last3=Park |first3=Geon Yeong |last4=Nam |first4=Hyelin |last5=Ye |first5=Jong Chul|class=cs.CV }}$$\backslash begin\{aligned\}$$

\epsilon_{\text{uncond}} &\leftarrow \epsilon_\theta(x_t, t) \\

\epsilon_{\text{cond}} &\leftarrow \epsilon_\theta(x_t, t, c) \\

\epsilon_{\text{CFG}} &\leftarrow \epsilon_{\text{uncond}} + \gamma(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})\\

x_0 &\leftarrow (x_t - \sigma_t \epsilon_{\text{CFG}}) / \sqrt{1 - \sigma_t^2}\\

x_s &\leftarrow \sqrt{1 - \sigma_s^2} x_0 + \sqrt{\sigma_s^2 - (\sigma_s')^2} \epsilon_{\text{uncond}} + \sigma_s' \epsilon\\

\end{aligned}A similar technique applies to language model sampling. Also, if the unconditional generation $\backslash epsilon\_\{\backslash text\{uncond\}\}\; \backslash leftarrow\; \backslash epsilon\_\backslash theta(x\_t,\; t)$ is replaced by $\backslash epsilon\_\{\backslash text\{neg\; cond\}\}\; \backslash leftarrow\; \backslash epsilon\_\backslash theta(x\_t,\; t,\; c\text{'})$, then it results in negative prompting, which pushes the generation away from $c\text{'}$ condition.{{cite arXiv |last1=Sanchez |first1=Guillaume |title=Stay on topic with Classifier-Free Guidance |date=2023-06-30 |eprint=2306.17806 |last2=Fan |first2=Honglu |last3=Spangher |first3=Alexander |last4=Levi |first4=Elad |last5=Ammanamanchi |first5=Pawan Sasanka |last6=Biderman |first6=Stella|class=cs.CL }}{{cite arXiv |last1=Armandpour |first1=Mohammadreza |title=Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond |date=2023-04-26 |eprint=2304.04968 |last2=Sadeghian |first2=Ali |last3=Zheng |first3=Huangjie |last4=Sadeghian |first4=Amir |last5=Zhou |first5=Mingyuan|class=cs.CV }}

Given a diffusion model, one may regard it either as a continuous process, and sample from it by integrating a SDE, or one can regard it as a discrete process, and sample from it by iterating the discrete steps. The choice of the "noise schedule" $\backslash beta\_t$ can also affect the quality of samples. A noise schedule is a function that sends a natural number to a noise level: $$t\; \backslash mapsto\; \backslash beta\_t,\; \backslash quad\; t\; \backslash in\; \backslash \{1,\; 2,\; \backslash dots\backslash \},\; \backslash beta\; \backslash in\; (0,\; 1)$$A noise schedule is more often specified by a map $t\; \backslash mapsto\; \backslash sigma\_t$. The two definitions are equivalent, since $\backslash beta\_t\; =\; 1\; -\; \backslash frac\{1\; -\; \backslash sigma\_t^2\}\{1\; -\; \backslash sigma\_\{t-1\}^2\}$.

In the DDPM perspective, one can use the DDPM itself (with noise), or DDIM (with adjustable amount of noise). The case where one adds noise is sometimes called ancestral sampling.{{cite arXiv |eprint=2206.00364 |last1=Yang |first1=Ling |last2=Zhang |first2=Zhilong |last3=Song |first3=Yang |last4=Hong |first4=Shenda |last5=Xu |first5=Runsheng |last6=Zhao |first6=Yue |last7=Zhang |first7=Wentao |last8=Cui |first8=Bin |last9=Yang |first9=Ming-Hsuan |date=2022 |title=Diffusion Models: A Comprehensive Survey of Methods and Applications|class=cs.CV }} One can interpolate between noise and no noise. The amount of noise is denoted $\backslash eta$ ("eta value") in the DDIM paper, with $\backslash eta\; =\; 0$ denoting no noise (as in deterministic DDIM), and $\backslash eta\; =\; 1$ denoting full noise (as in DDPM).

In the perspective of SDE, one can use any of the numerical integration methods, such as Euler–Maruyama method, Heun's method, linear multistep methods, etc. Just as in the discrete case, one can add an adjustable amount of noise during the integration.{{ Cite arXiv | eprint=2406.04329 | last1=Shi | first1=Jiaxin | last2=Han | first2=Kehang | last3=Wang | first3=Zhe | last4=Doucet | first4=Arnaud | last5=Titsias | first5=Michalis K. | title=Simplified and Generalized Masked Diffusion for Discrete Data | date=2024 | class=cs.LG }}

A survey and comparison of samplers in the context of image generation is in.{{ Cite arXiv |eprint=2206.00364v2 |last1=Karras |first1=Tero |last2=Aittala |first2=Miika |last3=Aila |first3=Timo |last4=Laine |first4=Samuli |date=2022 |title=Elucidating the Design Space of Diffusion-Based Generative Models|class=cs.CV }}

Notable variants include{{Cite journal |last1=Cao |first1=Hanqun |last2=Tan |first2=Cheng |last3=Gao |first3=Zhangyang |last4=Xu |first4=Yilun |last5=Chen |first5=Guangyong |last6=Heng |first6=Pheng-Ann |last7=Li |first7=Stan Z. |date=July 2024 |title=A Survey on Generative Diffusion Models |url=https://ieeexplore.ieee.org/document/10419041 |journal=IEEE Transactions on Knowledge and Data Engineering |volume=36 |issue=7 |pages=2814–2830 |doi=10.1109/TKDE.2024.3361474 |issn=1041-4347|url-access=subscription }} Poisson flow generative model,{{Cite journal |last1=Xu |first1=Yilun |last2=Liu |first2=Ziming |last3=Tian |first3=Yonglong |last4=Tong |first4=Shangyuan |last5=Tegmark |first5=Max |last6=Jaakkola |first6=Tommi |date=2023-07-03 |title=PFGM++: Unlocking the Potential of Physics-Inspired Generative Models |url=https://proceedings.mlr.press/v202/xu23m.html |journal=Proceedings of the 40th International Conference on Machine Learning |language=en |publisher=PMLR |pages=38566–38591|arxiv=2302.04265 }} consistency model,{{Cite journal |last1=Song |first1=Yang |last2=Dhariwal |first2=Prafulla |last3=Chen |first3=Mark |last4=Sutskever |first4=Ilya |date=2023-07-03 |title=Consistency Models |url=https://proceedings.mlr.press/v202/song23a |journal=Proceedings of the 40th International Conference on Machine Learning |language=en |publisher=PMLR |pages=32211–32252}} critically-damped Langevin diffusion,{{Cite arXiv |last1=Dockhorn |first1=Tim |last2=Vahdat |first2=Arash |last3=Kreis |first3=Karsten |date=2021-10-06 |title=Score-Based Generative Modeling with Critically-Damped Langevin Diffusion |class=stat.ML |eprint=2112.07068 }} GenPhys,{{cite arXiv |last1=Liu |first1=Ziming |title=GenPhys: From Physical Processes to Generative Models |date=2023-04-05 |eprint=2304.02637 |last2=Luo |first2=Di |last3=Xu |first3=Yilun |last4=Jaakkola |first4=Tommi |last5=Tegmark |first5=Max|class=cs.LG }} cold diffusion,{{Cite journal |last1=Bansal |first1=Arpit |last2=Borgnia |first2=Eitan |last3=Chu |first3=Hong-Min |last4=Li |first4=Jie |last5=Kazemi |first5=Hamid |last6=Huang |first6=Furong |last7=Goldblum |first7=Micah |last8=Geiping |first8=Jonas |last9=Goldstein |first9=Tom |date=2023-12-15 |title=Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise |url=https://proceedings.neurips.cc/paper_files/paper/2023/hash/80fe51a7d8d0c73ff7439c2a2554ed53-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |language=en |volume=36 |pages=41259–41282|arxiv=2208.09392 }} discrete diffusion,{{Cite journal |last1=Gulrajani |first1=Ishaan |last2=Hashimoto |first2=Tatsunori B. |date=2023-12-15 |title=Likelihood-Based Diffusion Language Models |url=https://proceedings.neurips.cc/paper_files/paper/2023/hash/35b5c175e139bff5f22a5361270fce87-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |language=en |volume=36 |pages=16693–16715|arxiv=2305.18619 }}{{cite arXiv |last1=Lou |first1=Aaron |title=Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution |date=2024-06-06 |eprint=2310.16834 |last2=Meng |first2=Chenlin |last3=Ermon |first3=Stefano|class=stat.ML }} etc.

Abstractly speaking, the idea of diffusion model is to take an unknown probability distribution (the distribution of natural-looking images), then progressively convert it to a known probability distribution (standard gaussian distribution), by building an absolutely continuous probability path connecting them. The probability path is in fact defined implicitly by the score function $\backslash nabla\; \backslash ln\; p\_t$.

In denoising diffusion models, the forward process adds noise, and the backward process removes noise. Both the forward and backward processes are SDEs, though the forward process is integrable in closed-form, so it can be done at no computational cost. The backward process is not integrable in closed-form, so it must be integrated step-by-step by standard SDE solvers, which can be very expensive. The probability path in diffusions model is defined through an Itô process and one can retrieve the deterministic process by using the Probability ODE flow formulation.

In flow-based diffusion models, the forward process is a deterministic flow along a time-dependent vector field, and the backward process is also a deterministic flow along the same vector field, but going backwards. Both processes are solutions to ODEs. If the vector field is well-behaved, the ODE will also be well-behaved.

Given two distributions $\backslash pi\_0$ and $\backslash pi\_1$, a flow-based model is a time-dependent velocity field $v\_t(x)$ in $[0,1]\; \backslash times\; \backslash mathbb\; R^d$, such that if we start by sampling a point $x\; \backslash sim\; \backslash pi\_0$, and let it move according to the velocity field:

$$\backslash frac\{d\}\{dt\}\; \backslash phi\_t(x)\; =\; v\_t(\backslash phi\_t(x))\; \backslash quad\; t\; \backslash in\; [0,1],\; \backslash quad\; \backslash text\{starting\; from\; \}\backslash phi\_0(x)\; =\; x$$

we end up with a point $x\_1\; \backslash sim\; \backslash pi\_1$. The solution $\backslash phi\_t$ of the above ODE define a probability path $p\_t\; =\; [\backslash phi\_t]\_\{\backslash \#\}\; \backslash pi\_0$ by the pushforward measure operator. In particular, $[\backslash phi\_1]\_\{\backslash \#\}\; \backslash pi\_0\; =\; \backslash pi\_1$.

The probability path and the velocity field also satisfy the continuity equation, in the sense of probability distribution:

$$\backslash partial\_t\; p\_t\; +\; \backslash nabla\; \backslash cdot\; (v\_t\; p\_t)\; =\; 0$$

To construct a probability path, we start by construct a conditional probability path $p\_t(x\; \backslash vert\; z)$ and the corresponding conditional velocity field $v\_t(x\; \backslash vert\; z)$ on some conditional distribution $q(z)$. A natural choice is the Gaussian conditional probability path:

$$p\_t(x\; \backslash vert\; z)\; =\; \backslash mathcal\{N\}\; \backslash left(\; m\_t(z),\; \backslash zeta\_t^2\; I\; \backslash right)$$

The conditional velocity field which corresponds to the geodesic path between conditional Gaussian path is

$$v\_t(x\; \backslash vert\; z)\; =\; \backslash frac\{\backslash zeta\_t\text{'}\}\{\backslash zeta\_t\}\; (x\; -\; m\_t(z))\; +\; m\_t\text{'}(z)$$

The probability path and velocity field are then computed by marginalizing

$p\_t(x)\; =\; \backslash int\; p\_t(x\; \backslash vert\; z)\; q(z)\; dz\; \backslash qquad\; \backslash text\{\; and\; \}\; \backslash qquad\; v\_t(x)\; =\; \backslash mathbb\{E\}\_\{q(z)\}\; \backslash left[\backslash frac\{v\_t(x\; \backslash vert\; z)\; p\_t(x\; \backslash vert\; z)\}\{p\_t(x)\}\; \backslash right]$

The idea of optimal transport flow {{Cite journal |last1=Tong |first1=Alexander |last2=Fatras |first2=Kilian |last3=Malkin |first3=Nikolay |last4=Huguet |first4=Guillaume |last5=Zhang |first5=Yanlei |last6=Rector-Brooks |first6=Jarrid |last7=Wolf |first7=Guy |last8=Bengio |first8=Yoshua |date=2023-11-08 |title=Improving and generalizing flow-based generative models with minibatch optimal transport |url=https://openreview.net/forum?id=CD9Snc73AW |journal=Transactions on Machine Learning Research |arxiv=2302.00482 |language=en |issn=2835-8856}} is to construct a probability path minimizing the Wasserstein metric. The distribution on which we condition is an approximation of the optimal transport plan between $\backslash pi\_0$ and $\backslash pi\_1$

: $z\; =\; (x\_0,\; x\_1)$ and $q(z)\; =\; \backslash Gamma(\backslash pi\_0,\; \backslash pi\_1)$, where $\backslash Gamma$ is the optimal transport plan, which can be approximated by mini-batch optimal transport. If the batch size is not large, then the transport it computes can be very far from the true optimal transport.

The idea of rectified flow{{cite arXiv|last1=Liu |first1=Xingchao |title=Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow |date=2022-09-07 |eprint=2209.03003 |last2=Gong |first2=Chengyue |last3=Liu |first3=Qiang|class=cs.LG }}{{cite arXiv |last=Liu |first=Qiang |title=Rectified Flow: A Marginal Preserving Approach to Optimal Transport |date=2022-09-29 |class=stat.ML |eprint=2209.14577}} is to learn a flow model such that the velocity is nearly constant along each flow path. This is beneficial, because we can integrate along such a vector field with very few steps. For example, if an ODE $\backslash dot\{\backslash phi\_t\}(x)\; =\; v\_t(\backslash phi\_t(x))$ follows perfectly straight paths, it simplifies to $\backslash phi\_t(x)\; =\; x\_0\; +\; t\; \backslash cdot\; v\_0(x\_0)$, allowing for exact solutions in one step. In practice, we cannot reach such perfection, but when the flow field is nearly so, we can take a few large steps instead of many little steps.

The general idea is to start with two distributions $\backslash pi\_0$ and $\backslash pi\_1$, then construct a flow field $\backslash phi^0\; =\; \backslash \{\backslash phi\_t:\; t\backslash in[0,1]\backslash \}$ from it, then repeatedly apply a "reflow" operation to obtain successive flow fields $\backslash phi^1,\; \backslash phi^2,\; \backslash dots$, each straighter than the previous one. When the flow field is straight enough for the application, we stop.

Generally, for any time-differentiable process $\backslash phi\_t$, $v\_t$ can be estimated by solving:

$$\backslash min\_\{\backslash theta\}\; \backslash int\_0^1\; \backslash mathbb\{E\}\_\{x\; \backslash sim\; p\_t\}\backslash left\; [\backslash lVert\{v\_t(x,\; \backslash theta)\; -\; v\_t(x)\}\backslash rVert^2\backslash right]\; \backslash ,\backslash mathrm\{d\}t.$$

In rectified flow, by injecting strong priors that intermediate trajectories are straight, it can achieve both theoretical relevance for optimal transport and computational efficiency, as ODEs with straight paths can be simulated precisely without time discretization.

File:Rectified Flow.png

Specifically, rectified flow seeks to match an ODE with the marginal distributions of the linear interpolation between points from distributions $\backslash pi\_0$ and $\backslash pi\_1$. Given observations $x\_0\; \backslash sim\; \backslash pi\_0$ and $x\_1\; \backslash sim\; \backslash pi\_1$, the canonical linear interpolation $x\_t=\; t\; x\_1\; +\; (1-t)x\_0,\; t\backslash in\; [0,1]$ yields a trivial case $\backslash dot\{x\}\_t\; =\; x\_1\; -\; x\_0$, which cannot be causally simulated without $x\_1$. To address this, $x\_t$ is "projected" into a space of causally simulatable ODEs, by minimizing the least squares loss with respect to the direction $x\_1\; -\; x\_0$:

$$\backslash min\_\{\backslash theta\}\; \backslash int\_0^1\; \backslash mathbb\{E\}\_\{\backslash pi\_0,\; \backslash pi\_1,\; p\_t\}\backslash left\; [\backslash lVert\{(x\_1-x\_0)\; -\; v\_t(x\_t)\}\backslash rVert^2\backslash right]\; \backslash ,\backslash mathrm\{d\}t.$$

The data pair $(x\_0,\; x\_1)$ can be any coupling of $\backslash pi\_0$ and $\backslash pi\_1$, typically independent (i.e., $(x\_0,x\_1)\; \backslash sim\; \backslash pi\_0\; \backslash times\; \backslash pi\_1$) obtained by randomly combining observations from $\backslash pi\_0$ and $\backslash pi\_1$. This process ensures that the trajectories closely mirror the density map of $x\_t$ trajectories but reroute at intersections to ensure causality.

File:Reflow Illustration.png

A distinctive aspect of rectified flow is its capability for "reflow", which straightens the trajectory of ODE paths. Denote the rectified flow $\backslash phi^0\; =\; \backslash \{\backslash phi\_t:\; t\backslash in[0,1]\backslash \}$ induced from $(x\_0,x\_1)$ as $\backslash phi^0\; =\; \backslash mathsf\{Rectflow\}((x\_0,x\_1))$. Recursively applying this $\backslash mathsf\{Rectflow\}(\backslash cdot)$ operator generates a series of rectified flows $\backslash phi^\{k+1\}\; =\; \backslash mathsf\{Rectflow\}((\backslash phi\_0^k(x\_0),\; \backslash phi\_1^k(x\_1)))$. This "reflow" process not only reduces transport costs but also straightens the paths of rectified flows, making $\backslash phi^k$ paths straighter with increasing $k$.

Rectified flow includes a nonlinear extension where linear interpolation $x\_t$ is replaced with any time-differentiable curve that connects $x\_0$ and $x\_1$, given by $x\_t\; =\; \backslash alpha\_t\; x\_1\; +\; \backslash beta\_t\; x\_0$. This framework encompasses DDIM and probability flow ODEs as special cases, with particular choices of $\backslash alpha\_t$ and $\backslash beta\_t$. However, in the case where the path of $x\_t$ is not straight, the reflow process no longer ensures a reduction in convex transport costs, and also no longer straighten the paths of $\backslash phi\_t$.

File:Diffusion Architecture.png

File:X-Y plot of algorithmically-generated AI art of European-style castle in Japan demonstrating DDIM diffusion steps.png

For generating images by DDPM, we need a neural network that takes a time $t$ and a noisy image $x\_t$, and predicts a noise $\backslash epsilon\_\backslash theta(x\_t,\; t)$ from it. Since predicting the noise is the same as predicting the denoised image, then subtracting it from $x\_t$, denoising architectures tend to work well. For example, the U-Net, which was found to be good for denoising images, is often used for denoising diffusion models that generate images.{{Cite journal |last1=Ho |first1=Jonathan |last2=Saharia |first2=Chitwan |last3=Chan |first3=William |last4=Fleet |first4=David J. |last5=Norouzi |first5=Mohammad |last6=Salimans |first6=Tim |date=2022-01-01 |title=Cascaded diffusion models for high fidelity image generation |url=https://dl.acm.org/doi/abs/10.5555/3586589.3586636 |journal=The Journal of Machine Learning Research |volume=23 |issue=1 |pages=47:2249–47:2281 |arxiv=2106.15282 |issn=1532-4435}}

{{Anchor|Diffusion Transformer|DiT}}For DDPM, the underlying architecture ("backbone") does not have to be a U-Net. It just has to predict the noise somehow. For example, the diffusion transformer (DiT) uses a Transformer to predict the mean and diagonal covariance of the noise, given the textual conditioning and the partially denoised image. It is the same as standard U-Net-based denoising diffusion model, with a Transformer replacing the U-Net.{{Cite arXiv |eprint=2212.09748v2 |last1=Peebles |first1=William |last2=Xie |first2=Saining |date=March 2023 |title=Scalable Diffusion Models with Transformers |class=cs.CV |language=en}} Mixture of experts-Transformer can also be applied.{{cite arXiv |last1=Fei |first1=Zhengcong |title=Scaling Diffusion Transformers to 16 Billion Parameters |date=2024-07-16 |eprint=2407.11633 |last2=Fan |first2=Mingyuan |last3=Yu |first3=Changqian |last4=Li |first4=Debang |last5=Huang |first5=Junshi|class=cs.CV }}

DDPM can be used to model general data distributions, not just natural-looking images. For example, Human Motion Diffusion{{Cite arXiv |eprint=2209.14916 |last1=Tevet |first1=Guy |last2=Raab |first2=Sigal |last3=Gordon |first3=Brian |last4=Shafir |first4=Yonatan |last5=Cohen-Or |first5=Daniel |last6=Bermano |first6=Amit H. |date=2022 |title=Human Motion Diffusion Model|class=cs.CV }} models human motion trajectory by DDPM. Each human motion trajectory is a sequence of poses, represented by either joint rotations or positions. It uses a Transformer network to generate a less noisy trajectory out of a noisy one.

The base diffusion model can only generate unconditionally from the whole distribution. For example, a diffusion model learned on ImageNet would generate images that look like a random image from ImageNet. To generate images from just one category, one would need to impose the condition, and then sample from the conditional distribution. Whatever condition one wants to impose, one needs to first convert the conditioning into a vector of floating point numbers, then feed it into the underlying diffusion model neural network. However, one has freedom in choosing how to convert the conditioning into a vector.

Stable Diffusion, for example, imposes conditioning in the form of cross-attention mechanism, where the query is an intermediate representation of the image in the U-Net, and both key and value are the conditioning vectors. The conditioning can be selectively applied to only parts of an image, and new kinds of conditionings can be finetuned upon the base model, as used in ControlNet.{{Cite arXiv |last1=Zhang |first1=Lvmin |last2=Rao |first2=Anyi |last3=Agrawala |first3=Maneesh |date=2023 |title=Adding Conditional Control to Text-to-Image Diffusion Models |class=cs.CV |eprint=2302.05543}}

As a particularly simple example, consider image inpainting. The conditions are $\backslash tilde\; x$, the reference image, and $m$, the inpainting mask. The conditioning is imposed at each step of the backward diffusion process, by first sampling $\backslash tilde\; x\_t\; \backslash sim\; N\backslash left(\backslash sqrt\{\backslash bar\backslash alpha\_t\}\; \backslash tilde\; x,\; \backslash sigma\_\{t\}^2\; I\; \backslash right)$, a noisy version of $\backslash tilde\; x$, then replacing $x\_t$ with $(1-m)\; \backslash odot\; x\_t\; +\; m\; \backslash odot\; \backslash tilde\; x\_t$, where $\backslash odot$ means elementwise multiplication.{{Cite arXiv |eprint=2201.09865v4 |last1=Lugmayr |first1=Andreas |last2=Danelljan |first2=Martin |last3=Romero |first3=Andres |last4=Yu |first4=Fisher |last5=Timofte |first5=Radu |last6=Van Gool |first6=Luc |date=2022 |title=RePaint: Inpainting Using Denoising Diffusion Probabilistic Models |class=cs.CV |language=en}} Another application of cross-attention mechanism is prompt-to-prompt image editing.{{cite arXiv |last1=Hertz |first1=Amir |title=Prompt-to-Prompt Image Editing with Cross Attention Control |date=2022-08-02 |eprint=2208.01626 |last2=Mokady |first2=Ron |last3=Tenenbaum |first3=Jay |last4=Aberman |first4=Kfir |last5=Pritch |first5=Yael |last6=Cohen-Or |first6=Daniel|class=cs.CV }}

Conditioning is not limited to just generating images from a specific category, or according to a specific caption (as in text-to-image). For example, demonstrated generating human motion, conditioned on an audio clip of human walking (allowing syncing motion to a soundtrack), or video of human running, or a text description of human motion, etc. For how conditional diffusion models are mathematically formulated, see a methodological summary in.{{cite arXiv |last1=Zhao |first1=Zheng |last2=Luo |first2=Ziwei |last3=Sjölund |first3=Jens |last4=Schön |first4=Thomas B. |title=Conditional sampling within generative diffusion models |eprint=2409.09650 |class=stat.ML |date=2024}}

As generating an image takes a long time, one can try to generate a small image by a base diffusion model, then upscale it by other models. Upscaling can be done by GAN,{{Cite conference |last1=Wang |first1=Xintao |last2=Xie |first2=Liangbin |last3=Dong |first3=Chao |last4=Shan |first4=Ying |date=2021 |title=Real-ESRGAN: Training Real-World Blind Super-Resolution With Pure Synthetic Data |url=https://openaccess.thecvf.com/content/ICCV2021W/AIM/papers/Wang_Real-ESRGAN_Training_Real-World_Blind_Super-Resolution_With_Pure_Synthetic_Data_ICCVW_2021_paper.pdf |conference=International Conference on Computer Vision |book-title=Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021 |language=en |pages=1905–1914|arxiv=2107.10833 }} Transformer,{{Cite conference |last1=Liang |first1=Jingyun |last2=Cao |first2=Jiezhang |last3=Sun |first3=Guolei |last4=Zhang |first4=Kai |last5=Van Gool |first5=Luc |last6=Timofte |first6=Radu |date=2021 |title=SwinIR: Image Restoration Using Swin Transformer |url=https://openaccess.thecvf.com/content/ICCV2021W/AIM/papers/Liang_SwinIR_Image_Restoration_Using_Swin_Transformer_ICCVW_2021_paper.pdf |book-title=Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops |conference=International Conference on Computer Vision, 2021 |language=en |pages=1833–1844|arxiv=2108.10257v1 }} or signal processing methods like Lanczos resampling.

Diffusion models themselves can be used to perform upscaling. Cascading diffusion model stacks multiple diffusion models one after another, in the style of Progressive GAN. The lowest level is a standard diffusion model that generate 32x32 image, then the image would be upscaled by a diffusion model specifically trained for upscaling, and the process repeats.

In more detail, the diffusion upscaler is trained as follows:

• Sample $(x\_0,\; z\_0,\; c)$, where $x\_0$ is the high-resolution image, $z\_0$ is the same image but scaled down to a low-resolution, and $c$ is the conditioning, which can be the caption of the image, the class of the image, etc.
• Sample two white noises $\backslash epsilon\_x,\; \backslash epsilon\_z$, two time-steps $t\_x,\; t\_z$. Compute the noisy versions of the high-resolution and low-resolution images: $\backslash begin\{cases\}$

x_{t_x} &= \sqrt{\bar\alpha_{t_x}} x_0 + \sigma_{t_x} \epsilon_x\\

z_{t_z} &= \sqrt{\bar\alpha_{t_z}} z_0 + \sigma_{t_z} \epsilon_z

\end{cases} .

• Train the denoising network to predict $\backslash epsilon\_x$ given $x\_\{t\_x\},\; z\_\{t\_z\},\; t\_x,\; t\_z,\; c$. That is, apply gradient descent on $\backslash theta$ on the L2 loss $\backslash |\; \backslash epsilon\_\backslash theta(x\_\{t\_x\},\; z\_\{t\_z\},\; t\_x,\; t\_z,\; c)\; -\; \backslash epsilon\_x\; \backslash |\_2^2$.

This section collects some notable diffusion models, and briefly describes their architecture.

{{Main|DALL-E|Sora (text-to-video model)}}

The DALL-E series by OpenAI are text-conditional diffusion models of images.

The first version of DALL-E (2021) is not actually a diffusion model. Instead, it uses a Transformer architecture that autoregressively generates a sequence of tokens, which is then converted to an image by the decoder of a discrete VAE. Released with DALL-E was the CLIP classifier, which was used by DALL-E to rank generated images according to how close the image fits the text.

GLIDE (2022-03){{Cite arXiv |eprint=2112.10741 |class=cs.CV |first1=Alex |last1=Nichol |first2=Prafulla |last2=Dhariwal |title=GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models |date=2022-03-08 |last3=Ramesh |first3=Aditya |last4=Shyam |first4=Pranav |last5=Mishkin |first5=Pamela |last6=McGrew |first6=Bob |last7=Sutskever |first7=Ilya |last8=Chen |first8=Mark}} is a 3.5-billion diffusion model, and a small version was released publicly.{{Citation |title=GLIDE |date=2023-09-22 |url=https://github.com/openai/glide-text2im |access-date=2023-09-24 |publisher=OpenAI}} Soon after, DALL-E 2 was released (2022-04).{{Cite arXiv |eprint=2204.06125 |class=cs.CV |first1=Aditya |last1=Ramesh |first2=Prafulla |last2=Dhariwal |title=Hierarchical Text-Conditional Image Generation with CLIP Latents |date=2022-04-12 |last3=Nichol |first3=Alex |last4=Chu |first4=Casey |last5=Chen |first5=Mark}} DALL-E 2 is a 3.5-billion cascaded diffusion model that generates images from text by "inverting the CLIP image encoder", the technique which they termed "unCLIP".

The unCLIP method contains 4 models: a CLIP image encoder, a CLIP text encoder, an image decoder, and a "prior" model (which can be a diffusion model, or an autoregressive model). During training, the prior model is trained to convert CLIP image encodings to CLIP text encodings. The image decoder is trained to convert CLIP image encodings back to images. During inference, a text is converted by the CLIP text encoder to a vector, then it is converted by the prior model to an image encoding, then it is converted by the image decoder to an image.

Sora (2024-02) is a diffusion Transformer model (DiT).

{{Main|Stable Diffusion}}

Stable Diffusion (2022-08), released by Stability AI, consists of a denoising latent diffusion model (860 million parameters), a VAE, and a text encoder. The denoising network is a U-Net, with cross-attention blocks to allow for conditional image generation.{{Cite web |last=Alammar |first=Jay |title=The Illustrated Stable Diffusion |url=https://jalammar.github.io/illustrated-stable-diffusion/ |access-date=2022-10-31 |website=jalammar.github.io}}

Stable Diffusion 3 (2024-03){{cite arXiv |last1=Esser |first1=Patrick |title=Scaling Rectified Flow Transformers for High-Resolution Image Synthesis |date=2024-03-05 |eprint=2403.03206 |last2=Kulal |first2=Sumith |last3=Blattmann |first3=Andreas |last4=Entezari |first4=Rahim |last5=Müller |first5=Jonas |last6=Saini |first6=Harry |last7=Levi |first7=Yam |last8=Lorenz |first8=Dominik |last9=Sauer |first9=Axel|class=cs.CV }} changed the latent diffusion model from the UNet to a Transformer model, and so it is a DiT. It uses rectified flow.

Stable Video 4D (2024-07){{cite arXiv |last1=Xie |first1=Yiming |title=SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency |date=2024-07-24 |eprint=2407.17470 |last2=Yao |first2=Chun-Han |last3=Voleti |first3=Vikram |last4=Jiang |first4=Huaizu |last5=Jampani |first5=Varun|class=cs.CV }} is a latent diffusion model for videos of 3D objects.

Imagen (2022){{Cite web |title=Imagen: Text-to-Image Diffusion Models |url=https://imagen.research.google/ |access-date=2024-04-04 |website=imagen.research.google}}{{Cite journal |last1=Saharia |first1=Chitwan |last2=Chan |first2=William |last3=Saxena |first3=Saurabh |last4=Li |first4=Lala |last5=Whang |first5=Jay |last6=Denton |first6=Emily L. |last7=Ghasemipour |first7=Kamyar |last8=Gontijo Lopes |first8=Raphael |last9=Karagol Ayan |first9=Burcu |last10=Salimans |first10=Tim |last11=Ho |first11=Jonathan |last12=Fleet |first12=David J. |last13=Norouzi |first13=Mohammad |date=2022-12-06 |title=Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding |url=https://proceedings.neurips.cc/paper_files/paper/2022/hash/ec795aeadae0b7d230fa35cbaf04c041-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |language=en |volume=35 |pages=36479–36494|arxiv=2205.11487 }} uses a T5-XXL language model to encode the input text into an embedding vector. It is a cascaded diffusion model with three sub-models. The first step denoises a white noise to a 64×64 image, conditional on the embedding vector of the text. This model has 2B parameters. The second step upscales the image by 64×64→256×256, conditional on embedding. This model has 650M parameters. The third step is similar, upscaling by 256×256→1024×1024. This model has 400M parameters. The three denoising networks are all U-Nets.

Muse (2023-01){{cite arXiv |last1=Chang |first1=Huiwen |title=Muse: Text-To-Image Generation via Masked Generative Transformers |date=2023-01-02 |eprint=2301.00704 |last2=Zhang |first2=Han |last3=Barber |first3=Jarred |last4=Maschinot |first4=A. J. |last5=Lezama |first5=Jose |last6=Jiang |first6=Lu |last7=Yang |first7=Ming-Hsuan |last8=Murphy |first8=Kevin |last9=Freeman |first9=William T.|class=cs.CV }} is not a diffusion model, but an encoder-only Transformer that is trained to predict masked image tokens from unmasked image tokens.

Imagen 2 (2023-12) is also diffusion-based. It can generate images based on a prompt that mixes images and text. No further information available.{{Cite web |title=Imagen 2 - our most advanced text-to-image technology |url=https://deepmind.google/technologies/imagen-2/ |access-date=2024-04-04 |website=Google DeepMind |language=en}} Imagen 3 (2024-05) is too. No further information available.{{Citation |last1=Imagen-Team-Google |title=Imagen 3 |date=2024-12-13 |arxiv=2408.07009 |last2=Baldridge |first2=Jason |last3=Bauer |first3=Jakob |last4=Bhutani |first4=Mukul |last5=Brichtova |first5=Nicole |last6=Bunner |first6=Andrew |last7=Castrejon |first7=Lluis |last8=Chan |first8=Kelvin |last9=Chen |first9=Yichang}}

Veo (2024) generates videos by latent diffusion. The diffusion is conditioned on a vector that encodes both a text prompt and an image prompt.{{Cite web |date=2024-05-14 |title=Veo |url=https://deepmind.google/technologies/veo/ |access-date=2024-05-17 |website=Google DeepMind |language=en}}

Make-A-Video (2022) is a text-to-video diffusion model.{{Cite web |url=https://ai.meta.com/blog/generative-ai-text-to-video/ |access-date=2024-09-20|title=Introducing Make-A-Video: An AI system that generates videos from text |website=ai.meta.com}}{{cite arXiv |last1=Singer |first1=Uriel |title=Make-A-Video: Text-to-Video Generation without Text-Video Data |date=2022-09-29 |eprint=2209.14792 |last2=Polyak |first2=Adam |last3=Hayes |first3=Thomas |last4=Yin |first4=Xi |last5=An |first5=Jie |last6=Zhang |first6=Songyang |last7=Hu |first7=Qiyuan |last8=Yang |first8=Harry |last9=Ashual |first9=Oron|class=cs.CV }}

CM3leon (2023) is not a diffusion model, but an autoregressive causally masked Transformer, with mostly the same architecture as LLaMa-2.{{Cite web |url=https://ai.meta.com/blog/generative-ai-text-images-cm3leon/ |title=Introducing CM3leon, a more efficient, state-of-the-art generative model for text and images|access-date=2024-09-20 |website=ai.meta.com}}{{cite arXiv |last=Chameleon Team |title=Chameleon: Mixed-Modal Early-Fusion Foundation Models |date=2024-05-16 |class=cs.CL |eprint=2405.09818}}

File:Transfusion_(2024)_Fig_1,_architectural_diagram.png

Transfusion (2024) is a Transformer that combines autoregressive text generation and denoising diffusion. Specifically, it generates text autoregressively (with causal masking), and generates images by denoising multiple times over image tokens (with all-to-all attention).{{Cite arXiv |last1=Zhou |first1=Chunting |last2=Yu |first2=Lili |last3=Babu |first3=Arun |last4=Tirumala |first4=Kushal |last5=Yasunaga |first5=Michihiro |last6=Shamis |first6=Leonid |last7=Kahn |first7=Jacob |last8=Ma |first8=Xuezhe |last9=Zettlemoyer |first9=Luke |date=2024-08-20 |title=Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model |class=cs.AI |eprint=2408.11039 |language=en}}

Movie Gen (2024) is a series of Diffusion Transformers operating on latent space and by flow matching.[https://ai.meta.com/static-resource/movie-gen-research-paper Movie Gen: A Cast of Media Foundation Models], The Movie Gen team @ Meta, October 4, 2024.

• Diffusion process
• Markov chain
• Variational inference
• Variational autoencoder

• Review papers
• {{Citation |last=Yang |first=Ling |title=YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy |date=2024-09-06 |url=https://github.com/YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy |access-date=2024-09-06}}
• {{Cite journal |last1=Yang |first1=Ling |last2=Zhang |first2=Zhilong |last3=Song |first3=Yang |last4=Hong |first4=Shenda |last5=Xu |first5=Runsheng |last6=Zhao |first6=Yue |last7=Zhang |first7=Wentao |last8=Cui |first8=Bin |last9=Yang |first9=Ming-Hsuan |date=2023-11-09 |title=Diffusion Models: A Comprehensive Survey of Methods and Applications |url=https://dl.acm.org/doi/abs/10.1145/3626235 |journal=ACM Comput. Surv. |volume=56 |issue=4 |pages=105:1–105:39 |doi=10.1145/3626235 |issn=0360-0300|arxiv=2209.00796 }}
• {{ Cite arXiv | eprint=2107.03006 | last1=Austin | first1=Jacob | last2=Johnson | first2=Daniel D. | last3=Ho | first3=Jonathan | last4=Tarlow | first4=Daniel | author5=Rianne van den Berg | title=Structured Denoising Diffusion Models in Discrete State-Spaces | date=2021 | class=cs.LG }}
• {{Cite journal |last1=Croitoru |first1=Florinel-Alin |last2=Hondru |first2=Vlad |last3=Ionescu |first3=Radu Tudor |last4=Shah |first4=Mubarak |date=2023-09-01 |title=Diffusion Models in Vision: A Survey |url=https://ieeexplore.ieee.org/document/10081412 |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=45 |issue=9 |pages=10850–10869 |doi=10.1109/TPAMI.2023.3261988 |pmid=37030794 |issn=0162-8828|arxiv=2209.04747 }}
• Mathematical details omitted in the article.
• {{Cite web |date=2022-09-25 |title=Power of Diffusion Models |url=https://astralord.github.io/posts/power-of-diffusion-models/ |access-date=2023-09-25 |website=AstraBlog |language=en}}
• {{Cite arXiv |last=Luo |first=Calvin |date=2022-08-25 |title=Understanding Diffusion Models: A Unified Perspective |class=cs.LG |eprint=2208.11970}}
• {{Cite web |last=Weng |first=Lilian |date=2021-07-11 |title=What are Diffusion Models? |url=https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ |access-date=2023-09-25 |website=lilianweng.github.io |language=en}}
• Tutorials
• {{Cite arXiv |eprint=2406.08929 |first1=Preetum |last1=Nakkiran |first2=Arwen |last2=Bradley |title=Step-by-Step Diffusion: An Elementary Tutorial |date=2024 |last3=Zhou |first3=Hattie |last4=Advani |first4=Madhu|class=cs.LG }}
• {{Cite web |url=https://benanne.github.io/2022/05/26/guidance.html |title=Guidance: a cheat code for diffusion models|date=26 May 2022 }} Overview of classifier guidance and classifier-free guidance, light on mathematical details.

{{reflist}}

Category:Markov models

Category:Machine learning algorithms