latent diffusion model

Diffusion models were introduced in 2015 as a method to learn a model that can sample from a highly complex probability distribution. They used techniques from non-equilibrium thermodynamics, especially diffusion.{{Cite journal |last1=Sohl-Dickstein |first1=Jascha |last2=Weiss |first2=Eric |last3=Maheswaranathan |first3=Niru |last4=Ganguli |first4=Surya |date=2015-06-01 |title=Deep Unsupervised Learning using Nonequilibrium Thermodynamics |url=http://proceedings.mlr.press/v37/sohl-dickstein15.pdf |journal=Proceedings of the 32nd International Conference on Machine Learning |language=en |publisher=PMLR |volume=37 |pages=2256–2265|arxiv=1503.03585 }} It was accompanied by a software implementation in Theano.{{cite web |last=Sohl-Dickstein |first=Jascha |title=Sohl-Dickstein/Diffusion-Probabilistic-Models |website=GitHub |date=2024-09-01 |url=https://github.com/Sohl-Dickstein/Diffusion-Probabilistic-Models |access-date=2024-09-07}}

A 2019 paper proposed the noise conditional score network (NCSN) or score-matching with Langevin dynamics (SMLD).{{cite web |title=ermongroup/ncsn |date=2019 |url=https://github.com/ermongroup/ncsn |access-date=2024-09-07 |publisher=ermongroup}} The paper was accompanied by a software package written in PyTorch release on GitHub.{{Cite journal |last1=Song |first1=Yang |last2=Ermon |first2=Stefano |date=2019 |title=Generative Modeling by Estimating Gradients of the Data Distribution |url=https://proceedings.neurips.cc/paper/2019/hash/3001ef257407d5a371a96dcd947c7d93-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=32|arxiv=1907.05600 }}

A 2020 paper{{Cite journal |last1=Ho |first1=Jonathan |last2=Jain |first2=Ajay |last3=Abbeel |first3=Pieter |date=2020 |title=Denoising Diffusion Probabilistic Models |url=https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=33 |pages=6840–6851}} proposed the Denoising Diffusion Probabilistic Model (DDPM), which improves upon the previous method by variational inference. The paper was accompanied by a software package written in TensorFlow release on GitHub.{{cite web |last=Ho |first=Jonathan |title=hojonathanho/diffusion |website=GitHub |date=Jun 20, 2020 |url=https://github.com/hojonathanho/diffusion |access-date=2024-09-07}} It was reimplemented in PyTorch by lucidrains.{{cite web |last=Wang |first=Phil |title=lucidrains/denoising-diffusion-pytorch |website=GitHub |date=2024-09-07 |url=https://github.com/lucidrains/denoising-diffusion-pytorch |access-date=2024-09-07}}{{Cite web |title=The Annotated Diffusion Model |url=https://huggingface.co/blog/annotated-diffusion |access-date=2024-09-07 |website=huggingface.co}}

On December 20, 2021, the LDM paper was published on arXiv,{{cite arXiv |last1=Rombach |first1=Robin |title=High-Resolution Image Synthesis with Latent Diffusion Models |date=2021-12-20 |eprint=2112.10752 |last2=Blattmann |first2=Andreas |last3=Lorenz |first3=Dominik |last4=Esser |first4=Patrick |last5=Ommer |first5=Björn|class=cs.CV }} and both Stable Diffusion{{Cite web |title=Update README.md · CompVis/stable-diffusion@17e64e3 |url=https://github.com/CompVis/stable-diffusion/commit/17e64e320556d49e429f41a7093b291c49255470 |access-date=2024-09-07 |website=GitHub |language=en}} and LDM{{Cite web |title=Update README.md · CompVis/latent-diffusion@17e64e3 |url=https://github.com/CompVis/latent-diffusion/commit/17e64e320556d49e429f41a7093b291c49255470 |access-date=2024-09-07 |website=GitHub |language=en}} repositories were published on GitHub. However, they remained roughly the same. Substantial information concerning Stable Diffusion v1 was only added to GitHub on August 10, 2022.{{Cite web |title=stable diffusion · CompVis/stable-diffusion@2ff270f |url=https://github.com/CompVis/stable-diffusion/commit/2ff270f4e0c884d9684fa038f6d84d8600a94b39 |access-date=2024-09-07 |website=GitHub |language=en}}

All of Stable Diffusion (SD) versions 1.1 to XL were particular instantiations of the LDM architecture.

SD 1.1 to 1.4 were released by CompVis in August 2022. There is no "version 1.0". SD 1.1 was a LDM trained on the laion2B-en dataset. SD 1.1 was finetuned to 1.2 on more aesthetic images. SD 1.2 was finetuned to 1.3, 1.4 and 1.5, with 10% of text-conditioning dropped, to improve classifier-free guidance.{{Cite web |date=2023-08-23 |title=CompVis (CompVis) |url=https://huggingface.co/CompVis |access-date=2024-03-06 |website=huggingface.co}} SD 1.5 was released by RunwayML in October 2022.{{Cite web |title=runwayml/stable-diffusion-v1-5 · Hugging Face |url=https://huggingface.co/runwayml/stable-diffusion-v1-5 |url-status=live |archive-url=https://web.archive.org/web/20230921025150/https://huggingface.co/runwayml/stable-diffusion-v1-5 |archive-date=September 21, 2023 |access-date=2023-08-17 |website=huggingface.co}}

While the LDM can work for generating arbitrary data conditional on arbitrary data, for concreteness, we describe its operation in conditional text-to-image generation.

LDM consists of a variational autoencoder (VAE), a modified U-Net, and a text encoder.

The VAE encoder compresses the image from pixel space to a smaller dimensional latent space, capturing a more fundamental semantic meaning of the image. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output from forward diffusion backwards to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space.

The denoising step can be conditioned on a string of text, an image, or another modality. The encoded conditioning data is exposed to denoising U-Nets via a cross-attention mechanism. For conditioning on text, the fixed, a pretrained CLIP ViT-L/14 text encoder is used to transform text prompts to an embedding space.

To compress the image data, a variational autoencoder (VAE) is first trained on a dataset of images. The encoder part of the VAE takes an image as input and outputs a lower-dimensional latent representation of the image. This latent representation is then used as input to the U-Net. Once the model is trained, the encoder is used to encode images into latent representations, and the decoder is used to decode latent representations back into images.

Let the encoder and the decoder of the VAE be $E,\; D$.

To encode an RGB image, its three channels are divided by the maximum value, resulting in a tensor $x$ of shape $(3,\; 512,\; 512)$ with all entries within range $[0,\; 1]$. The encoded vector is $0.18215\; \backslash times\; E(2x\; -\; 1)$, with shape $(4,\; 64,\; 64)$, where 0.18215 is a hyperparameter, which the original authors picked to roughly whiten the encoded vector to roughly unit variance. Conversely, given a latent tensor $y$, the decoded image is $(D(y\; /\; 0.18125)\; +\; 1)/2$, then clipped to the range $[0,\; 1]$.{{Cite web |title=Explanation of the 0.18215 factor in textual_inversion? · Issue #437 · huggingface/diffusers |url=https://github.com/huggingface/diffusers/issues/437 |access-date=2024-09-19 |website=GitHub |language=en}}{{Cite web |title=diffusion-nbs/Stable Diffusion Deep Dive.ipynb at master · fastai/diffusion-nbs |url=https://github.com/fastai/diffusion-nbs/blob/master/Stable%20Diffusion%20Deep%20Dive.ipynb |access-date=2024-09-19 |website=GitHub |language=en}}

In the implemented version,{{Pg|location=ldm/models/autoencoder.py}} the encoder is a convolutional neural network (CNN) with a single self-attention mechanism near the end. It takes a tensor of shape $(3,\; H,\; W)$ and outputs a tensor of shape $(8,\; H/8,\; W/8)$, being the concatenation of the predicted mean and variance of the latent vector, each of shape $(4,\; H/8,\; W/8)$. The variance is used in training, but after training, usually only the mean is taken, with the variance discarded.

The decoder is also a CNN with a single self-attention mechanism near the end. It takes a tensor of shape $(4,\; H/8,\; W/8)$ and outputs a tensor of shape $(3,\; H,\; W)$.

The U-Net backbone takes the following kinds of inputs:

• A latent image array, produced by the VAE encoder. It has dimensions $(\backslash text\{channel\},\; \backslash text\{width\},\; \backslash text\{height\})$. Typically, $(\backslash text\{channel\},\; \backslash text\{width\},\; \backslash text\{height\})\; =\; (4,\; 64,\; 64)$.
• A timestep-embedding vector, which tells the backbone how much noise there is in the image. For example, an embedding of timestep $t\; =\; 0$ would indicate that the input image is already noiseless, while $t\; =\; 100$ would mean there is much noise.
• A modality-embedding vector sequence, which indicates to the backbone about additional conditions for denoising. For example, in text-to-image generation, the text is divided into a sequence of tokens, then encoded by a text encoder, such as a CLIP encoder, before feeding into the backbone. As another example, an input image can be processed by a Vision Transformer into a sequence of vectors, which can then be used to condition the backbone for tasks such as generating an image in the same style.

Each run through the U-Net backbone produces a predicted noise vector. This noise vector is scaled down and subtracted away from the latent image array, resulting in a slightly less noisy latent image. The denoising is repeated according to a denoising schedule ("noise schedule"), and the output of the last step is processed by the VAE decoder into a finished image.

File:Encoder cross-attention.png

File:Transformer,_full_architecture.png

Similar to the standard U-Net, the U-Net backbone used in the SD 1.5 is essentially composed of down-scaling layers followed by up-scaling layers. However, the U-Net backbone has additional modules to allow for it to handle the embedding. As an illustration, we describe a single down-scaling layer in the backbone:

• The latent array and the time-embedding are processed by a ResBlock:
• The latent array is processed by a convolutional layer.
• The time-embedding vector is processed by a one-layered feedforward network, then added to the previous array (broadcast over all pixels).
• This is processed by another convolutional layer, then another time-embedding.
• The latent array and the embedding vector sequence are processed by a SpatialTransformer, which is essentially a standard pre-LN Transformer decoder without causal masking.
• In the cross-attentional blocks, the latent array itself serves as the query sequence, one query-vector per pixel. For example, if, at this layer in the U-Net, the latent array has dimensions $(128,\; 32,\; 32)$, then the query sequence has $1024$ vectors, each of which has $128$ dimensions. The embedding vector sequence serves as both the key sequence and as the value sequence.
• When no embedding vector sequence is input, a cross-attentional block defaults to self-attention, with the latent array serving as the query, key, and value.{{Cite web |title=latent-diffusion/ldm/modules/attention.py at main · CompVis/latent-diffusion |url=https://github.com/CompVis/latent-diffusion/blob/main/ldm/modules/attention.py#L251 |access-date=2024-09-09 |website=GitHub |language=en}}{{Pg|location=line 251}}

In pseudocode,

def ResBlock(x, time, residual_channels):

x_in = x

time_embedding = feedforward_network(time)

x = concatenate(x, residual_channels)

x = conv_layer_1(activate(normalize_1(x))) + time_embedding

x = conv_layer_2(dropout(activate(normalize_2(x))))

return x_in + x

def SpatialTransformer(x, cond):

x_in = x

x = normalize(x)

x = proj_in(x)

x = cross_attention(x, cond)

x = proj_out(x)

return x_in + x

def unet(x, time, cond):

residual_channels = []

for resblock, spatialtransformer in downscaling_layers:

x = resblock(x, time)

residual_channels.append(x)

x = spatialtransformer(x, cond)

x = middle_layer.resblock_1(x, time)

x = middle_layer.spatialtransformer(x, time)

x = middle_layer.resblock_2(x, time)

for resblock, spatialtransformer in upscaling_layers:

residual = residual_channels.pop()

x = resblock(concatenate(x, residual), time)

x = spatialtransformer(x, cond)

return x

The detailed architecture may be found in.{{Cite web |title=U-Net for Stable Diffusion |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet.html |access-date=2024-08-31 |website=U-Net for Stable Diffusion |language=en}}{{Cite web |title=Transformer for Stable Diffusion U-Net |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet_attention.html |access-date=2024-09-07 |website=Transformer for Stable Diffusion U-Net |language=en}}

The LDM is trained by using a Markov chain to gradually add noise to the training images. The model is then trained to reverse this process, starting with a noisy image and gradually removing the noise until it recovers the original image.

More specifically, the training process can be described as follows:

• Forward diffusion process: Given a real image $x\_0$, a sequence of latent variables $x\_\{1:T\}$ are generated by gradually adding Gaussian noise to the image, according to a pre-determined "noise schedule".
• Reverse diffusion process: Starting from a Gaussian noise sample $x\_T$, the model learns to predict the noise added at each step, in order to reverse the diffusion process and obtain a reconstruction of the original image $x\_0$.

The model is trained to minimize the difference between the predicted noise and the actual noise added at each step. This is typically done using a mean squared error (MSE) loss function.

Once the model is trained, it can be used to generate new images by simply running the reverse diffusion process starting from a random noise sample. The model gradually removes the noise from the sample, guided by the learned noise distribution, until it generates a final image.

See the diffusion model page for details.

• Diffusion model
• Generative adversarial network
• Variational autoencoder
• Stable Diffusion

• {{cite web |last=Wang |first=Phil |title=lucidrains/denoising-diffusion-pytorch |website=GitHub |date=2024-09-07 |url=https://github.com/lucidrains/denoising-diffusion-pytorch?tab=readme-ov-file |access-date=2024-09-07}}
• {{Cite web |title=The Annotated Diffusion Model |url=https://huggingface.co/blog/annotated-diffusion |access-date=2024-09-07 |website=huggingface.co}}
• {{Cite web |title=U-Net for Stable Diffusion |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet.html |access-date=2024-08-31 |website=U-Net for Stable Diffusion |language=en}}
• {{Cite web |title=Transformer for Stable Diffusion U-Net |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet_attention.html |access-date=2024-09-07 |website=Transformer for Stable Diffusion U-Net |language=en}}

{{Artificial intelligence navbox}}

Category:Deep learning

Category:Generative artificial intelligence

Category:Image processing

Category:Artificial intelligence art

Category:Text-to-image generation

Category:Unsupervised learning

Category:2021 software