model collapse

Using synthetic data as training data can lead to issues with the quality and reliability of the trained model.{{cite arXiv |eprint=2307.01850 |class=cs.LG |first1=Sina |last1=Alemohammad |first2=Josue |last2=Casco-Rodriguez |title=Self-Consuming Generative Models Go MAD |date=July 4, 2023 |last3=Luzi |first3=Lorenzo |last4=Humayun |first4=Ahmed Imtiaz |last5=Babaei |first5=Hossein |last6=LeJeune |first6=Daniel |last7=Siahkoohi |first7=Ali |last8=Baraniuk |first8=Richard G.}}{{cite conference |title=Self-Consuming Generative Models Go MAD |url=https://openreview.net/forum?id=ShjMHfmPs0 |conference=The Twelfth International Conference on Learning Representations |language=English}} Model collapse occurs for three main reasons:

1. functional approximation errors
2. sampling errors
3. learning errors

Importantly, it happens in even the simplest of models, where not all of the error sources are present. In more complex models the errors often compound, leading to faster collapse.

File:Model Collapse in Generative Models Can Be Avoided By Accumulating Data.png

Some researchers and commentators on model collapse warn that the phenomenon could fundamentally threaten future generative AI development: As AI-generated data is shared on the Internet, it will inevitably end up in future training datasets, which are often crawled from the Internet. If training on "slop" (large quantities of unlabeled synthetic data) inevitably leads to model collapse, this could therefore pose a difficult problem.{{cite web |title=What is Model Collapse and how to avoid it |url=https://www.theregister.com/2024/01/26/what_is_model_collapse/ |website=The Register |access-date=11 July 2024}}

However, recently, other researchers have disagreed with this argument, showing that if synthetic data accumulates alongside human-generated data, model collapse is avoided.{{Cite arXiv |eprint=2404.01413 |class=cs.LG |first1=Matthias |last1=Gerstgrasser |first2=Rylan |last2=Schaeffer |title=Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data |date=2024-04-01 |last3=Dey |first3=Apratim |last4=Rafailov |first4=Rafael |last5=Sleight |first5=Henry |last6=Hughes |first6=John |last7=Korbak |first7=Tomasz |last8=Agrawal |first8=Rajashree |last9=Pai |first9=Dhruv |last10=Gromov |first10=Andrey |last11=Roberts |first11=Daniel A. |last12=Yang |first12=Diyi |last13=Donoho |first13=David L. |last14=Koyejo |first14=Sanmi}} The researchers argue that data accumulating over time is a more realistic description of reality than deleting all existing data every year, and that the real-world impact of model collapse may not be as catastrophic as feared.{{cite web |title=Big brains divided over training AI with more AI: Is model collapse inevitable? |url=https://www.theregister.com/2024/05/09/ai_model_collapse/ |website=The Register |access-date=11 July 2024}}

An alternative branch of the literature investigates the use of machine learning detectors and watermarking to identify model generated data and filter it out.{{Cite journal |last1=Kirchenbauer |first1=John |last2=Geiping |first2=Jonas |last3=Wen |first3=Yuxin |last4=Katz |first4=Jonathan |last5=Miers |first5=Ian |last6=Goldstein |first6=Tom |date=2023-07-03 |title=A Watermark for Large Language Models |url=https://proceedings.mlr.press/v202/kirchenbauer23a.html |journal=Proceedings of the 40th International Conference on Machine Learning |language=en |publisher=PMLR |pages=17061–17084}}{{Cite web |date=2022-11-29 |title=My AI Safety Lecture for UT Effective Altruism |url=https://scottaaronson.blog/?p=6823 |access-date=2024-06-22 |website=Shtetl-Optimized |language=en-US}}

In 2024, a first attempt has been made at illustrating collapse for the simplest possible model — a single dimensional normal distribution fit using unbiased estimators of mean and variance, computed on samples from the previous generation.

To make this more precise, we say that original data follows a normal distribution $X^0\; \backslash sim\; \backslash mathcal\{N\}(\backslash mu,\backslash sigma^2)$, and we possess $M\_0$ samples $X^0\_j$ for $j\; \backslash in\; \{\backslash \{\backslash ,\; 1,\; \backslash dots,\; M\_0\; \backslash ,\{\}\backslash \}\}$. Denoting a general sample $X^i\_j$ as sample $j\; \backslash in\; \{\backslash \{\backslash ,\; 1,\; \backslash dots,\; M\_i\; \backslash ,\{\}\backslash \}\}$ at generation $i$, then the next generation model is estimated using the sample mean and variance:

$\backslash mu\_\{i+1\}\; =\; \backslash frac\{1\}\{M\_i\}\backslash sum\_j\; X^i\_j;\; \backslash quad\; \backslash sigma\_\{i+1\}^2\; =\; \backslash frac\{1\}\{M\_i-1\}\backslash sum\; \_j(X^i\_j-\backslash mu\_\{i+1\})^2.$

Leading to a conditionally normal next generation model $X^\{i+1\}\_j|\backslash mu\_\{i+1\},\backslash ;\backslash sigma\_\{i+1\}\backslash sim\; \backslash mathcal\{N\}(\backslash mu\_\{i+1\},\backslash sigma\_\{i+1\}^2)$. In theory, this is enough to calculate the full distribution of $X^i\_j$. However, even after the first generation, the full distribution is no longer normal: It follows a variance-gamma distribution.

To continue the analysis, instead of writing the probability density function at each generation, it is possible to explicitly construct them in terms of independent random variables using Cochran's theorem. To be precise, $\backslash mu\_1$ and $\backslash sigma\_1$are independent, with $\backslash mu\_1\; \backslash sim\; \backslash mathcal\{N\}\backslash left(\backslash mu,\; \backslash frac\{\backslash sigma^2\}\{M\_0\}\backslash right)$ and $(M\_0-1)\backslash ,\backslash sigma\_1^2\; \backslash sim\; \backslash sigma^2\backslash ,\backslash Gamma\backslash left(\backslash frac\{M\_0-1\}\{2\},\; \backslash frac12\backslash right)$, following a Gamma distribution. Denoting with $Z$ Gaussian random variables distributed according to $\backslash mathcal\{N\}(0,\; 1)$ and with $S^i$ random variables distributed with $\backslash frac\{1\}\{M\_\{i-1\}-1\}\backslash Gamma\backslash left(\backslash frac\{M\_\{i-1\}-1\}\{2\},\; \backslash frac12\backslash right)$, it turns out to be possible to write samples at each generation as

$X^0\_j\; =\; \backslash mu\; +\; \backslash sigma\; Z^0\_j,$

$X^1\_j\; =\; \backslash mu\; +\; \backslash frac\{\backslash sigma\}\{\backslash sqrt\{M\_0\}\}Z^1\; +\; \backslash sigma\backslash sqrt\{S^1\}Z^1\_j,$

and more generally

$$X^n\_j\; =\; \backslash mu\; +\; \backslash frac\{\backslash sigma\}\{\backslash sqrt\{M\_0\}\}Z^1\; +\; \backslash frac\{\backslash sigma\}\{\backslash sqrt\{M\_1\}\}\backslash sqrt\{S^1\}Z^2\; +\; \backslash dots$$

+ \frac{\sigma}{\sqrt{M_{n-1}}}\sqrt{S^1\times\dots\times S^{n-1}}Z^n+\sigma\sqrt{S^1\times\dots\times S^{n}}Z^n_j.

Note, that these are not joint distributions, as $Z^n$ and $S^n$ depend directly on $Z^\{n-1\}\_j$, but when considering $X^n\_j$ on its own the formula above provides all the information about the full distribution.

To analyse the model collapse, we can first calculate variance and mean of samples at generation $n$. This would tell us what kind of distributions we expect to arrive at after $n$

generations. It is possible to find its exact value in closed form, but the mean and variance of the square root of gamma distribution are expressed in terms of gamma functions, making the result quite clunky. Following, it is possible to expand all results to second order in each of $1/M\_i$, assuming each sample size to be large. It is then possible to show that

$\backslash frac\{1\}\{\backslash sigma^2\}\backslash operatorname\{Var\}(X^n\_j)\; =\; \backslash frac\{1\}\{M\_0\}+\backslash frac\{1\}\{M\_1\}+\; \backslash dots\; +\; \backslash frac\{1\}\{M\_\{n-1\}\}+1\; +\; \backslash mathcal\{O\}\backslash left(M\_i^\{-2\}\backslash right).$

And if all sample sizes $M\_i\; =\; M$ are constant, this diverges linearly as $n\backslash to\backslash infty$:

$\backslash operatorname\{Var\}(X^n\_j)\; =\; \backslash sigma^2\backslash left(1+\backslash frac\{n\}\{M\}\backslash right);\; \backslash quad\; \backslash mathbb\{E\}(X^n\_j)\; =\; \backslash mu.$

This is the same scaling as for a single dimensional Gaussian random walk. However, divergence of the variance of $X^n\_j$ does not directly provide any information about the corresponding estimates of $\backslash mu\_\{n+1\}$ and $\backslash sigma\_\{n+1\}$, particularly how different they are from the original $\backslash mu$ and $\backslash sigma$. It turns out to be possible to calculate the distance between the true distribution and the approximated distribution at step $n+1$, using the Wasserstein-2 distance (which is also sometimes referred to as risk):

$\backslash mathbb\{E\}\backslash left[\backslash mathbb\{W\}^2\_2\backslash left(\backslash mathcal\{N\}(\backslash mu,\backslash sigma^2),\backslash mathcal\{N\}(\backslash mu\_\{n+1\},\backslash sigma^2\_\{n+1\})\backslash right)\backslash right]=\backslash frac\{3\}\{2\}\backslash sigma^2\backslash left(\backslash frac\{1\}\{M\_0\}+\backslash frac\{1\}\{M\_1\}+\; \backslash dots\; +\; \backslash frac\{1\}\{M\_\{n\}\}\backslash right)+\backslash mathcal\{O\}\backslash left(M\_i^\{-2\}\backslash right),$

$\backslash operatorname\{Var\}\backslash left[\backslash mathbb\{W\}^2\_2\backslash left(\backslash mathcal\{N\}(\backslash mu,\backslash sigma^2),\backslash mathcal\{N\}(\backslash mu\_\{n+1\},\backslash sigma^2\_\{n+1\})\backslash right)\backslash right]=\backslash frac\{1\}\{2\}\backslash sigma^4\backslash left(\backslash frac\{3\}\{M\_0^2\}+\backslash frac\{3\}\{M\_1^2\}+\; \backslash dots\; +\; \backslash frac\{3\}\{M\_\{n\}^2\}\; +\; \backslash sum\_\{i\backslash neq\; j\}\backslash frac\{4\}\{M\_iM\_j\}\backslash right)+\backslash mathcal\{O\}\backslash left(M\_i^\{-3\}\backslash right).$

This directly shows why model collapse occurs in this simple model. Due to errors from re-sampling the approximated distribution, each generation ends up corresponding to a new step in a random walk of model parameters. For a constant sample size at each generation, the average distance from the starting point diverges, and in order for the end distribution approximation to be accurate, or for the distance to be finite, the sampling rate $M\_i$ needs to increase superlinearly, i.e. one needs to collect increasingly more samples over time, perhaps quadratically. However, even in that case the expected distance after $n$ steps remains non-zero and the only case in which it does in fact end up being zero is when sampling is infinite at each step. Overall, this only shows us how far on average one ends up from the original distribution, but the process can only "terminate", if the estimated variance at a certain generation becomes small enough, effectively turning the distribution into a delta function. This is shown to occur for a general gaussian model in the subsection below. Empirical investigation has confirmed this theoretical analysis.{{cite arXiv |last1=Borji |first1=Ali |title=A Note on Shumailov et al. (2024): "AI Models Collapse When Trained on Recursively Generated Data" |date=2024-10-16 |eprint=2410.12954 |class=cs.LG}}

Furthermore, in the case of multidimensional model with fully synthetic data, exact collapse can be shown.

In the case of a linear regression model,{{Cite arXiv |last1=Dohmatob |first1=Elvis |last2=Feng |first2=Yunzhen |last3=Kempe |first3=Julia |date=2024-02-12 |title=Model Collapse Demystified: The Case of Regression |class=cs.LG |eprint =2402.07712}}{{cite arXiv |last1=Dohmatob |first1=Elvis |title=A Tale of Tails: Model Collapse as a Change of Scaling Laws |date=2024-02-10 |eprint =2402.07043 |last2=Feng |first2=Yunzhen |last3=Yang |first3=Pu |last4=Charton |first4=Francois |last5=Kempe |first5=Julia|class=cs.LG }} scaling laws and bounds on learning can be obtained.

In the case of a linear softmax classifier for next token prediction,{{cite arXiv |last1=Seddik |first1=Mohamed El Amine |title=How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse |date=2024-04-07|eprint = 2404.05090 |last2=Chen |first2=Suei-Wen |last3=Hayou |first3=Soufiane |last4=Youssef |first4=Pierre |last5=Debbah |first5=Merouane|class=cs.LG }} exact bounds on learning with even a partially synthetic dataset can be obtained.

In the context of large language models, research found that training LLMs on predecessor-generated text — language models are trained on the synthetic data produced by previous models — causes a consistent decrease in the lexical, syntactic, and semantic diversity of the model outputs through successive iterations, notably remarkable for tasks demanding high levels of creativity.{{cite arXiv |last1=Guo |first1=Yanzhu |title=The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text |date=2024-04-16 |eprint=2311.09807 |last2=Shang |first2=Guokan |last3=Vazirgiannis |first3=Michalis |last4=Clavel |first4=Chloé|class=cs.CL }}

• Generation loss
• Generative artificial intelligence

{{Reflist|group=note}}

{{reflist}}

Category:Generative artificial intelligence