MNIST database

In 1988, a dataset of digits from the US Postal Service was constructed. It contained 16×16 grayscale images digitized from handwritten zip codes that appeared on U.S. mail passing through the Buffalo, New York post office. The training set had 7291 images, and test set had 2007, making a total of 9298. Both training and test set contained ambiguous, unclassifiable, and misclassified data. The dataset was used to train and benchmark the 1989 LeNet.{{Cite journal |last1=Denker |first1=John |last2=Gardner |first2=W. |last3=Graf |first3=Hans |last4=Henderson |first4=Donnie |last5=Howard |first5=R. |last6=Hubbard |first6=W. |last7=Jackel |first7=L. D. |last8=Baird |first8=Henry |last9=Guyon |first9=Isabelle |date=1988 |title=Neural Network Recognizer for Hand-Written Zip Code Digits |url=https://proceedings.neurips.cc/paper/1988/hash/a97da629b098b75c294dffdc3e463904-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Morgan-Kaufmann |volume=1}}{{Cite journal |last1=LeCun |first1=Y. |last2=Boser |first2=B. |last3=Denker |first3=J. S. |last4=Henderson |first4=D. |last5=Howard |first5=R. E. |last6=Hubbard |first6=W. |last7=Jackel |first7=L. D. |date=December 1989 |title=Backpropagation Applied to Handwritten Zip Code Recognition |journal=Neural Computation |volume=1 |issue=4 |pages=541–551 |doi=10.1162/neco.1989.1.4.541 |issn=0899-7667 |s2cid=41312633}}

The task is rather difficult. On the test set, two humans made errors at an average rate of 2.5%.{{Cite journal |last1=Simard |first1=Patrice |last2=LeCun |first2=Yann |last3=Denker |first3=John |date=1992 |title=Efficient Pattern Recognition Using a New Transformation Distance |url=https://proceedings.neurips.cc/paper_files/paper/1992/hash/26408ffa703a72e8ac0117e74ad46f33-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Morgan-Kaufmann |volume=5}}

File:NIST_SD-19_f1002_33.png

In the late 1980s, the Census Bureau was interested in automatic digitization of handwritten census forms, so it enlisted the Image Recognition Group (IRG) at NIST to evaluate OCR systems.{{Cite report |url=https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir4912.pdf |title=The first census optical character recognition system conference. NIST Interagency/Internal Report (NISTIR) - 4912 |last1=Wilkinson |first1=R Allen |last2=Geist |first2=Jon |last3=Janet |first3=Stanley |last4=Grother |first4=Patrick J |last5=Burges |first5=Christopher J C |last6=Creecy |first6=Robert |last7=Hammond |first7=Bob |last8=Hull |first8=Jonathan J |last9=Larsen |first9=Norman L |date=1992 |publisher=National Institute of Standards and Technology |issue= |doi=10.6028/nist.ir.4912 |location=Gaithersburg, MD |volume=}} Several years of work resulted in several "Special Databases" and benchmarks. Of particular importance to MNIST are Special Database 1 (SD-1), released in 1990-05,C. L. Wilson and M. D. Garris. Handprinted character database. Technical Report Special Database 1, HWDB, National Institute of Standards and Technology, April 1990. Special Database 3 (SD-3), released in 1992-02,M. D. Garris and R. A. Wilkinson. Handwritten segmented characters database. Technical Report Special Database 3, HWSC, National Institute of Standards and Technology, February 1992. and Special Database 7 (SD-7), or NIST Test Data 1 (TD-1), released in 1992-04.R. A. Wilkinson. Handprinted Segmented Characters Database. Technical Report Test Database 1, TST1, National Institute of Standards and Technology, April 1992. They were released on ISO-9660 CD-ROMs. They were obtained by asking people to write on "Handwriting Sample Forms" (HSFs), then digitizing the HSFs, then segmenting out the alphanumerical characters. Each writer wrote a single HSF.

Each HSF contains multiple entry fields, wherein people were asked to write. There are 34 fields: name and date entries, a city/state field, 28 digit fields, one upper-case field, one lower-case field, and an unconstrained Constitution text paragraph. Each HSF was scanned at resolution 300 dots per inch (11.8 dots per millimeter).

SD-1 and SD-3 were constructed from the same set of HSFs by 2100 out of 3400 permanent census field workers as part of the 1990 United States census.{{Pg|page=10}} SD-1 contained the segmented data entry fields, but not the segmented alphanumericals. SD-3 contained binary 128×128 images digitized from segmented alphanumericals, with 223,125 digits, 44,951 upper-case letters, and 45,313 lower case letters.

SD-7 or TD-1 was the test set, and it contained 58,646 128×128 binary images written by 500 high school students in Bethesda, Maryland. They were described as "math and science students in a high school as a short exercise during class".{{Pg|page=10}} Each image is accompanied by a unique integer ID for the identity of its writer. SD-7 was released without labels on CD-ROMs, and the labels were later released on floppy drives. It did not contain the HSFs. SD-7 was difficult enough that the human error rate on it was 1.5%.{{Cite journal |last1=Smith |first1=S.J. |last2=Bourgoin |first2=M.O. |last3=Sims |first3=K. |last4=Voorhees |first4=H.L. |date=September 1994 |title=Handwritten character classification using nearest neighbor in large databases |url=https://ieeexplore.ieee.org/document/310689 |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=16 |issue=9 |pages=915–919 |doi=10.1109/34.310689}}

SD-3 was much cleaner and easier to recognize than images in SD-7. The European crossed seven (7) is far more abundant in SD-7 than in SD-3.{{Cite journal |last=Grother |first=Patrick J. |date=1993-01-01 |title=Cross Validation Comparison of NIST OCR Databases |url=https://www.nist.gov/publications/cross-validation-comparison-nist-ocr-databases |journal=NIST |volume=1906 |page=296 |doi=10.1117/12.143632 |bibcode=1993SPIE.1906..296G |language=en}} It was suspected that SD-3 was produced by people more motivated than those who produced SD-7. Also, the character segmenter for SD-3 was an older design than that of SD-7, and failed more often. It was suspected that the harder instances were filtered out of the construction of SD-3, since the hard instances failed to even pass the segmenter.{{Pg|page=10}} It was found that machine learning systems trained and validated on SD-3 suffered significant drops in performance on SD-7, from an error rate of less than 1% to ~10%.{{cite book |last1=Bottou |first1=Léon |title=Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5) |last2=Cortes |first2=Corinna |last3=Denker |first3=John S. |last4=Drucker |first4=Harris |last5=Guyon |first5=Isabelle |last6=Jackel |first6=L. D. |last7=LeCun |first7=Y. |last8=Muller |first8=U. A. |last9=Sackinger |first9=E. |year=1994 |isbn=0-8186-6270-0 |volume=2 |location=Jerusalem, Israel |pages=77–82 |chapter=Comparison of classifier methods: A case study in handwritten digit recognition |doi=10.1109/ICPR.1994.576879 |first10=P. |last10=Simard |first11=V. |last11=Vapnik}}{{Pg|page=9}}

In 1992, NIST and the Census Bureau sponsored a competition and a conference to determine the state of the art in this industry. In the competition, teams were given SD-3 as the training set before March 23, SD-7 as the test set before April 13, and would submit one or more systems for classifying SD-7 before April 27.{{Pg|location=Appendix C}} A total of 45 algorithms were submitted from 26 companies from 7 different countries. On May 27 and 28, all parties that submitted results convened in Gaithersburg, Maryland at the First Census OCR Systems Conference. Observers from FBI, IRS, and USPS were in attendance.{{Pg|page=1}} The winning entry did not use SD-3 for training, but a much larger proprietary training set, thus was not affected by the distribution shift. Among the 25 entries that did use SD-3 for training, the winning entry was a nearest-neighbor classifier using a handcrafted metric that is invariant to Euclidean transforms.{{Cite journal |last1=Simard |first1=Patrice |last2=LeCun |first2=Yann |last3=Denker |first3=John |date=1992 |title=Efficient Pattern Recognition Using a New Transformation Distance |url=https://proceedings.neurips.cc/paper_files/paper/1992/hash/26408ffa703a72e8ac0117e74ad46f33-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Morgan-Kaufmann |volume=5}}

SD-19 was published in 1995, as a compilation of SD-1, SD-3, SD-7 and some further data. It contained 814,255 binary images of alphanumericals and binary images of 4169 HSFs, including those 500 HSFs that were used to generate SD-7. It was updated in 2016.

The MNIST was constructed sometime before summer 1994. It was constructed by mixing 128x128 binary images from SD-3 and SD-7. Specifically, they first took all images from SD-7 and divided them into a training set and a test set, each from 250 writers. This resulted in nearly 30000 images in each set. They then added more images from SD-3 until each set contained exactly 60000 images.

Each image was size-normalized to fit in a 20x20 pixel box while preserving their aspect ratio, and anti-aliased to grayscale. Then it was put into a 28x28 image by translating it until the center of mass of the pixels is in the center of the image. The details of how the downsampling proceeded was reconstructed.{{Cite journal |last1=Yadav |first1=Chhavi |last2=Bottou |first2=Leon |date=2019 |title=Cold Case: The Lost MNIST Digits |url=https://proceedings.neurips.cc/paper/2019/hash/51c68dc084cb0b8467eafad1330bce66-Abstract.html |journal=Advances in Neural Information Processing Systems |volume=32 |arxiv=1905.10498 |quote=Article has a detailed history and a reconstruction of the discarded testing set.}}

The training set and the test set both originally had 60k samples, but 50k of the test set samples were discarded, and only the samples indexed 24476 to 34475 were used, giving just 10k samples in the test set.{{Cite journal |last1=Decoste |first1=Dennis |last2=Schölkopf |first2=Bernhard |date=2002 |title=Training invariant support vector machines |url=http://link.springer.com/10.1023/A:1012454411458 |journal=Machine Learning |volume=46 |issue=1/3 |pages=161–190 |doi=10.1023/A:1012454411458}}

In 2019, the full 60k test set from MNIST was restored to construct the QMNIST, which has 60k images in the training set and 60k in the test set.{{Citation |title=facebookresearch/qmnist |date=2024-09-23 |url=https://github.com/facebookresearch/qmnist |access-date=2024-10-25 |publisher=Meta Research}}

Extended MNIST (EMNIST) is a newer dataset developed and released by NIST to be the (final) successor to MNIST, released in 2017.{{cite arXiv |eprint=1702.05373 |class=cs.CV |first1=G. |last1=Cohen |first2=S. |last2=Afshar |title=EMNIST: an extension of MNIST to handwritten letters. |last3=Tapson |first3=J. |last4=van Schaik |first4=A. |year=2017}}{{cite web |last=NIST |date=4 April 2017 |title=The EMNIST Dataset |url=https://www.nist.gov/itl/products-and-services/emnist-dataset |access-date=11 April 2022 |website=NIST}} MNIST included images only of handwritten digits. EMNIST was constructed from all the images from SD-19,{{cite web |last=NIST |date=27 August 2010 |title=NIST Special Database 19 |url=https://www.nist.gov/srd/nist-special-database-19 |access-date=11 April 2022 |website=NIST}}Grother, Patrick J., and K. K. Hanaoka. "[https://www.nist.gov/system/files/documents/srd/nistsd19.pdf NIST special database 19]." Handprinted forms and characters database, National Institute of Standards and Technology 10 (1995): 69. converted into the same 28x28 pixel format, by the same process, as were the MNIST images. Accordingly, tools which work with MNIST would likely work unmodified with EMNIST.

Fashion MNIST was created in 2017 as a more challenging alternative for MNIST. The dataset consists of 70,000 28x28 grayscale images of fashion products from 10 categories.{{cite arXiv |last1=Xiao |first1=Han |title=Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms |date=2017-09-15 |eprint=1708.07747 |last2=Rasul |first2=Kashif |last3=Vollgraf |first3=Roland|class=cs.LG }}

Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks.{{cite book|last=Cires¸an|first=Dan|chapter-url=http://repository.supsi.ch/5145/1/IDSIA-04-12.pdf|chapter=Multi-column deep neural networks for image classification|author2=Ueli Meier|author3=Jürgen Schmidhuber|title=2012 IEEE Conference on Computer Vision and Pattern Recognition|year=2012|isbn=978-1-4673-1228-8|pages=3642{{en dash}}3649|arxiv=1202.2745|citeseerx=10.1.1.300.3283|doi=10.1109/CVPR.2012.6248110|s2cid=2161592}} The highest error rate listed on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing.

In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles.{{cite journal|last=Kussul|first=Ernst|author2=Tatiana Baidyk|author2-link=Tetyana Baydyk|title=Improved method of handwritten digit recognition tested on MNIST database|journal=Image and Vision Computing|year=2004|volume=22|issue=12|pages=971{{en dash}}981|doi=10.1016/j.imavis.2004.03.008|url=https://vlabdownload.googlecode.com/files/Image_VisionComputing.pdf|access-date=20 September 2013|archive-url=https://web.archive.org/web/20130921060416/https://vlabdownload.googlecode.com/files/Image_VisionComputing.pdf|archive-date=21 September 2013|url-status=dead}}

Some studies have used Data Augmentation to increase the training data set size and thereby performance. The systems in these cases are usually neural networks and the distortions used tend to be either affine distortions or elastic distortions. Sometimes, these systems can be very successful; one such system achieved an error rate on the database of 0.39 percent.{{cite journal|last=Ranzato|first=Marc'Aurelio|author2=Christopher Poultney |author3=Sumit Chopra |author4=Yann LeCun |title=Efficient Learning of Sparse Representations with an Energy-Based Model|journal=Advances in Neural Information Processing Systems|year=2006|volume=19|pages=1137{{en dash}}1144|url=http://yann.lecun.com/exdb/publis/pdf/ranzato-06.pdf|access-date=20 September 2013}}

In 2011, an error rate of 0.27 percent, improving on the previous best result, was reported by researchers using a similar system of neural networks.{{cite book|last=Ciresan|first=Dan Claudiu|author2=Ueli Meier|author3=Luca Maria Gambardella|author4=Jürgen Schmidhuber|chapter=Convolutional neural network committees for handwritten character classification|title=2011 International Conference on Document Analysis and Recognition (ICDAR)|year=2011|pages=1135{{en dash}}1139|doi=10.1109/ICDAR.2011.229|chapter-url=http://www.icdar2011.org/fileup/PDF/4520b135.pdf|access-date=20 September 2013|isbn=978-1-4577-1350-7|citeseerx=10.1.1.465.2138|s2cid=10122297|archive-url=https://web.archive.org/web/20160222152015/http://www.icdar2011.org/fileup/PDF/4520b135.pdf|archive-date=22 February 2016|url-status=dead}} In 2013, an approach based on regularization of neural networks using DropConnect has been claimed to achieve a 0.21 percent error rate.{{cite conference|last=Wan|first=Li|author2=Matthew Zeiler|author3=Sixin Zhang|author4=Yann LeCun|author5=Rob Fergus|title=Regularization of Neural Network using DropConnect|conference=International Conference on Machine Learning(ICML)|year=2013}} In 2016, the single convolutional neural network best performance was 0.25 percent error rate.{{Cite web|last=SimpleNet|year=2016|title=Lets Keep it simple, Using simple architectures to outperform deeper and more complex architectures|url=https://github.com/Coderx7/SimpleNet|access-date=3 December 2020|arxiv=1608.06037}} As of August 2018, the best performance of a single convolutional neural network trained on MNIST training data using no data augmentation is 0.25 percent error rate.{{Cite web|last=SimpNet|title=Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet|url=https://github.com/Coderx7/SimpNet|access-date=3 December 2020|website=Github|year=2018|arxiv=1802.06205}} Also, the Parallel Computing Center (Khmelnytskyi, Ukraine) obtained an ensemble of only 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate.{{cite web|last=Romanuke|first=Vadim|title=Parallel Computing Center (Khmelnytskyi, Ukraine) represents an ensemble of 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate.|url=https://drive.google.com/file/d/0B1WkCFOvGHDddElkdkl6bzRLRE0/view?usp=sharing|access-date=24 November 2016}}{{cite journal |last1=Romanuke |first1=Vadim |title=Training data expansion and boosting of convolutional neural networks for reducing the MNIST dataset error rate|journal=Research Bulletin of NTUU "Kyiv Polytechnic Institute"|date=2016 |volume=6|issue=6 |pages=29{{en dash}}34|doi=10.20535/1810-0546.2016.6.84115|ref=24|doi-access=free}}

This is a table of some of the machine learning methods used on the dataset and their error rates, by type of classifier:

|-

| Support-vector machine (SVM) || Virtual SVM, deg-9 poly, 2-pixel jittered || {{okay|None}} || Deskewing || 0.56{{Cite journal|last1=Decoste|first1=Dennis|last2=Schölkopf|first2=Bernhard|year=2002|title=Training Invariant Support Vector Machines|journal=Machine Learning|volume=46|pages=161{{en dash}}190|issue=1–3|doi=10.1023/A:1012454411458|oclc=703649027|issn=0885-6125|doi-access=free}}

|-

| Neural network || 2-layer 784-800-10 || {{okay|None}} || {{okay|None}} || 1.6{{cite book|chapter=Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis|author=Patrice Y. Simard|author2=Dave Steinkraus|author3=John C. Platt|year=2003|chapter-url=http://research.microsoft.com/apps/pubs/?id=68920|publisher=Institute of Electrical and Electronics Engineers|doi=10.1109/ICDAR.2003.1227801|title=Proceedings of the Seventh International Conference on Document Analysis and Recognition |volume=1|pages=958|isbn=978-0-7695-1960-9|s2cid=4659176}}

|-

| Neural network || 2-layer 784-800-10 || Elastic distortions || {{okay|None}} || 0.7

|-

| Deep neural network (DNN) || 6-layer 784-2500-2000-1500-1000-500-10 || Elastic distortions || {{okay|None}} || 0.35{{cite journal|last=Ciresan|first=Claudiu Dan |author2=Ueli Meier |author3=Luca Maria Gambardella |author4=Juergen Schmidhuber |title=Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition|journal=Neural Computation|date=December 2010|volume=22|issue=12|pages=3207{{en dash}}20 |doi=10.1162/NECO_a_00052|pmid=20858131 |arxiv=1003.0358|s2cid=1918673}}

|-

| {{nowrap|Convolutional neural network (CNN)}} || 6-layer 784-40-80-500-1000-2000-10 || {{okay|None}} || {{nowrap|Expansion of the training data}} || 0.31{{cite web|last=Romanuke|first=Vadim|title=The single convolutional neural network best performance in 18 epochs on the expanded training data at Parallel Computing Center, Khmelnytskyi, Ukraine|url=https://drive.google.com/file/d/0B1WkCFOvGHDdWlZvWUlLd0V3ZFU/view?usp=sharing|access-date=16 November 2016}}

|-

| Convolutional neural network || 6-layer 784-50-100-500-1000-10-10 || {{okay|None}} || Expansion of the training data || 0.27{{cite web|last=Romanuke|first=Vadim|title=Parallel Computing Center (Khmelnytskyi, Ukraine) gives a single convolutional neural network performing on MNIST at 0.27 percent error rate|url=https://drive.google.com/file/d/0B1WkCFOvGHDdOC0yR0tfbmpidjg/view?usp=sharing|access-date=24 November 2016}}

|-

|{{nowrap|Convolutional neural network (CNN)}}

|13-layer 64-128(5x)-256(3x)-512-2048-256-256-10|| {{okay|None}} || {{okay|None}}

|0.25

|-

| Convolutional neural network || Committee of 35 CNNs, 1-20-P-40-P-150-10 || Elastic distortions || Width normalizations || 0.23

|-

| Convolutional neural network || {{nowrap|Committee of 5 CNNs, 6-layer 784-50-100-500-1000-10-10}} || {{okay|None}} || Expansion of the training data || 0.21

|-

|Convolutional neural network

|Committee of 20 CNNS with Squeeze-and-Excitation Networks{{Cite journal|arxiv = 1709.01507|last1 = Hu|first1 = Jie|title = Squeeze-and-Excitation Networks|last2 = Shen|first2 = Li|last3 = Albanie|first3 = Samuel|last4 = Sun|first4 = Gang|last5 = Wu|first5 = Enhua|journal = IEEE Transactions on Pattern Analysis and Machine Intelligence|year = 2019|volume = 42|issue = 8|pages = 2011{{en dash}}2023|doi = 10.1109/TPAMI.2019.2913372|pmid = 31034408|s2cid = 140309863}}|| {{okay|None}}

|Data augmentation

|0.17{{Cite web | url=https://github.com/Matuzas77/MNIST-0.17.git |title = GitHub - Matuzas77/MNIST-0.17: MNIST classifier with average 0.17% error|website = GitHub|date = 25 February 2020}}

|-

|Convolutional neural network

|Ensemble of 3 CNNs with varying kernel sizes

|{{okay|None}}

|Data augmentation consisting of rotation and translation

|0.09{{Cite arXiv |last1=An |first1=Sanghyeon |last2=Lee |first2=Minjun |last3=Park |first3=Sanglee |last4=Yang |first4=Heerin |last5=So |first5=Jungmin |date=2020-10-04 |title=An Ensemble of Simple Convolutional Neural Network Models for MNIST Digit Recognition |class=cs.CV |eprint=2008.10400 }}

|}

• List of datasets for machine learning research
• Caltech 101
• LabelMe
• OCR

{{reflist|2}}

• {{cite book |last1=Ciresan |first1=Dan |first2=Ueli |last2=Meier |first3=Jürgen |last3=Schmidhuber |chapter=Multi-column deep neural networks for image classification |title=2012 IEEE Conference on Computer Vision and Pattern Recognition |date=June 2012 |pages=3642{{en dash}}3649 |chapter-url=http://repository.supsi.ch/5145/1/IDSIA-04-12.pdf |doi=10.1109/CVPR.2012.6248110 |arxiv=1202.2745 |access-date=2013-12-09 |isbn=9781467312264 |oclc=812295155 |publisher=Institute of Electrical and Electronics Engineers |location=New York, NY|citeseerx=10.1.1.300.3283 |s2cid=2161592}}

• {{official website}}
• [https://github.com/mbornet-hl/MNIST/tree/master/IMAGES/GROUPS Visualization of the MNIST database]{{snd}} groups of images of MNIST handwritten digits on GitHub

{{Differentiable computing}}

{{Standard test item}}

Category:Datasets in computer vision