speech recognition#Hidden Markov models

The key areas of growth were: vocabulary size, speaker independence, and processing speed.

• 1952 – Three Bell Labs researchers, Stephen Balashek,{{Cite news |date=22 July 2012 |title=Obituaries: Stephen Balashek |url=https://obits.nj.com/obituaries/starledger/obituary.aspx?page=lifestory&pid=158702138 |work=The Star-Ledger |access-date=9 September 2024 |archive-date=4 April 2019 |archive-url=https://web.archive.org/web/20190404231352/https://obits.nj.com/obituaries/starledger/obituary.aspx?page=lifestory&pid=158702138 |url-status=live }} R. Biddulph, and K. H. Davis built a system called "Audrey"{{Cite web |title=IBM-Shoebox-front.jpg |url=https://cdn57.androidauthority.net/wp-content/uploads/2012/04/IBM-Shoebox-front.jpg |access-date=4 April 2019 |publisher=androidauthority.net |archive-date=9 August 2018 |archive-url=https://web.archive.org/web/20180809153221/https://cdn57.androidauthority.net/wp-content/uploads/2012/04/IBM-Shoebox-front.jpg |url-status=live }} for single-speaker digit recognition. Their system located the formants in the power spectrum of each utterance.{{Cite web |last1=Juang |first1=B. H. |last2=Rabiner |first2=Lawrence R. |title=Automatic speech recognition–a brief history of the technology development |url=http://www.ece.ucsb.edu/faculty/Rabiner/ece259/Reprints/354_LALI-ASRHistory-final-10-8.pdf |url-status=live |archive-url=https://web.archive.org/web/20140817193243/http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/354_LALI-ASRHistory-final-10-8.pdf |archive-date=17 August 2014 |access-date=17 January 2015 |page=6}}
• 1960 – Gunnar Fant developed and published the source-filter model of speech production.
• 1962 – IBM demonstrated its 16-word "Shoebox" machine's speech recognition capability at the 1962 World's Fair.{{Cite magazine |last=Melanie Pinola |date=2 November 2011 |title=Speech Recognition Through the Decades: How We Ended Up With Siri |url=https://www.pcworld.com/article/243060/speech_recognition_through_the_decades_how_we_ended_up_with_siri.html |access-date=22 October 2018 |magazine=PC World |archive-date=3 November 2018 |archive-url=https://web.archive.org/web/20181103105727/https://www.pcworld.com/article/243060/speech_recognition_through_the_decades_how_we_ended_up_with_siri.html |url-status=live }}
• 1966 – Linear predictive coding (LPC), a speech coding method, was first proposed by Fumitada Itakura of Nagoya University and Shuzo Saito of Nippon Telegraph and Telephone (NTT), while working on speech recognition.{{Cite journal |last=Gray |first=Robert M. |date=2010 |title=A History of Realtime Digital Speech on Packet Networks: Part II of Linear Predictive Coding and the Internet Protocol |url=https://ee.stanford.edu/~gray/lpcip.pdf |journal=Found. Trends Signal Process. |volume=3 |issue=4 |pages=203–303 |doi=10.1561/2000000036 |issn=1932-8346 |doi-access=free |access-date=9 September 2024 |archive-date=9 October 2022 |archive-url=https://ghostarchive.org/archive/20221009/https://ee.stanford.edu/~gray/lpcip.pdf |url-status=live }}
• 1969 – Funding at Bell Labs dried up for several years when, in 1969, the influential John Pierce wrote an open letter that was critical of and defunded speech recognition research.{{Cite journal |last=John R. Pierce |author-link=John R. Pierce |date=1969 |title=Whither speech recognition? |journal=Journal of the Acoustical Society of America |volume=46 |issue=48 |pages=1049–1051 |bibcode=1969ASAJ...46.1049P |doi=10.1121/1.1911801}} This defunding lasted until Pierce retired and James L. Flanagan took over.

Raj Reddy was the first person to take on continuous speech recognition as a graduate student at Stanford University in the late 1960s. Previous systems required users to pause after each word. Reddy's system issued spoken commands for playing chess.

Around this time Soviet researchers invented the dynamic time warping (DTW) algorithm and used it to create a recognizer capable of operating on a 200-word vocabulary.{{Cite book |last1=Benesty |first1=Jacob |title=Springer Handbook of Speech Processing |last2=Sondhi |first2=M. M. |last3=Huang |first3=Yiteng |date=2008 |publisher=Springer Science & Business Media |isbn=978-3540491255}} DTW processed speech by dividing it into short frames, e.g. 10ms segments, and processing each frame as a single unit. Although DTW would be superseded by later algorithms, the technique carried on. Achieving speaker independence remained unsolved at this time period.

• 1971 – DARPA funded five years for Speech Understanding Research, speech recognition research seeking a minimum vocabulary size of 1,000 words. They thought speech understanding would be key to making progress in speech recognition, but this later proved untrue.{{Cite web |last=John Makhoul |title=ISCA Medalist: For leadership and extensive contributions to speech and language processing |url=https://www.superlectures.com/interspeech2016/isca-medalist-for-leadership-and-extensive-contributions-to-speech-and-language-processing |url-status=live |archive-url=https://web.archive.org/web/20180124071005/https://www.superlectures.com/interspeech2016/isca-medalist-for-leadership-and-extensive-contributions-to-speech-and-language-processing |archive-date=24 January 2018 |access-date=23 January 2018 |df=dmy-all}} BBN, IBM, Carnegie Mellon and Stanford Research Institute all participated in the program.{{Cite magazine |last1=Blechman |first1=R. O. |last2=Blechman |first2=Nicholas |date=23 June 2008 |title=Hello, Hal |url=https://www.newyorker.com/magazine/2008/06/23/hello-hal |url-status=live |archive-url=https://web.archive.org/web/20150120042048/http://www.newyorker.com/magazine/2008/06/23/hello-hal |archive-date=20 January 2015 |access-date=17 January 2015 |magazine=The New Yorker |df=dmy-all}}{{Cite journal |last=Klatt |first=Dennis H. |year=1977 |title=Review of the ARPA speech understanding project |journal=The Journal of the Acoustical Society of America |volume=62 |issue=6 |pages=1345–1366 |bibcode=1977ASAJ...62.1345K |doi=10.1121/1.381666}} This revived speech recognition research post John Pierce's letter.
• 1972 – The IEEE Acoustics, Speech, and Signal Processing group held a conference in Newton, Massachusetts.
• 1976 – The first ICASSP was held in Philadelphia, which since then has been a major venue for the publication of research on speech recognition.{{Cite web |last=Rabiner |date=1984 |title=The Acoustics, Speech, and Signal Processing Society. A Historical Perspective |url=http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/216_historical%20perspective.pdf |url-status=live |archive-url=https://web.archive.org/web/20170809113828/http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/216_historical%20perspective.pdf |archive-date=9 August 2017 |access-date=23 January 2018 |df=dmy-all}}

During the late 1960s Leonard Baum developed the mathematics of Markov chains at the Institute for Defense Analysis. A decade later, at CMU, Raj Reddy's students James Baker and Janet M. Baker began using the hidden Markov model (HMM) for speech recognition.{{Cite web |date=12 January 2015 |title=First-Hand:The Hidden Markov Model – Engineering and Technology History Wiki |url=http://ethw.org/First-Hand:The_Hidden_Markov_Model |url-status=live |archive-url=https://web.archive.org/web/20180403191314/http://ethw.org/First-Hand:The_Hidden_Markov_Model |archive-date=3 April 2018 |access-date=1 May 2018 |website=ethw.org |df=dmy-all}} James Baker had learned about HMMs from a summer job at the Institute of Defense Analysis during his undergraduate education. The use of HMMs allowed researchers to combine different sources of knowledge, such as acoustics, language, and syntax, in a unified probabilistic model.

• By the mid-1980s IBM's Fred Jelinek's team created a voice activated typewriter called Tangora, which could handle a 20,000-word vocabulary{{Cite web |date=2012-03-07 |title=Pioneering Speech Recognition |url=http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/speechreco/ |url-status=dead |archive-url=https://web.archive.org/web/20150219080748/http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/speechreco/ |archive-date=19 February 2015 |access-date=18 January 2015 |df=dmy-all}} Jelinek's statistical approach put less emphasis on emulating the way the human brain processes and understands speech in favor of using statistical modeling techniques like HMMs. (Jelinek's group independently discovered the application of HMMs to speech.{{Cite web |title=James Baker interview |url=http://www.sarasinstitute.org/Audio/JimBaker(2006).mp3 |url-status=live |archive-url=https://web.archive.org/web/20170828105222/http://www.sarasinstitute.org/Audio/JimBaker(2006).mp3 |archive-date=28 August 2017 |access-date=9 February 2017 |df=dmy-all}}) This was controversial with linguists since HMMs are too simplistic to account for many common features of human languages.{{Cite journal |last1=Huang |first1=Xuedong |last2=Baker |first2=James |last3=Reddy |first3=Raj |date=January 2014 |title=A historical perspective of speech recognition |url=https://dl.acm.org/doi/fullHtml/10.1145/2500887 |journal=Communications of the ACM |language=en |volume=57 |issue=1 |pages=94–103 |doi=10.1145/2500887 |issn=0001-0782 |s2cid=6175701 |archive-url=https://web.archive.org/web/20231208161616/https://dl.acm.org/doi/fullHtml/10.1145/2500887 |archive-date=2023-12-08}} However, the HMM proved to be a highly useful way for modeling speech and replaced dynamic time warping to become the dominant speech recognition algorithm in the 1980s.{{Cite report |url=http://www.ece.ucsb.edu/faculty/Rabiner/ece259/Reprints/354_LALI-ASRHistory-final-10-8.pdf |title=Automatic speech recognition–a brief history of the technology development |last1=Juang |first1=B. H. |last2=Rabiner |first2=Lawrence R. |page=10 |access-date=17 January 2015 |archive-url=https://web.archive.org/web/20140817193243/http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/354_LALI-ASRHistory-final-10-8.pdf |archive-date=17 August 2014 |url-status=live}}{{Cite journal |last=Li |first=Xiaochang |date=2023-07-01 |title="There's No Data Like More Data": Automatic Speech Recognition and the Making of Algorithmic Culture |url=https://www.journals.uchicago.edu/doi/10.1086/725132 |journal=Osiris |language=en |volume=38 |pages=165–182 |doi=10.1086/725132 |issn=0369-7827 |s2cid=259502346}}
• 1982 – Dragon Systems, founded by James and Janet M. Baker,{{Cite web |title=History of Speech Recognition |url=http://www.dragon-medical-transcription.com/history_speech_recognition.html |archive-url=https://web.archive.org/web/20150813223326/http://dragon-medical-transcription.com/history_speech_recognition.html |archive-date=13 August 2015 |access-date=17 January 2015 |website=Dragon Medical Transcription}} was one of IBM's few competitors.

The 1980s also saw the introduction of the n-gram language model.

• 1987 – The back-off model allowed language models to use multiple length n-grams, and CSELT{{Cite journal |last1=Billi |first1=Roberto |last2=Canavesio |first2=Franco |last3=Ciaramella |first3=Alberto |last4=Nebbia |first4=Luciano |date=1 November 1995 |title=Interactive voice technology at work: The CSELT experience |url=https://www.sciencedirect.com/science/article/abs/pii/016763939500030R |journal=Speech Communication |volume=17 |issue=3 |pages=263–271 |doi=10.1016/0167-6393(95)00030-R}} used HMM to recognize languages (both in software and in hardware specialized processors, e.g. RIPAC).

Much of the progress in the field is owed to the rapidly increasing capabilities of computers. At the end of the DARPA program in 1976, the best computer available to researchers was the PDP-10 with 4 MB ram.{{Cite web |last1=Xuedong Huang |last2=James Baker |last3=Raj Reddy |date=January 2014 |title=A Historical Perspective of Speech Recognition |url=http://cacm.acm.org/magazines/2014/1/170863-a-historical-perspective-of-speech-recognition/fulltext#R5 |url-status=live |archive-url=http://archive.wikiwix.com/cache/20150120074239/http://cacm.acm.org/magazines/2014/1/170863-a-historical-perspective-of-speech-recognition/fulltext#R5 |archive-date=20 January 2015 |access-date=20 January 2015 |publisher=Communications of the ACM |df=dmy-all}} It could take up to 100 minutes to decode just 30 seconds of speech.{{Cite news |last=Kevin McKean |date=8 April 1980 |title=When Cole talks, computers listen |url=https://news.google.com/newspapers?nid=1798&dat=19800408&id=xgsdAAAAIBAJ&pg=6057,1141823 |access-date=23 November 2015 |publisher=Sarasota Journal |agency=AP}}

Two practical products were:

• 1984 – was released the Apricot Portable with up to 4096 words support, of which only 64 could be held in RAM at a time.{{Cite web |title=ACT/Apricot - Apricot history |url=http://actapricot.org/history/apricot_review_1.html |access-date=2016-02-02 |website=actapricot.org |archive-date=21 December 2016 |archive-url=https://web.archive.org/web/20161221091131/http://actapricot.org/history/apricot_review_1.html |url-status=live }}
• 1987 – a recognizer from Kurzweil Applied Intelligence
• 1990 – Dragon Dictate, a consumer product released in 1990{{Cite web |last=Melanie Pinola |date=2011-11-02 |title=Speech Recognition Through the Decades: How We Ended Up With Siri |url=http://www.pcworld.com/article/243060/speech_recognition_through_the_decades_how_we_ended_up_with_siri.html?page=2 |url-status=live |archive-url=https://web.archive.org/web/20170113074944/http://www.pcworld.com/article/243060/speech_recognition_through_the_decades_how_we_ended_up_with_siri.html?page=2 |archive-date=13 January 2017 |access-date=28 July 2017 |website=PC World |df=dmy-all}}{{Cite web |title=Ray Kurzweil biography |url=http://www.kurzweilai.net/ray-kurzweil-bio |url-status=live |archive-url=https://web.archive.org/web/20140205002828/http://www.kurzweilai.net/ray-kurzweil-bio |archive-date=5 February 2014 |access-date=25 September 2014 |publisher=KurzweilAINetwork |df=dmy-all}} AT&T deployed the Voice Recognition Call Processing service in 1992 to route telephone calls without the use of a human operator.{{Cite report |url=http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/354_LALI-ASRHistory-final-10-8.pdf |title=Automatic Speech Recognition – A Brief History of the Technology Development |last1=Juang |first1=B.H. |last2=Rabiner |first2=Lawrence |access-date=28 July 2017 |archive-url=https://web.archive.org/web/20170809211311/http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/354_LALI-ASRHistory-final-10-8.pdf |archive-date=9 August 2017 |url-status=live}} The technology was developed by Lawrence Rabiner and others at Bell Labs.

By this point, the vocabulary of the typical commercial speech recognition system was larger than the average human vocabulary. Raj Reddy's former student, Xuedong Huang, developed the Sphinx-II system at CMU. The Sphinx-II system was the first to do speaker-independent, large vocabulary, continuous speech recognition and it had the best performance in DARPA's 1992 evaluation. Handling continuous speech with a large vocabulary was a major milestone in the history of speech recognition. Huang went on to found the speech recognition group at Microsoft in 1993. Raj Reddy's student Kai-Fu Lee joined Apple where, in 1992, he helped develop a speech interface prototype for the Apple computer known as Casper.

Lernout & Hauspie, a Belgium-based speech recognition company, acquired several other companies, including Kurzweil Applied Intelligence in 1997 and Dragon Systems in 2000. The L&H speech technology was used in the Windows XP operating system. L&H was an industry leader until an accounting scandal brought an end to the company in 2001. The speech technology from L&H was bought by ScanSoft which became Nuance in 2005. Apple originally licensed software from Nuance to provide speech recognition capability to its digital assistant Siri.{{Cite web |date=10 October 2011 |title=Nuance Exec on iPhone 4S, Siri, and the Future of Speech |url=http://techpinions.com/nuance-exec-on-iphone-4s-siri-and-the-future-of-speech/3307 |url-status=live |archive-url=https://web.archive.org/web/20111119211021/http://techpinions.com/nuance-exec-on-iphone-4s-siri-and-the-future-of-speech/3307 |archive-date=19 November 2011 |access-date=23 November 2011 |publisher=Tech.pinions |df=dmy-all}}

In the 2000s DARPA sponsored two speech recognition programs: Effective Affordable Reusable Speech-to-Text (EARS) in 2002 and Global Autonomous Language Exploitation (GALE). Four teams participated in the EARS program: IBM, a team led by BBN with LIMSI and Univ. of Pittsburgh, Cambridge University, and a team composed of ICSI, SRI and University of Washington. EARS funded the collection of the Switchboard telephone speech corpus containing 260 hours of recorded conversations from over 500 speakers.{{Cite web |title=Switchboard-1 Release 2 |url=https://catalog.ldc.upenn.edu/LDC97S62 |url-status=live |archive-url=https://web.archive.org/web/20170711061225/https://catalog.ldc.upenn.edu/LDC97S62 |archive-date=11 July 2017 |access-date=26 July 2017 |df=dmy-all}} The GALE program focused on Arabic and Mandarin broadcast news speech. Google's first effort at speech recognition came in 2007 after hiring some researchers from Nuance.{{Cite web |last=Jason Kincaid |date=13 February 2011 |title=The Power of Voice: A Conversation With The Head Of Google's Speech Technology |url=https://techcrunch.com/2011/02/13/the-power-of-voice-a-conversation-with-the-head-of-googles-speech-technology/ |url-status=live |archive-url=https://web.archive.org/web/20150721034447/http://techcrunch.com/2011/02/13/the-power-of-voice-a-conversation-with-the-head-of-googles-speech-technology/ |archive-date=21 July 2015 |access-date=21 July 2015 |website=Tech Crunch |df=dmy-all}} The first product was GOOG-411, a telephone based directory service. The recordings from GOOG-411 produced valuable data that helped Google improve their recognition systems. Google Voice Search is now supported in over 30 languages.

In the United States, the National Security Agency has made use of a type of speech recognition for keyword spotting since at least 2006.{{Cite web |last=Froomkin |first=Dan |date=2015-05-05 |title=THE COMPUTERS ARE LISTENING |url=https://firstlook.org/theintercept/2015/05/05/nsa-speech-recognition-snowden-searchable-text/ |url-status=live |archive-url=https://web.archive.org/web/20150627185007/https://firstlook.org/theintercept/2015/05/05/nsa-speech-recognition-snowden-searchable-text/ |archive-date=27 June 2015 |access-date=20 June 2015 |website=The Intercept |df=dmy-all}} This technology allows analysts to search through large volumes of recorded conversations and isolate mentions of keywords. Recordings can be indexed and analysts can run queries over the database to find conversations of interest. Some government research programs focused on intelligence applications of speech recognition, e.g. DARPA's EARS's program and IARPA's Babel program.

In the early 2000s, speech recognition was still dominated by traditional approaches such as hidden Markov models combined with feedforward artificial neural networks.Herve Bourlard and Nelson Morgan, Connectionist Speech Recognition: A Hybrid Approach, The Kluwer International Series in Engineering and Computer Science; v. 247, Boston: Kluwer Academic Publishers, 1994.

Today, however, many aspects of speech recognition have been taken over by a deep learning method called Long short-term memory (LSTM), a recurrent neural network published by Sepp Hochreiter & Jürgen Schmidhuber in 1997.{{Cite journal |last1=Sepp Hochreiter |author-link=Sepp Hochreiter |last2=J. Schmidhuber |author-link2=Jürgen Schmidhuber |year=1997 |title=Long Short-Term Memory |journal=Neural Computation |volume=9 |issue=8 |pages=1735–1780 |doi=10.1162/neco.1997.9.8.1735 |pmid=9377276 |s2cid=1915014}} LSTM RNNs avoid the vanishing gradient problem and can learn "Very Deep Learning" tasks{{Cite journal |last=Schmidhuber |first=Jürgen |author-link=Jürgen Schmidhuber |year=2015 |title=Deep learning in neural networks: An overview |journal=Neural Networks |volume=61 |pages=85–117 |arxiv=1404.7828 |doi=10.1016/j.neunet.2014.09.003 |pmid=25462637 |s2cid=11715509}} that require memories of events that happened thousands of discrete time steps ago, which is important for speech.

Around 2007, LSTM trained by Connectionist Temporal Classification (CTC)Alex Graves, Santiago Fernandez, Faustino Gomez, and Jürgen Schmidhuber (2006). [https://mediatum.ub.tum.de/doc/1292048/file.pdf Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural nets] {{Webarchive|url=https://web.archive.org/web/20240909053409/https://mediatum.ub.tum.de/doc/1292048/file.pdf |date=9 September 2024 }}. Proceedings of ICML'06, pp. 369–376. started to outperform traditional speech recognition in certain applications.Santiago Fernandez, Alex Graves, and Jürgen Schmidhuber (2007). [http://www6.in.tum.de/pub/Main/Publications/Fernandez2007b.pdf An application of recurrent neural networks to discriminative keyword spotting]{{Dead link|date=March 2023 |bot=InternetArchiveBot |fix-attempted=yes }}. Proceedings of ICANN (2), pp. 220–229. In 2015, Google's speech recognition reportedly experienced a dramatic performance jump of 49% through CTC-trained LSTM, which is now available through Google Voice to all smartphone users.Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays and Johan Schalkwyk (September 2015): "{{Cite web |title=Google voice search: faster and more accurate |url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html |access-date=5 April 2016 |archive-date=9 March 2016 |archive-url=https://web.archive.org/web/20160309191532/http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html |url-status=dead }}." Transformers, a type of neural network based solely on "attention", have been widely adopted in computer vision{{Cite arXiv |eprint=2010.11929 |class=cs.CV |first1=Alexey |last1=Dosovitskiy |first2=Lucas |last2=Beyer |title=An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale |date=2021-06-03 |last3=Kolesnikov |first3=Alexander |last4=Weissenborn |first4=Dirk |last5=Zhai |first5=Xiaohua |last6=Unterthiner |first6=Thomas |last7=Dehghani |first7=Mostafa |last8=Minderer |first8=Matthias |last9=Heigold |first9=Georg |last10=Gelly |first10=Sylvain |last11=Uszkoreit |first11=Jakob |last12=Houlsby |first12=Neil}}{{Cite arXiv |eprint=2103.15808 |class=cs.CV |first1=Haiping |last1=Wu |first2=Bin |last2=Xiao |title=CvT: Introducing Convolutions to Vision Transformers |date=2021-03-29 |last3=Codella |first3=Noel |last4=Liu |first4=Mengchen |last5=Dai |first5=Xiyang |last6=Yuan |first6=Lu |last7=Zhang |first7=Lei}} and language modeling,{{Cite journal |last1=Vaswani |first1=Ashish |last2=Shazeer |first2=Noam |last3=Parmar |first3=Niki |last4=Uszkoreit |first4=Jakob |last5=Jones |first5=Llion |last6=Gomez |first6=Aidan N |last7=Kaiser |first7=Łukasz |last8=Polosukhin |first8=Illia |date=2017 |title=Attention is All you Need |url=https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates |volume=30 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053411/https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html |url-status=live }}{{Cite arXiv |eprint=1810.04805 |class=cs.CL |first1=Jacob |last1=Devlin |first2=Ming-Wei |last2=Chang |title=BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding |date=2019-05-24 |last3=Lee |first3=Kenton |last4=Toutanova |first4=Kristina}} sparking the interest of adapting such models to new domains, including speech recognition.{{Cite arXiv |eprint=2104.01778 |class=cs.SD |first1=Yuan |last1=Gong |first2=Yu-An |last2=Chung |title=AST: Audio Spectrogram Transformer |date=2021-07-08 |last3=Glass |first3=James}}{{Cite arXiv |eprint=2203.09581 |class=cs.CV |first1=Nicolae-Catalin |last1=Ristea |first2=Radu Tudor |last2=Ionescu |title=SepTr: Separable Transformer for Audio Spectrogram Processing |date=2022-06-20 |last3=Khan |first3=Fahad Shahbaz}}{{Cite arXiv |eprint=2104.00120 |class=eess.AS |first1=Timo |last1=Lohrenz |first2=Zhengyang |last2=Li |title=Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition |date=2021-07-14 |last3=Fingscheidt |first3=Tim}} Some recent papers reported superior performance levels using transformer models for speech recognition, but these models usually require large scale training datasets to reach high performance levels.

The use of deep feedforward (non-recurrent) networks for acoustic modeling was introduced during the later part of 2009 by Geoffrey Hinton and his students at the University of Toronto and by Li Deng{{Cite web |title=Li Deng |url=https://lidengsite.wordpress.com/ |publisher=Li Deng Site |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052323/https://lidengsite.wordpress.com/ |url-status=live }} and colleagues at Microsoft Research, initially in the collaborative work between Microsoft and the University of Toronto which was subsequently expanded to include IBM and Google (hence "The shared views of four research groups" subtitle in their 2012 review paper).NIPS Workshop: Deep Learning for Speech Recognition and Related Applications, Whistler, BC, Canada, Dec. 2009 (Organizers: Li Deng, Geoff Hinton, D. Yu). A Microsoft research executive called this innovation "the most dramatic change in accuracy since 1979".{{Cite news |last=Markoff |first=John |date=23 November 2012 |title=Scientists See Promise in Deep-Learning Programs |url=https://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html |url-status=live |archive-url=https://web.archive.org/web/20121130080314/http://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html |archive-date=30 November 2012 |access-date=20 January 2015 |work=New York Times |df=dmy-all}} In contrast to the steady incremental improvements of the past few decades, the application of deep learning decreased word error rate by 30%. This innovation was quickly adopted across the field. Researchers have begun to use deep learning techniques for language modeling as well.

In the long history of speech recognition, both shallow form and deep form (e.g. recurrent nets) of artificial neural networks had been explored for many years during 1980s, 1990s and a few years into the 2000s.Morgan, Bourlard, Renals, Cohen, Franco (1993) "Hybrid neural network/hidden Markov model systems for continuous speech recognition. ICASSP/IJPRAI"{{Cite book |last=T. Robinson |author-link=Tony Robinson (speech recognition) |title=[Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing |year=1992 |isbn=0-7803-0532-9 |pages=617–620 vol.1 |chapter=A real-time recurrent error propagation network word recognition system |doi=10.1109/ICASSP.1992.225833 |chapter-url=https://www.researchgate.net/publication/3532171 |s2cid=62446313}}Waibel, Hanazawa, Hinton, Shikano, Lang. (1989) "[http://www.inf.ufrgs.br/~engel/data/media/file/cmp121/waibel89_TDNN.pdf Phoneme recognition using time-delay neural networks] {{Webarchive|url=https://web.archive.org/web/20210225163001/http://www.inf.ufrgs.br/~engel/data/media/file/cmp121/waibel89_TDNN.pdf |date=25 February 2021 }}. IEEE Transactions on Acoustics, Speech, and Signal Processing."

But these methods never won over the non-uniform internal-handcrafting Gaussian mixture model/hidden Markov model (GMM-HMM) technology based on generative models of speech trained discriminatively.{{Cite journal |last1=Baker |first1=J. |last2=Li Deng |last3=Glass |first3=J. |last4=Khudanpur |first4=S. |last5=Chin-Hui Lee |author-link5=Chin-Hui Lee |last6=Morgan |first6=N. |last7=O'Shaughnessy |first7=D. |year=2009 |title=Developments and Directions in Speech Recognition and Understanding, Part 1 |journal=IEEE Signal Processing Magazine |volume=26 |issue=3 |pages=75–80 |bibcode=2009ISPM...26...75B |doi=10.1109/MSP.2009.932166 |s2cid=357467 |hdl-access=free |hdl=1721.1/51891}} A number of key difficulties had been methodologically analyzed in the 1990s, including gradient diminishingSepp Hochreiter (1991), [http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf Untersuchungen zu dynamischen neuronalen Netzen] {{webarchive|url=https://web.archive.org/web/20150306075401/http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf |date=6 March 2015 }}, Diploma thesis. Institut f. Informatik, Technische Univ. Munich. Advisor: J. Schmidhuber. and weak temporal correlation structure in the neural predictive models.{{Cite thesis |last=Bengio |first=Y. |title=Artificial Neural Networks and their Application to Speech/Sequence Recognition |degree=Ph.D. |publisher=McGill University |url=https://elibrary.ru/item.asp?id=5790854 |year=1991}}{{Cite journal |last1=Deng |first1=L. |last2=Hassanein |first2=K. |last3=Elmasry |first3=M. |year=1994 |title=Analysis of the correlation structure for a neural predictive model with application to speech recognition |journal=Neural Networks |volume=7 |issue=2 |pages=331–339 |doi=10.1016/0893-6080(94)90027-2}} All these difficulties were in addition to the lack of big training data and big computing power in these early days. Most speech recognition researchers who understood such barriers hence subsequently moved away from neural nets to pursue generative modeling approaches until the recent resurgence of deep learning starting around 2009–2010 that had overcome all these difficulties. Hinton et al. and Deng et al. reviewed part of this recent history about how their collaboration with each other and then with colleagues across four groups (University of Toronto, Microsoft, Google, and IBM) ignited a renaissance of applications of deep feedforward neural networks for speech recognition.{{Cite journal |last1=Hinton |first1=Geoffrey |last2=Deng |first2=Li |last3=Yu |first3=Dong |last4=Dahl |first4=George |last5=Mohamed |first5=Abdel-Rahman |last6=Jaitly |first6=Navdeep |last7=Senior |first7=Andrew |last8=Vanhoucke |first8=Vincent |last9=Nguyen |first9=Patrick |last10=Sainath |first10=Tara |author-link10=Tara Sainath |last11=Kingsbury |first11=Brian |year=2012 |title=Deep Neural Networks for Acoustic Modeling in Speech Recognition: The shared views of four research groups |journal=IEEE Signal Processing Magazine |volume=29 |issue=6 |pages=82–97 |bibcode=2012ISPM...29...82H |doi=10.1109/MSP.2012.2205597 |s2cid=206485943}}{{Cite book |last1=Deng |first1=L. |title=2013 IEEE International Conference on Acoustics, Speech and Signal Processing: New types of deep neural network learning for speech recognition and related applications: An overview |last2=Hinton |first2=G. |last3=Kingsbury |first3=B. |date=2013 |isbn=978-1-4799-0356-6 |pages=8599 |chapter=New types of deep neural network learning for speech recognition and related applications: An overview |doi=10.1109/ICASSP.2013.6639344 |s2cid=13953660}}Keynote talk: Recent Developments in Deep Neural Networks. ICASSP, 2013 (by Geoff Hinton).Keynote talk: "[https://www.isca-speech.org/archive/interspeech_2014/i14_3505.html Achievements and Challenges of Deep Learning: From Speech Analysis and Recognition To Language and Multimodal Processing] {{Webarchive|url=https://web.archive.org/web/20210305043518/https://www.isca-speech.org/archive/interspeech_2014/i14_3505.html|date=5 March 2021}}," Interspeech, September 2014 (by Li Deng).

By early 2010s speech recognition, also called voice recognition{{Cite web |date=27 August 2002 |title=Improvements in voice recognition software increase |url=https://www.techrepublic.com/article/improvements-in-voice-recognition-software-increase-productivity |url-status=dead |archive-url=https://web.archive.org/web/20181023080207/https://www.techrepublic.com/article/improvements-in-voice-recognition-software-increase-productivity/ |archive-date=23 October 2018 |access-date=22 October 2018 |website=TechRepublic.com |quote=Maners said IBM has worked on advancing speech recognition ... or on the floor of a noisy trade show.}}{{Cite web |date=3 March 1997 |title=Voice Recognition To Ease Travel Bookings: Business Travel News |url=http://www.businesstravelnews.com/More-News/Voice-Recognition-To-Ease-Travel-Bookings |website=BusinessTravelNews.com |quote=The earliest applications of speech recognition software were dictation ... Four months ago, IBM introduced a 'continual dictation product' designed to ... debuted at the National Business Travel Association trade show in 1994. |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052252/https://www.businesstravelnews.com/More-News/Voice-Recognition-To-Ease-Travel-Bookings |url-status=live }}{{Cite news |last=Ellis Booker |date=14 March 1994 |title=Voice recognition enters the mainstream |work=Computerworld |page=45 |quote=Just a few years ago, speech recognition was limited to ...}} was clearly differentiated from speaker recognition, and speaker independence was considered a major breakthrough. Until then, systems required a "training" period. A 1987 ad for a doll had carried the tagline "Finally, the doll that understands you." – despite the fact that it was described as "which children could train to respond to their voice".

In 2017, Microsoft researchers reached a historical human parity milestone of transcribing conversational telephony speech on the widely benchmarked Switchboard task. Multiple deep learning models were used to optimize speech recognition accuracy. The speech recognition word error rate was reported to be as low as 4 professional human transcribers working together on the same benchmark, which was funded by IBM Watson speech team on the same task.{{Cite web |date=21 August 2017 |title=Microsoft researchers achieve new conversational speech recognition milestone |url=https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/ |website=Microsoft |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052234/https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/ |url-status=live }}

Both acoustic modeling and language modeling are important parts of modern statistically based speech recognition algorithms. Hidden Markov models (HMMs) are widely used in many systems. Language modeling is also used in many other natural language processing applications such as document classification or statistical machine translation.

{{Main|Hidden Markov model}}

Modern general-purpose speech recognition systems are based on hidden Markov models. These are statistical models that output a sequence of symbols or quantities. HMMs are used in speech recognition because a speech signal can be viewed as a piecewise stationary signal or a short-time stationary signal. In a short time scale (e.g., 10 milliseconds), speech can be approximated as a stationary process. Speech can be thought of as a Markov model for many stochastic purposes.

Another reason why HMMs are popular is that they can be trained automatically and are simple and computationally feasible to use. In speech recognition, the hidden Markov model would output a sequence of n-dimensional real-valued vectors (with n being a small integer, such as 10), outputting one of these every 10 milliseconds. The vectors would consist of cepstral coefficients, which are obtained by taking a Fourier transform of a short time window of speech and decorrelating the spectrum using a cosine transform, then taking the first (most significant) coefficients. The hidden Markov model will tend to have in each state a statistical distribution that is a mixture of diagonal covariance Gaussians, which will give a likelihood for each observed vector. Each word, or (for more general speech recognition systems), each phoneme, will have a different output distribution; a hidden Markov model for a sequence of words or phonemes is made by concatenating the individual trained hidden Markov models for the separate words and phonemes.

Described above are the core elements of the most common, HMM-based approach to speech recognition. Modern speech recognition systems use various combinations of a number of standard techniques in order to improve results over the basic approach described above. A typical large-vocabulary system would need context dependency for the phonemes (so that phonemes with different left and right context would have different realizations as HMM states); it would use cepstral normalization to normalize for a different speaker and recording conditions; for further speaker normalization, it might use vocal tract length normalization (VTLN) for male-female normalization and maximum likelihood linear regression (MLLR) for more general speaker adaptation. The features would have so-called delta and delta-delta coefficients to capture speech dynamics and in addition, might use heteroscedastic linear discriminant analysis (HLDA); or might skip the delta and delta-delta coefficients and use splicing and an LDA-based projection followed perhaps by heteroscedastic linear discriminant analysis or a global semi-tied co variance transform (also known as maximum likelihood linear transform, or MLLT). Many systems use so-called discriminative training techniques that dispense with a purely statistical approach to HMM parameter estimation and instead optimize some classification-related measure of the training data. Examples are maximum mutual information (MMI), minimum classification error (MCE), and minimum phone error (MPE).

Decoding of the speech (the term for what happens when the system is presented with a new utterance and must compute the most likely source sentence) would probably use the Viterbi algorithm to find the best path, and here there is a choice between dynamically creating a combination hidden Markov model, which includes both the acoustic and language model information and combining it statically beforehand (the finite state transducer, or FST, approach).

A possible improvement to decoding is to keep a set of good candidates instead of just keeping the best candidate, and to use a better scoring function (re scoring) to rate these good candidates so that we may pick the best one according to this refined score. The set of candidates can be kept either as a list (the N-best list approach) or as a subset of the models (a lattice). Re scoring is usually done by trying to minimize the Bayes risk{{Cite journal |last1=Goel |first1=Vaibhava |last2=Byrne |first2=William J. |year=2000 |title=Minimum Bayes-risk automatic speech recognition |url=http://www.clsp.jhu.edu/people/vgoel/publications/CSAL.ps |url-status=live |journal=Computer Speech & Language |volume=14 |issue=2 |pages=115–135 |doi=10.1006/csla.2000.0138 |s2cid=206561058 |archive-url=https://web.archive.org/web/20110725225846/http://www.clsp.jhu.edu/people/vgoel/publications/CSAL.ps |archive-date=25 July 2011 |access-date=28 March 2011 |doi-access=free |df=dmy-all}} (or an approximation thereof) Instead of taking the source sentence with maximal probability, we try to take the sentence that minimizes the expectancy of a given loss function with regards to all possible transcriptions (i.e., we take the sentence that minimizes the average distance to other possible sentences weighted by their estimated probability). The loss function is usually the Levenshtein distance, though it can be different distances for specific tasks; the set of possible transcriptions is, of course, pruned to maintain tractability. Efficient algorithms have been devised to re score lattices represented as weighted finite state transducers with edit distances represented themselves as a finite state transducer verifying certain assumptions.{{Cite journal |last=Mohri |first=M. |year=2002 |title=Edit-Distance of Weighted Automata: General Definitions and Algorithms |url=http://www.cs.nyu.edu/~mohri/pub/edit.pdf |url-status=live |journal=International Journal of Foundations of Computer Science |volume=14 |issue=6 |pages=957–982 |doi=10.1142/S0129054103002114 |archive-url=https://web.archive.org/web/20120318032640/http://www.cs.nyu.edu/~mohri/pub/edit.pdf |archive-date=18 March 2012 |access-date=28 March 2011 |df=dmy-all}}

{{Main|Dynamic time warping}}

Dynamic time warping is an approach that was historically used for speech recognition but has now largely been displaced by the more successful HMM-based approach.

Dynamic time warping is an algorithm for measuring similarity between two sequences that may vary in time or speed. For instance, similarities in walking patterns would be detected, even if in one video the person was walking slowly and if in another he or she were walking more quickly, or even if there were accelerations and deceleration during the course of one observation. DTW has been applied to video, audio, and graphics – indeed, any data that can be turned into a linear representation can be analyzed with DTW.

A well-known application has been automatic speech recognition, to cope with different speaking speeds. In general, it is a method that allows a computer to find an optimal match between two given sequences (e.g., time series) with certain restrictions. That is, the sequences are "warped" non-linearly to match each other. This sequence alignment method is often used in the context of hidden Markov models.

{{Main|Artificial neural network}}

Neural networks emerged as an attractive acoustic modeling approach in ASR in the late 1980s. Since then, neural networks have been used in many aspects of speech recognition such as phoneme classification,{{Cite journal |last1=Waibel |first1=A. |last2=Hanazawa |first2=T. |last3=Hinton |first3=G. |last4=Shikano |first4=K. |last5=Lang |first5=K. J. |year=1989 |title=Phoneme recognition using time-delay neural networks |journal=IEEE Transactions on Acoustics, Speech, and Signal Processing |volume=37 |issue=3 |pages=328–339 |doi=10.1109/29.21701 |s2cid=9563026 |hdl-access=free |hdl=10338.dmlcz/135496}} phoneme classification through multi-objective evolutionary algorithms,{{Cite journal |last1=Bird |first1=Jordan J. |last2=Wanner |first2=Elizabeth |last3=Ekárt |first3=Anikó |last4=Faria |first4=Diego R. |year=2020 |title=Optimisation of phonetic aware speech recognition through multi-objective evolutionary algorithms |url=https://publications.aston.ac.uk/id/eprint/41416/1/Speech_Recog_ESWA_2_.pdf |journal=Expert Systems with Applications |publisher=Elsevier BV |volume=153 |page=113402 |doi=10.1016/j.eswa.2020.113402 |issn=0957-4174 |s2cid=216472225 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053419/https://publications.aston.ac.uk/id/eprint/41416/1/Speech_Recog_ESWA_2_.pdf |url-status=live }} isolated word recognition,{{Cite journal |last1=Wu |first1=J. |last2=Chan |first2=C. |year=1993 |title=Isolated Word Recognition by Neural Network Models with Cross-Correlation Coefficients for Speech Dynamics |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=15 |issue=11 |pages=1174–1185 |doi=10.1109/34.244678}} audiovisual speech recognition, audiovisual speaker recognition and speaker adaptation.

Neural networks make fewer explicit assumptions about feature statistical properties than HMMs and have several qualities making them more attractive recognition models for speech recognition. When used to estimate the probabilities of a speech feature segment, neural networks allow discriminative training in a natural and efficient manner. However, in spite of their effectiveness in classifying short-time units such as individual phonemes and isolated words,S. A. Zahorian, A. M. Zimmer, and F. Meng, (2002) "[https://www.researchgate.net/profile/Stephen_Zahorian/publication/221480228_Vowel_classification_for_computer-based_visual_feedback_for_speech_training_for_the_hearing_impaired/links/00b7d525d25f51c585000000.pdf Vowel Classification for Computer based Visual Feedback for Speech Training for the Hearing Impaired]," in ICSLP 2002 early neural networks were rarely successful for continuous recognition tasks because of their limited ability to model temporal dependencies.

One approach to this limitation was to use neural networks as a pre-processing, feature transformation or dimensionality reduction,{{Cite book |last1=Hu |first1=Hongbing |title=ICASSP 2010 |last2=Zahorian |first2=Stephen A. |year=2010 |chapter=Dimensionality Reduction Methods for HMM Phonetic Recognition |chapter-url=http://bingweb.binghamton.edu/~hhu1/paper/Hu2010Dimensionality.pdf |archive-url=http://archive.wikiwix.com/cache/20120706063756/http://bingweb.binghamton.edu/~hhu1/paper/Hu2010Dimensionality.pdf |archive-date=6 July 2012 |url-status=live |df=dmy-all}} step prior to HMM based recognition. However, more recently, LSTM and related recurrent neural networks (RNNs),{{Cite book |last1=Fernandez |first1=Santiago |title=Proceedings of IJCAI |last2=Graves |first2=Alex |last3=Schmidhuber |first3=Jürgen |author-link3=Jürgen Schmidhuber |year=2007 |chapter=Sequence labelling in structured domains with hierarchical recurrent neural networks |chapter-url=http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-124.pdf |archive-url=https://web.archive.org/web/20170815003130/http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-124.pdf |archive-date=15 August 2017 |url-status=live |df=dmy-all}}{{Cite arXiv |eprint=1303.5778 |class=cs.NE |first1=Alex |last1=Graves |first2=Abdel-rahman |last2=Mohamed |title=Speech recognition with deep recurrent neural networks |first3=Geoffrey |last3=Hinton |year=2013}} ICASSP 2013. Time Delay Neural Networks(TDNN's),{{Cite journal |last=Waibel |first=Alex |year=1989 |title=Modular Construction of Time-Delay Neural Networks for Speech Recognition |url=http://isl.anthropomatik.kit.edu/cmu-kit/Modular_Construction_of_Time-Delay_Neural_Networks_for_Speech_Recognition.pdf |url-status=live |journal=Neural Computation |volume=1 |issue=1 |pages=39–46 |doi=10.1162/neco.1989.1.1.39 |s2cid=236321 |archive-url=https://web.archive.org/web/20160629180846/http://isl.anthropomatik.kit.edu/cmu-kit/Modular_Construction_of_Time-Delay_Neural_Networks_for_Speech_Recognition.pdf |archive-date=29 June 2016 |df=dmy-all}} and transformers have demonstrated improved performance in this area.

{{Main|Deep learning}}

Deep neural networks and denoising autoencoders{{Cite book |last1=Maas |first1=Andrew L. |title=Proceedings of Interspeech 2012 |last2=Le |first2=Quoc V. |last3=O'Neil |first3=Tyler M. |last4=Vinyals |first4=Oriol |last5=Nguyen |first5=Patrick |last6=Ng |first6=Andrew Y. |author-link6=Andrew Ng |year=2012 |chapter=Recurrent Neural Networks for Noise Reduction in Robust ASR}} are also under investigation. A deep feedforward neural network (DNN) is an artificial neural network with multiple hidden layers of units between the input and output layers. Similar to shallow neural networks, DNNs can model complex non-linear relationships. DNN architectures generate compositional models, where extra layers enable composition of features from lower layers, giving a huge learning capacity and thus the potential of modeling complex patterns of speech data.

A success of DNNs in large vocabulary speech recognition occurred in 2010 by industrial researchers, in collaboration with academic researchers, where large output layers of the DNN based on context dependent HMM states constructed by decision trees were adopted.{{Cite journal |last1=Yu |first1=D. |last2=Deng |first2=L. |last3=Dahl |first3=G. |date=2010 |title=Roles of Pre-Training and Fine-Tuning in Context-Dependent DBN-HMMs for Real-World Speech Recognition |url=https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/dbn4asr-nips2010.pdf |journal=NIPS Workshop on Deep Learning and Unsupervised Feature Learning}}{{Cite journal |last1=Dahl |first1=George E. |last2=Yu |first2=Dong |last3=Deng |first3=Li |last4=Acero |first4=Alex |date=2012 |title=Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition |journal=IEEE Transactions on Audio, Speech, and Language Processing |volume=20 |issue=1 |pages=30–42 |doi=10.1109/TASL.2011.2134090 |s2cid=14862572}}

Deng L., Li, J., Huang, J., Yao, K., Yu, D., Seide, F. et al. [https://pdfs.semanticscholar.org/6bdc/cfe195bc49d218acc5be750aa49e41f408e4.pdf Recent Advances in Deep Learning for Speech Research at Microsoft] {{Webarchive|url=https://web.archive.org/web/20240909052236/https://pdfs.semanticscholar.org/6bdc/cfe195bc49d218acc5be750aa49e41f408e4.pdf |date=9 September 2024 }}. ICASSP, 2013. See comprehensive reviews of this development and of the state of the art as of October 2014 in the recent Springer book from Microsoft Research. See also the related background of automatic speech recognition and the impact of various machine learning paradigms, notably including deep learning, in

recent overview articles.{{Cite journal |last1=Deng |first1=L. |last2=Li |first2=Xiao |date=2013 |title=Machine Learning Paradigms for Speech Recognition: An Overview |url=http://cvsp.cs.ntua.gr/courses/patrec/slides_material2018/slides-2018/DengLi_MLParadigms-SpeechRecogn-AnOverview_TALSP13.pdf |journal=IEEE Transactions on Audio, Speech, and Language Processing |volume=21 |issue=5 |pages=1060–1089 |doi=10.1109/TASL.2013.2244083 |s2cid=16585863 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052239/http://cvsp.cs.ntua.gr/courses/patrec/slides_material2018/slides-2018/DengLi_MLParadigms-SpeechRecogn-AnOverview_TALSP13.pdf |url-status=live }}{{Cite journal |last=Schmidhuber |first=Jürgen |author-link=Jürgen Schmidhuber |year=2015 |title=Deep Learning |journal=Scholarpedia |volume=10 |issue=11 |page=32832 |bibcode=2015SchpJ..1032832S |doi=10.4249/scholarpedia.32832 |doi-access=free}}

One fundamental principle of deep learning is to do away with hand-crafted feature engineering and to use raw features. This principle was first explored successfully in the architecture of deep autoencoder on the "raw" spectrogram or linear filter-bank features,L. Deng, M. Seltzer, D. Yu, A. Acero, A. Mohamed, and G. Hinton (2010) [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.185.1908&rep=rep1&type=pdf Binary Coding of Speech Spectrograms Using a Deep Auto-encoder]. Interspeech. showing its superiority over the Mel-Cepstral features which contain a few stages of fixed transformation from spectrograms.

The true "raw" features of speech, waveforms, have more recently been shown to produce excellent larger-scale speech recognition results.{{Cite book |last1=Tüske |first1=Zoltán |title=Interspeech 2014 |last2=Golik |first2=Pavel |last3=Schlüter |first3=Ralf |last4=Ney |first4=Hermann |year=2014 |chapter=Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR |chapter-url=https://www-i6.informatik.rwth-aachen.de/publications/download/937/T%7Bu%7DskeZolt%7Ba%7DnGolikPavelSchl%7Bu%7DterRalfNeyHermann--AcousticModelingwithDeepNeuralNetworksUsingRawTimeSignalfor%7BLVCSR%7D--2014.pdf |archive-url=https://web.archive.org/web/20161221174753/https://www-i6.informatik.rwth-aachen.de/publications/download/937/T%7Bu%7DskeZolt%7Ba%7DnGolikPavelSchl%7Bu%7DterRalfNeyHermann--AcousticModelingwithDeepNeuralNetworksUsingRawTimeSignalfor%7BLVCSR%7D--2014.pdf |archive-date=21 December 2016 |url-status=live |df=dmy-all}}

Since 2014, there has been much research interest in "end-to-end" ASR. Traditional phonetic-based (i.e., all HMM-based model) approaches required separate components and training for the pronunciation, acoustic, and language model. End-to-end models jointly learn all the components of the speech recognizer. This is valuable since it simplifies the training process and deployment process. For example, a n-gram language model is required for all HMM-based systems, and a typical n-gram language model often takes several gigabytes in memory making them impractical to deploy on mobile devices.{{Cite book |last=Jurafsky |first=Daniel |title=Speech and Language Processing |year=2016}} Consequently, modern commercial ASR systems from Google and Apple ({{as of|2017|lc=y}}) are deployed on the cloud and require a network connection as opposed to the device locally.

The first attempt at end-to-end ASR was with Connectionist Temporal Classification (CTC)-based systems introduced by Alex Graves of Google DeepMind and Navdeep Jaitly of the University of Toronto in 2014.{{Cite journal |last=Graves |first=Alex |year=2014 |title=Towards End-to-End Speech Recognition with Recurrent Neural Networks |url=http://www.jmlr.org/proceedings/papers/v32/graves14.pdf |url-status=dead |journal=ICML |archive-url=https://web.archive.org/web/20170110184531/http://jmlr.org/proceedings/papers/v32/graves14.pdf |archive-date=10 January 2017 |access-date=22 July 2019}} The model consisted of recurrent neural networks and a CTC layer. Jointly, the RNN-CTC model learns the pronunciation and acoustic model together, however it is incapable of learning the language due to conditional independence assumptions similar to a HMM. Consequently, CTC models can directly learn to map speech acoustics to English characters, but the models make many common spelling mistakes and must rely on a separate language model to clean up the transcripts. Later, Baidu expanded on the work with extremely large datasets and demonstrated some commercial success in Chinese Mandarin and English.{{Cite arXiv |eprint=1512.02595 |class=cs.CL |first=Dario |last=Amodei |title=Deep Speech 2: End-to-End Speech Recognition in English and Mandarin |year=2016}} In 2016, University of Oxford presented LipNet,{{Cite web |date=4 November 2016 |title=LipNet: How easy do you think lipreading is? |url=https://www.youtube.com/watch?v=fa5QGremQf8 |url-status=live |archive-url=https://web.archive.org/web/20170427104009/https://www.youtube.com/watch?v=fa5QGremQf8 |archive-date=27 April 2017 |access-date=5 May 2017 |website=YouTube |df=dmy-all}} the first end-to-end sentence-level lipreading model, using spatiotemporal convolutions coupled with an RNN-CTC architecture, surpassing human-level performance in a restricted grammar dataset.{{Cite arXiv |eprint=1611.01599 |class=cs.CV |first1=Yannis |last1=Assael |first2=Brendan |last2=Shillingford |title=LipNet: End-to-End Sentence-level Lipreading |date=5 November 2016 |last3=Whiteson |first3=Shimon |last4=de Freitas |first4=Nando}} A large-scale CNN-RNN-CTC architecture was presented in 2018 by Google DeepMind achieving 6 times better performance than human experts.{{Cite arXiv |eprint=1807.05162 |class=cs.CV |first1=Brendan |last1=Shillingford |first2=Yannis |last2=Assael |title=Large-Scale Visual Speech Recognition |date=2018-07-13 |last3=Hoffman |first3=Matthew W. |last4=Paine |first4=Thomas |last5=Hughes |first5=Cían |last6=Prabhu |first6=Utsav |last7=Liao |first7=Hank |last8=Sak |first8=Hasim |last9=Rao |first9=Kanishka}} In 2019, Nvidia launched two CNN-CTC ASR models, Jasper and QuarzNet, with an overall performance WER of 3%.{{Cite book |last1=Li |first1=Jason |last2=Lavrukhin |first2=Vitaly |last3=Ginsburg |first3=Boris |last4=Leary |first4=Ryan |last5=Kuchaiev |first5=Oleksii |last6=Cohen |first6=Jonathan M. |last7=Nguyen |first7=Huyen |last8=Gadde |first8=Ravi Teja |title=Interspeech 2019 |date=2019 |chapter=Jasper: An End-to-End Convolutional Neural Acoustic Model |chapter-url=https://www.isca-archive.org/interspeech_2019/li19_interspeech.html |pages=71–75 |doi=10.21437/Interspeech.2019-1819|arxiv=1904.03288 }}{{Citation |last1=Kriman |first1=Samuel |title=QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions |date=2019-10-22 |arxiv=1910.10261 |last2=Beliaev |first2=Stanislav |last3=Ginsburg |first3=Boris |last4=Huang |first4=Jocelyn |last5=Kuchaiev |first5=Oleksii |last6=Lavrukhin |first6=Vitaly |last7=Leary |first7=Ryan |last8=Li |first8=Jason |last9=Zhang |first9=Yang}} Similar to other deep learning applications, transfer learning and domain adaptation are important strategies for reusing and extending the capabilities of deep learning models, particularly due to the high costs of training models from scratch, and the small size of available corpus in many languages and/or specific domains.{{Cite journal |last1=Medeiros |first1=Eduardo |last2=Corado |first2=Leonel |last3=Rato |first3=Luís |last4=Quaresma |first4=Paulo |last5=Salgueiro |first5=Pedro |date=May 2023 |title=Domain Adaptation Speech-to-Text for Low-Resource European Portuguese Using Deep Learning |journal=Future Internet |language=en |volume=15 |issue=5 |pages=159 |doi=10.3390/fi15050159 |doi-access=free |issn=1999-5903}}{{Cite journal |last1=Joshi |first1=Raviraj |last2=Singh |first2=Anupam |date=May 2022 |editor-last=Malmasi |editor-first=Shervin |editor2-last=Rokhlenko |editor2-first=Oleg |editor3-last=Ueffing |editor3-first=Nicola |editor4-last=Guy |editor4-first=Ido |editor5-last=Agichtein |editor5-first=Eugene |editor6-last=Kallumadi |editor6-first=Surya |title=A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Data |url=https://aclanthology.org/2022.ecnlp-1.28/ |journal=Proceedings of the Fifth Workshop on E-Commerce and NLP (ECNLP 5) |location=Dublin, Ireland |publisher=Association for Computational Linguistics |pages=244–249 |doi=10.18653/v1/2022.ecnlp-1.28|arxiv=2206.13240 }}{{Cite book |last1=Sukhadia |first1=Vrunda N. |last2=Umesh |first2=S. |chapter=Domain Adaptation of Low-Resource Target-Domain Models Using Well-Trained ASR Conformer Models |date=2023-01-09 |title=2022 IEEE Spoken Language Technology Workshop (SLT) |chapter-url=https://ieeexplore.ieee.org/document/10023233 |publisher=IEEE |pages=295–301 |doi=10.1109/SLT54892.2023.10023233 |arxiv=2202.09167 |isbn=979-8-3503-9690-4}}

An alternative approach to CTC-based models are attention-based models. Attention-based ASR models were introduced simultaneously by Chan et al. of Carnegie Mellon University and Google Brain and Bahdanau et al. of the University of Montreal in 2016.{{Cite journal |last1=Chan |first1=William |last2=Jaitly |first2=Navdeep |last3=Le |first3=Quoc |last4=Vinyals |first4=Oriol |year=2016 |title=Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition |url=https://storage.googleapis.com/pub-tools-public-publication-data/pdf/44926.pdf |journal=ICASSP |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053931/https://storage.googleapis.com/pub-tools-public-publication-data/pdf/44926.pdf |url-status=live }}{{Cite arXiv |eprint=1508.04395 |class=cs.CL |first=Dzmitry |last=Bahdanau |title=End-to-End Attention-based Large Vocabulary Speech Recognition |year=2016}} The model named "Listen, Attend and Spell" (LAS), literally "listens" to the acoustic signal, pays "attention" to different parts of the signal and "spells" out the transcript one character at a time. Unlike CTC-based models, attention-based models do not have conditional-independence assumptions and can learn all the components of a speech recognizer including the pronunciation, acoustic and language model directly. This means, during deployment, there is no need to carry around a language model making it very practical for applications with limited memory. By the end of 2016, the attention-based models have seen considerable success including outperforming the CTC models (with or without an external language model).{{Cite arXiv |eprint=1612.02695 |class=cs.NE |first1=Jan |last1=Chorowski |first2=Navdeep |last2=Jaitly |title=Towards better decoding and language model integration in sequence to sequence models |date=8 December 2016}} Various extensions have been proposed since the original LAS model. Latent Sequence Decompositions (LSD) was proposed by Carnegie Mellon University, MIT and Google Brain to directly emit sub-word units which are more natural than English characters;{{Cite arXiv |eprint=1610.03035 |class=stat.ML |first1=William |last1=Chan |first2=Yu |last2=Zhang |title=Latent Sequence Decompositions |date=10 October 2016 |last3=Le |first3=Quoc |last4=Jaitly |first4=Navdeep}} University of Oxford and Google DeepMind extended LAS to "Watch, Listen, Attend and Spell" (WLAS) to handle lip reading surpassing human-level performance.{{Cite book |last1=Chung |first1=Joon Son |title=2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) |last2=Senior |first2=Andrew |last3=Vinyals |first3=Oriol |last4=Zisserman |first4=Andrew |date=16 November 2016 |isbn=978-1-5386-0457-1 |pages=3444–3453 |chapter=Lip Reading Sentences in the Wild |doi=10.1109/CVPR.2017.367 |arxiv=1611.05358 |s2cid=1662180}}

Typically a manual control input, for example by means of a finger control on the steering-wheel, enables the speech recognition system and this is signaled to the driver by an audio prompt. Following the audio prompt, the system has a "listening window" during which it may accept a speech input for recognition. {{citation needed|date=March 2014}}

Simple voice commands may be used to initiate phone calls, select radio stations or play music from a compatible smartphone, MP3 player or music-loaded flash drive. Voice recognition capabilities vary between car make and model. Some of the most recent{{When|date=April 2014}} car models offer natural-language speech recognition in place of a fixed set of commands, allowing the driver to use full sentences and common phrases. With such systems there is, therefore, no need for the user to memorize a set of fixed command words.{{citation needed|date=March 2014}}

{{main|Pronunciation assessment}}

Automatic pronunciation assessment is the use of speech recognition to verify the correctness of pronounced speech,{{Citation |last1=El Kheir |first1=Yassine |title=Automatic Pronunciation Assessment — A Review |date=October 21, 2023 |publisher=Conference on Empirical Methods in Natural Language Processing |arxiv=2310.13974 |s2cid=264426545 |display-authors=1 |last2=Ali |first2=Ahmed}} as distinguished from manual assessment by an instructor or proctor.{{Cite journal |last1=Isaacs |first1=Talia |last2=Harding |first2=Luke |date=July 2017 |title=Pronunciation assessment |journal=Language Teaching |language=en |volume=50 |issue=3 |pages=347–366 |doi=10.1017/S0261444817000118 |issn=0261-4448 |s2cid=209353525 |doi-access=free}} Also called speech verification, pronunciation evaluation, and pronunciation scoring, the main application of this technology is computer-aided pronunciation teaching (CAPT) when combined with computer-aided instruction for computer-assisted language learning (CALL), speech remediation, or accent reduction. Pronunciation assessment does not determine unknown speech (as in dictation or automatic transcription) but instead, knowing the expected word(s) in advance, it attempts to verify the correctness of the learner's pronunciation and ideally their intelligibility to listeners,{{Citation |last1=Loukina |first1=Anastassia |title=INTERSPEECH 2015 |date=September 6, 2015 |pages=1917–1921 |chapter=Pronunciation accuracy and intelligibility of non-native speech |chapter-url=https://www.isca-speech.org/archive/pdfs/interspeech_2015/loukina15_interspeech.pdf |place=Dresden, Germany |publisher=International Speech Communication Association |quote=only 16% of the variability in word-level intelligibility can be explained by the presence of obvious mispronunciations. |display-authors=1 |last2=Lopez |first2=Melissa |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053932/https://www.isca-speech.org/archive/pdfs/interspeech_2015/loukina15_interspeech.pdf |url-status=live }}{{Cite journal |last1=O’Brien |first1=Mary Grantham |last2=Derwing |first2=Tracey M. |display-authors=1 |date=31 December 2018 |title=Directions for the future of technology in pronunciation research and teaching |journal=Journal of Second Language Pronunciation |language=en |volume=4 |issue=2 |pages=182–207 |doi=10.1075/jslp.17001.obr |issn=2215-1931 |s2cid=86440885 |quote=pronunciation researchers are primarily interested in improving L2 learners’ intelligibility and comprehensibility, but they have not yet collected sufficient amounts of representative and reliable data (speech recordings with corresponding annotations and judgments) indicating which errors affect these speech dimensions and which do not. These data are essential to train ASR algorithms to assess L2 learners’ intelligibility. |doi-access=free |hdl-access=free |hdl=2066/199273}} sometimes along with often inconsequential prosody such as intonation, pitch, tempo, rhythm, and stress.{{Cite journal |last=Eskenazi |first=Maxine |date=January 1999 |title=Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype |url=https://www.lltjournal.org/item/10125-25043/ |journal=Language Learning & Technology |language=en |volume=2 |issue=2 |pages=62–76 |access-date=11 February 2023 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053942/https://www.lltjournal.org/item/10125-25043/ |url-status=live }} Pronunciation assessment is also used in reading tutoring, for example in products such as Microsoft Teams{{Cite news |last=Tholfsen |first=Mike |date=9 February 2023 |title=Reading Coach in Immersive Reader plus new features coming to Reading Progress in Microsoft Teams |url=https://techcommunity.microsoft.com/t5/education-blog/reading-coach-in-immersive-reader-plus-new-features-coming-to/ba-p/3734079 |access-date=12 February 2023 |work=Techcommunity Education Blog |publisher=Microsoft |language=en |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052822/https://techcommunity.microsoft.com/t5/education-blog/reading-coach-in-immersive-reader-plus-new-features-coming-to/ba-p/3734079 |url-status=live }} and from Amira Learning.{{Cite news |last=Banerji |first=Olina |date=7 March 2023 |title=Schools Are Using Voice Technology to Teach Reading. Is It Helping? |url=https://www.edsurge.com/news/2023-03-07-schools-are-using-voice-technology-to-teach-reading-is-it-helping |access-date=7 March 2023 |work=EdSurge News |language=en |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054611/https://www.edsurge.com/news/2023-03-07-schools-are-using-voice-technology-to-teach-reading-is-it-helping |url-status=live }} Automatic pronunciation assessment can also be used to help diagnose and treat speech disorders such as apraxia.{{Cite book |last1=Hair |first1=Adam |url=https://psi.engr.tamu.edu/wp-content/uploads/2018/04/hair2018idc.pdf |title=Proceedings of the 17th ACM Conference on Interaction Design and Children |last2=Monroe |first2=Penelope |date=19 June 2018 |isbn=9781450351522 |pages=119–131 |chapter=Apraxia world: A speech therapy game for children with speech sound disorders |doi=10.1145/3202185.3202733 |display-authors=1 |s2cid=13790002 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052803/https://psi.engr.tamu.edu/wp-content/uploads/2018/04/hair2018idc.pdf |url-status=live }}

Assessing authentic listener intelligibility is essential for avoiding inaccuracies from accent bias, especially in high-stakes assessments;{{Cite news |date=8 August 2017 |title=Computer says no: Irish vet fails oral English test needed to stay in Australia |url=https://www.theguardian.com/australia-news/2017/aug/08/computer-says-no-irish-vet-fails-oral-english-test-needed-to-stay-in-australia |access-date=12 February 2023 |work=The Guardian |agency=Australian Associated Press |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909052806/https://www.theguardian.com/australia-news/2017/aug/08/computer-says-no-irish-vet-fails-oral-english-test-needed-to-stay-in-australia |url-status=live }}{{Cite news |last=Ferrier |first=Tracey |date=9 August 2017 |title=Australian ex-news reader with English degree fails robot's English test |url=https://www.smh.com.au/technology/australian-exnews-reader-with-english-degree-fails-robots-english-test-20170809-gxsjv2.html |access-date=12 February 2023 |work=The Sydney Morning Herald |language=en |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053307/https://www.smh.com.au/technology/australian-exnews-reader-with-english-degree-fails-robots-english-test-20170809-gxsjv2.html |url-status=live }}{{Cite news |last1=Main |first1=Ed |last2=Watson |first2=Richard |date=9 February 2022 |title=The English test that ruined thousands of lives |url=https://www.bbc.com/news/uk-60264106 |access-date=12 February 2023 |work=BBC News |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054614/https://www.bbc.com/news/uk-60264106 |url-status=live }} from words with multiple correct pronunciations;{{Cite web |last=Joyce |first=Katy Spratte |date=January 24, 2023 |title=13 Words That Can Be Pronounced Two Ways |url=https://www.rd.com/list/words-that-can-be-pronounced-two-ways/ |access-date=23 February 2023 |publisher=Reader's Digest |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054447/https://www.rd.com/list/words-that-can-be-pronounced-two-ways/ |url-status=live }} and from phoneme coding errors in machine-readable pronunciation dictionaries.E.g., CMUDICT, {{Cite web |title=The CMU Pronouncing Dictionary |url=http://www.speech.cs.cmu.edu/cgi-bin/cmudict |access-date=15 February 2023 |website=www.speech.cs.cmu.edu |archive-date=15 August 2010 |archive-url=https://web.archive.org/web/20100815023012/http://www.speech.cs.cmu.edu/cgi-bin/cmudict |url-status=live }} Compare "four" given as "F AO R" with the vowel AO as in "caught," to "row" given as "R OW" with the vowel OW as in "oat." In 2022, researchers found that some newer speech to text systems, based on end-to-end reinforcement learning to map audio signals directly into words, produce word and phrase confidence scores very closely correlated with genuine listener intelligibility.{{Cite conference |last1=Tu |first1=Zehai |last2=Ma |first2=Ning |last3=Barker |first3=Jon |date=2022 |title=Unsupervised Uncertainty Measures of Automatic Speech Recognition for Non-intrusive Speech Intelligibility Prediction |url=https://www.isca-speech.org/archive/pdfs/interspeech_2022/tu22b_interspeech.pdf |conference=INTERSPEECH 2022 |publisher=ISCA |pages=3493–3497 |doi=10.21437/Interspeech.2022-10408 |access-date=17 December 2023 |book-title=Proc. Interspeech 2022 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053824/https://www.isca-speech.org/archive/pdfs/interspeech_2022/tu22b_interspeech.pdf |url-status=live }} In the Common European Framework of Reference for Languages (CEFR) assessment criteria for "overall phonological control", intelligibility outweighs formally correct pronunciation at all levels.{{Cite book |url=https://rm.coe.int/cefr-companion-volume-with-new-descriptors-2018/1680787989 |title=Common European framework of reference for languages learning, teaching, assessment: Companion volume with new descriptors |date=February 2018 |publisher=Language Policy Programme, Education Policy Division, Education Department, Council of Europe |page=136 |oclc=1090351600 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053825/https://rm.coe.int/cefr-companion-volume-with-new-descriptors-2018/1680787989 |url-status=live }}

In the health care sector, speech recognition can be implemented in front-end or back-end of the medical documentation process. Front-end speech recognition is where the provider dictates into a speech-recognition engine, the recognized words are displayed as they are spoken, and the dictator is responsible for editing and signing off on the document. Back-end or deferred speech recognition is where the provider dictates into a digital dictation system, the voice is routed through a speech-recognition machine and the recognized draft document is routed along with the original voice file to the editor, where the draft is edited and report finalized. Deferred speech recognition is widely used in the industry currently.

One of the major issues relating to the use of speech recognition in healthcare is that the American Recovery and Reinvestment Act of 2009 (ARRA) provides for substantial financial benefits to physicians who utilize an EMR according to "Meaningful Use" standards. These standards require that a substantial amount of data be maintained by the EMR (now more commonly referred to as an Electronic Health Record or EHR). The use of speech recognition is more naturally suited to the generation of narrative text, as part of a radiology/pathology interpretation, progress note or discharge summary: the ergonomic gains of using speech recognition to enter structured discrete data (e.g., numeric values or codes from a list or a controlled vocabulary) are relatively minimal for people who are sighted and who can operate a keyboard and mouse.

A more significant issue is that most EHRs have not been expressly tailored to take advantage of voice-recognition capabilities. A large part of the clinician's interaction with the EHR involves navigation through the user interface using menus, and tab/button clicks, and is heavily dependent on keyboard and mouse: voice-based navigation provides only modest ergonomic benefits. By contrast, many highly customized systems for radiology or pathology dictation implement voice "macros", where the use of certain phrases – e.g., "normal report", will automatically fill in a large number of default values and/or generate boilerplate, which will vary with the type of the exam – e.g., a chest X-ray vs. a gastrointestinal contrast series for a radiology system.

Prolonged use of speech recognition software in conjunction with word processors has shown benefits to short-term-memory restrengthening in brain AVM patients who have been treated with resection. Further research needs to be conducted to determine cognitive benefits for individuals whose AVMs have been treated using radiologic techniques.{{citation needed|date=November 2016}}

Substantial efforts have been devoted in the last decade to the test and evaluation of speech recognition in fighter aircraft. Of particular note have been the US program in speech recognition for the Advanced Fighter Technology Integration (AFTI)/F-16 aircraft (F-16 VISTA), the program in France for Mirage aircraft, and other programs in the UK dealing with a variety of aircraft platforms. In these programs, speech recognizers have been operated successfully in fighter aircraft, with applications including setting radio frequencies, commanding an autopilot system, setting steer-point coordinates and weapons release parameters, and controlling flight display.

Working with Swedish pilots flying in the JAS-39 Gripen cockpit, Englund (2004) found recognition deteriorated with increasing g-loads. The report also concluded that adaptation greatly improved the results in all cases and that the introduction of models for breathing was shown to improve recognition scores significantly. Contrary to what might have been expected, no effects of the broken English of the speakers were found. It was evident that spontaneous speech caused problems for the recognizer, as might have been expected. A restricted vocabulary, and above all, a proper syntax, could thus be expected to improve recognition accuracy substantially.{{Cite thesis |last=Englund |first=Christine |title=Speech recognition in the JAS 39 Gripen aircraft: Adaptation to speech at different G-loads |degree=Masters thesis |publisher=Stockholm Royal Institute of Technology |url=http://www.speech.kth.se/prod/publications/files/1664.pdf |year=2004 |url-status=live |archive-url=https://web.archive.org/web/20081002002102/http://www.speech.kth.se/prod/publications/files/1664.pdf |archive-date=2 October 2008 |df=dmy-all}}

The Eurofighter Typhoon, currently in service with the UK RAF, employs a speaker-dependent system, requiring each pilot to create a template. The system is not used for any safety-critical or weapon-critical tasks, such as weapon release or lowering of the undercarriage, but is used for a wide range of other cockpit functions. Voice commands are confirmed by visual and/or aural feedback. The system is seen as a major design feature in the reduction of pilot workload,{{Cite web |title=The Cockpit |url=https://www.eurofighter.com/the-aircraft#cockpit |url-status=live |archive-url=https://web.archive.org/web/20170301222529/https://www.eurofighter.com/the-aircraft#cockpit |archive-date=1 March 2017 |website=Eurofighter Typhoon |df=dmy-all}} and even allows the pilot to assign targets to his aircraft with two simple voice commands or to any of his wingmen with only five commands.{{Cite web |title=Eurofighter Typhoon – The world's most advanced fighter aircraft |url=http://www.eurofighter.com/capabilities/technology/voice-throttle-stick/direct-voice-input.html |url-status=live |archive-url=https://web.archive.org/web/20130511025203/http://www.eurofighter.com/capabilities/technology/voice-throttle-stick/direct-voice-input.html |archive-date=11 May 2013 |access-date=1 May 2018 |website=www.eurofighter.com |df=dmy-all}}

Speaker-independent systems are also being developed and are under test for the F-35 Lightning II (JSF) and the Alenia Aermacchi M-346 Master lead-in fighter trainer. These systems have produced word accuracy scores in excess of 98%.{{Cite web |last=Schutte |first=John |date=15 October 2007 |title=Researchers fine-tune F-35 pilot-aircraft speech system |url=https://www.af.mil/News/story/id/123071861/ |url-status=live |archive-url=https://web.archive.org/web/20071020030310/http://www.af.mil/news/story.asp?id=123071861 |archive-date=20 October 2007 |publisher=United States Air Force}}

The problems of achieving high recognition accuracy under stress and noise are particularly relevant in the helicopter environment as well as in the jet fighter environment. The acoustic noise problem is actually more severe in the helicopter environment, not only because of the high noise levels but also because the helicopter pilot, in general, does not wear a facemask, which would reduce acoustic noise in the microphone. Substantial test and evaluation programs have been carried out in the past decade in speech recognition systems applications in helicopters, notably by the U.S. Army Avionics Research and Development Activity (AVRADA) and by the Royal Aerospace Establishment (RAE) in the UK. Work in France has included speech recognition in the Puma helicopter. There has also been much useful work in Canada. Results have been encouraging, and voice applications have included: control of communication radios, setting of navigation systems, and control of an automated target handover system.

As in fighter applications, the overriding issue for voice in helicopters is the impact on pilot effectiveness. Encouraging results are reported for the AVRADA tests, although these represent only a feasibility demonstration in a test environment. Much remains to be done both in speech recognition and in overall speech technology in order to consistently achieve performance improvements in operational settings.

Training for air traffic controllers (ATC) represents an excellent application for speech recognition systems. Many ATC training systems currently require a person to act as a "pseudo-pilot", engaging in a voice dialog with the trainee controller, which simulates the dialog that the controller would have to conduct with pilots in a real ATC situation. Speech recognition and synthesis techniques offer the potential to eliminate the need for a person to act as a pseudo-pilot, thus reducing training and support personnel. In theory, Air controller tasks are also characterized by highly structured speech as the primary output of the controller, hence reducing the difficulty of the speech recognition task should be possible. In practice, this is rarely the case. The FAA document 7110.65 details the phrases that should be used by air traffic controllers. While this document gives less than 150 examples of such phrases, the number of phrases supported by one of the simulation vendors speech recognition systems is in excess of 500,000.

The USAF, USMC, US Army, US Navy, and FAA as well as a number of international ATC training organizations such as the Royal Australian Air Force and Civil Aviation Authorities in Italy, Brazil, and Canada are currently using ATC simulators with speech recognition from a number of different vendors.{{citation needed|date=December 2012}}

ASR is now commonplace in the field of telephony and is becoming more widespread in the field of computer gaming and simulation. In telephony systems, ASR is now being predominantly used in contact centers by integrating it with IVR systems. Despite the high level of integration with word processing in general personal computing, in the field of document production, ASR has not seen the expected increases in use.

The improvement of mobile processor speeds has made speech recognition practical in smartphones. Speech is used mostly as a part of a user interface, for creating predefined or custom speech commands.

People with disabilities can benefit from speech recognition programs. For individuals that are Deaf or Hard of Hearing, speech recognition software is used to automatically generate a closed-captioning of conversations such as discussions in conference rooms, classroom lectures, and/or religious services.{{Cite web |date=18 March 2010 |title=Overcoming Communication Barriers in the Classroom |url=http://www.massmatch.org/aboutus/listserv/2010/2010-03-31.html |url-status=usurped |archive-url=https://web.archive.org/web/20130725024622/http://www.massmatch.org/aboutus/listserv/2010/2010-03-31.html |archive-date=25 July 2013 |access-date=15 June 2013 |publisher=MassMATCH |df=dmy-all}}

Students who are blind (see Blindness and education) or have very low vision can benefit from using the technology to convey words and then hear the computer recite them, as well as use a computer by commanding with their voice, instead of having to look at the screen and keyboard.{{Cite web |year=2010 |title=Speech Recognition for Learning |url=http://www.brainline.org/content/2010/12/speech-recognition-for-learning_pageall.html |url-status=live |archive-url=https://web.archive.org/web/20140413100513/http://www.brainline.org/content/2010/12/speech-recognition-for-learning_pageall.html |archive-date=13 April 2014 |access-date=26 March 2014 |publisher=National Center for Technology Innovation |df=dmy-all}}

Students who are physically disabled have a Repetitive strain injury/other injuries to the upper extremities can be relieved from having to worry about handwriting, typing, or working with scribe on school assignments by using speech-to-text programs. They can also utilize speech recognition technology to enjoy searching the Internet or using a computer at home without having to physically operate a mouse and keyboard.

Speech recognition can allow students with learning disabilities to become better writers. By saying the words aloud, they can increase the fluidity of their writing, and be alleviated of concerns regarding spelling, punctuation, and other mechanics of writing.{{Cite web |last1=Follensbee |first1=Bob |last2=McCloskey-Dale |first2=Susan |year=2000 |title=Speech recognition in schools: An update from the field |url=http://www.csun.edu/~hfdss006/conf/2000/proceedings/0219Follansbee.htm |url-status=live |archive-url=https://web.archive.org/web/20060821213145/http://www.csun.edu/~hfdss006/conf/2000/proceedings/0219Follansbee.htm |archive-date=21 August 2006 |access-date=26 March 2014 |website=Technology And Persons With Disabilities Conference 2000 |df=dmy-all}} Also, see Learning disability.

The use of voice recognition software, in conjunction with a digital audio recorder and a personal computer running word-processing software has proven to be positive for restoring damaged short-term memory capacity, in stroke and craniotomy individuals.

Speech recognition is also very useful for people who have difficulty using their hands, ranging from mild repetitive stress injuries to involve disabilities that preclude using conventional computer input devices. In fact, people who used the keyboard a lot and developed RSI became an urgent early market for speech recognition.{{Cite web |title=Speech recognition for disabled people |url=http://www.businessweek.com/1998/08/b3566022.htm |url-status=dead |archive-url=https://web.archive.org/web/20080404013302/http://www.businessweek.com/1998/08/b3566022.htm |archive-date=4 April 2008 |df=dmy-all}}Friends International Support Group Speech recognition is used in deaf telephony, such as voicemail to text, relay services, and captioned telephone. Individuals with learning disabilities who have problems with thought-to-paper communication (essentially they think of an idea but it is processed incorrectly causing it to end up differently on paper) can possibly benefit from the software but the technology is not bug proof.{{Cite journal |last=Garrett |first=Jennifer Tumlin |display-authors=etal |year=2011 |title=Using Speech Recognition Software to Increase Writing Fluency for Individuals with Physical Disabilities |url=https://scholarworks.gsu.edu/epse_diss/46 |journal=Journal of Special Education Technology |volume=26 |issue=1 |pages=25–41 |doi=10.1177/016264341102600104 |s2cid=142730664 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053848/https://scholarworks.gsu.edu/epse_diss/46/ |url-status=live }} Also the whole idea of speak to text can be hard for intellectually disabled person's due to the fact that it is rare that anyone tries to learn the technology to teach the person with the disability.Forgrave, Karen E. "Assistive Technology: Empowering Students with Disabilities." Clearing House 75.3 (2002): 122–6. Web.

This type of technology can help those with dyslexia but other disabilities are still in question. The effectiveness of the product is the problem that is hindering it from being effective. Although a kid may be able to say a word depending on how clear they say it the technology may think they are saying another word and input the wrong one. Giving them more work to fix, causing them to have to take more time with fixing the wrong word.{{Cite journal |last1=Tang |first1=K. W. |last2=Kamoua |first2=Ridha |last3=Sutan |first3=Victor |year=2004 |title=Speech Recognition Technology for Disabilities Education |journal=Journal of Educational Technology Systems |volume=33 |issue=2 |pages=173–84 |citeseerx=10.1.1.631.3736 |doi=10.2190/K6K8-78K2-59Y7-R9R2 |s2cid=143159997}}

• Aerospace (e.g. space exploration, spacecraft, etc.) NASA's Mars Polar Lander used speech recognition technology from Sensory, Inc. in the Mars Microphone on the Lander{{Cite web |title=Projects: Planetary Microphones |url=http://www.planetary.org/programs/projects/planetary_microphones/mars_microphone.html |url-status=dead |archive-url=https://web.archive.org/web/20120127161038/http://www.planetary.org/programs/projects/planetary_microphones/mars_microphone.html |archive-date=27 January 2012 |publisher=The Planetary Society}}
• Automatic subtitling with speech recognition
• Automatic emotion recognition{{Cite book |last1=Caridakis |first1=George |title=Artificial Intelligence and Innovations 2007: From Theory to Applications |last2=Castellano |first2=Ginevra |last3=Kessous |first3=Loic |last4=Raouzaiou |first4=Amaryllis |last5=Malatesta |first5=Lori |last6=Asteriadis |first6=Stelios |last7=Karpouzis |first7=Kostas |date=19 September 2007 |publisher=Springer US |isbn=978-0-387-74160-4 |series=IFIP the International Federation for Information Processing |volume=247 |pages=375–388 |language=en |chapter=Multimodal emotion recognition from expressive faces, body gestures and speech |doi=10.1007/978-0-387-74161-1_41}}
• Automatic shot listing in audiovisual production
• Automatic translation
• eDiscovery (Legal discovery)
• Hands-free computing: Speech recognition computer user interface
• Home automation
• Interactive voice response
• Mobile telephony, including mobile email
• Multimodal interaction
• Real Time Captioning{{Cite web |title=What is real-time captioning? {{!}} DO-IT |url=https://www.washington.edu/doit/what-real-time-captioning |access-date=2021-04-11 |website=www.washington.edu |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054510/https://www.washington.edu/doit/what-real-time-captioning |url-status=live }}
• Robotics
• Security, including usage with other biometric scanners for multi-factor authentication{{Cite book |last1=Zheng |first1=Thomas Fang |url=http://link.springer.com/10.1007/978-981-10-3238-7 |title=Robustness-Related Issues in Speaker Recognition |last2=Li |first2=Lantian |date=2017 |publisher=Springer Singapore |isbn=978-981-10-3237-0 |series=SpringerBriefs in Electrical and Computer Engineering |location=Singapore |doi=10.1007/978-981-10-3238-7 |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053948/https://link.springer.com/book/10.1007/978-981-10-3238-7 |url-status=live }}
• Speech to text (transcription of speech into text, real time video captioning, Court reporting )
• Telematics (e.g. vehicle Navigation Systems)
• Transcription (digital speech-to-text)
• Video games, with Tom Clancy's EndWar and Lifeline as working examples
• Virtual assistant (e.g. Apple's Siri)

The performance of speech recognition systems is usually evaluated in terms of accuracy and speed.Ciaramella, Alberto. "A prototype performance evaluation report." Sundial workpackage 8000 (1993).{{Cite book |last1=Gerbino |first1=E. |title=IEEE International Conference on Acoustics Speech and Signal Processing |last2=Baggia |first2=P. |last3=Ciaramella |first3=A. |last4=Rullent |first4=C. |year=1993 |isbn=0-7803-0946-4 |pages=135–138 vol.2 |chapter=Test and evaluation of a spoken dialogue system |doi=10.1109/ICASSP.1993.319250 |s2cid=57374050}} Accuracy is usually rated with word error rate (WER), whereas speed is measured with the real time factor. Other measures of accuracy include Single Word Error Rate (SWER) and Command Success Rate (CSR).

Speech recognition by machine is a very complex problem, however. Vocalizations vary in terms of accent, pronunciation, articulation, roughness, nasality, pitch, volume, and speed. Speech is distorted by a background noise and echoes, electrical characteristics. Accuracy of speech recognition may vary with the following:National Institute of Standards and Technology. "[http://www.itl.nist.gov/iad/mig/publications/ASRhistory/ The History of Automatic Speech Recognition Evaluation at NIST] {{webarchive|url=https://web.archive.org/web/20131008210040/http://www.itl.nist.gov/iad/mig/publications/ASRhistory/ |date=8 October 2013 }}".{{Citation needed|date=May 2013}}

• Vocabulary size and confusability
• Speaker dependence versus independence
• Isolated, discontinuous or continuous speech
• Task and language constraints
• Read versus spontaneous speech
• Adverse conditions

As mentioned earlier in this article, the accuracy of speech recognition may vary depending on the following factors:

• Error rates increase as the vocabulary size grows:

::e.g. the 10 digits "zero" to "nine" can be recognized essentially perfectly, but vocabulary sizes of 200, 5000 or 100000 may have error rates of 3%, 7%, or 45% respectively.

• Vocabulary is hard to recognize if it contains confusing letters:

::e.g. the 26 letters of the English alphabet are difficult to discriminate because they are confusing words (most notoriously, the E-set: "B, C, D, E, G, P, T, V, Z — when "Z" is pronounced "zee" rather than "zed" depending on the English region); an 8% error rate is considered good for this vocabulary.{{Cite web |title=Letter Names Can Cause Confusion and Other Things to Know About Letter–Sound Relationships |url=https://www.naeyc.org/resources/pubs/yc/mar2015/letter-sound-relationships |access-date=2023-10-27 |website=NAEYC |language=en |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054452/https://www.naeyc.org/resources/pubs/yc/mar2015/letter-sound-relationships |url-status=live }}

• Speaker dependence vs. independence:

:: A speaker-dependent system is intended for use by a single speaker.

:: A speaker-independent system is intended for use by any speaker (more difficult).

• Isolated, Discontinuous or continuous speech

:: With isolated speech, single words are used, therefore it becomes easier to recognize the speech.

With discontinuous speech full sentences separated by silence are used, therefore it becomes easier to recognize the speech as well as with isolated speech.

With continuous speech naturally spoken sentences are used, therefore it becomes harder to recognize the speech, different from both isolated and discontinuous speech.

• Task and language constraints
• e.g. Querying application may dismiss the hypothesis "The apple is red."
• e.g. Constraints may be semantic; rejecting "The apple is angry."
• e.g. Syntactic; rejecting "Red is apple the."

Constraints are often represented by grammar.

• Read vs. Spontaneous Speech – When a person reads it's usually in a context that has been previously prepared, but when a person uses spontaneous speech, it is difficult to recognize the speech because of the disfluencies (like "uh" and "um", false starts, incomplete sentences, stuttering, coughing, and laughter) and limited vocabulary.
• Adverse conditions – Environmental noise (e.g. Noise in a car or a factory). Acoustical distortions (e.g. echoes, room acoustics)

Speech recognition is a multi-leveled pattern recognition task.

• Acoustical signals are structured into a hierarchy of units, e.g. Phonemes, Words, Phrases, and Sentences;
• Each level provides additional constraints;

e.g. Known word pronunciations or legal word sequences, which can compensate for errors or uncertainties at a lower level;

• This hierarchy of constraints is exploited. By combining decisions probabilistically at all lower levels, and making more deterministic decisions only at the highest level, speech recognition by a machine is a process broken into several phases. Computationally, it is a problem in which a sound pattern has to be recognized or classified into a category that represents a meaning to a human. Every acoustic signal can be broken into smaller more basic sub-signals. As the more complex sound signal is broken into the smaller sub-sounds, different levels are created, where at the top level we have complex sounds, which are made of simpler sounds on the lower level, and going to lower levels, even more, we create more basic and shorter and simpler sounds. At the lowest level, where the sounds are the most fundamental, a machine would check for simple and more probabilistic rules of what sound should represent. Once these sounds are put together into more complex sounds on upper level, a new set of more deterministic rules should predict what the new complex sound should represent. The most upper level of a deterministic rule should figure out the meaning of complex expressions. In order to expand our knowledge about speech recognition, we need to take into consideration neural networks. There are four steps of neural network approaches:
• Digitize the speech that we want to recognize

For telephone speech the sampling rate is 8000 samples per second;

• Compute features of spectral-domain of the speech (with Fourier transform);

computed every 10 ms, with one 10 ms section called a frame;

Analysis of four-step neural network approaches can be explained by further information. Sound is produced by air (or some other medium) vibration, which we register by ears, but machines by receivers. Basic sound creates a wave which has two descriptions: amplitude (how strong is it), and frequency (how often it vibrates per second).

Accuracy can be computed with the help of word error rate (WER). Word error rate can be calculated by aligning the recognized word and referenced word using dynamic string alignment. The problem may occur while computing the word error rate due to the difference between the sequence lengths of the recognized word and referenced word.

The formula to compute the word error rate (WER) is:

$WER\; =\; \{(s+d+i)\; \backslash over\; n\}$

where s is the number of substitutions, d is the number of deletions, i is the number of insertions, and n is the number of word references.

While computing, the word recognition rate (WRR) is used. The formula is:

: $WRR\; =\; 1\; -\; WER\; =\; \{(n-s-d-i)\; \backslash over\; n\}\; =\; \{h-i\; \backslash over\; n\}$

where h is the number of correctly recognized words:

: $h\; =\; n\; -(s+d).$

Speech recognition can become a means of attack, theft, or accidental operation. For example, activation words like "Alexa" spoken in an audio or video broadcast can cause devices in homes and offices to start listening for input inappropriately, or possibly take an unwanted action.{{Cite news |date=6 March 2016 |title=Listen Up: Your AI Assistant Goes Crazy For NPR Too |url=https://www.npr.org/2016/03/06/469383361/listen-up-your-ai-assistant-goes-crazy-for-npr-too |url-status=live |archive-url=https://web.archive.org/web/20170723210358/http://www.npr.org/2016/03/06/469383361/listen-up-your-ai-assistant-goes-crazy-for-npr-too |archive-date=23 July 2017 |work=NPR |df=dmy-all}} Voice-controlled devices are also accessible to visitors to the building, or even those outside the building if they can be heard inside. Attackers may be able to gain access to personal information, like calendar, address book contents, private messages, and documents. They may also be able to impersonate the user to send messages or make online purchases.

Two attacks have been demonstrated that use artificial sounds. One transmits ultrasound and attempt to send commands without nearby people noticing.{{Cite news |last=Claburn |first=Thomas |date=25 August 2017 |title=Is it possible to control Amazon Alexa, Google Now using inaudible commands? Absolutely |url=https://www.theregister.co.uk/2017/08/25/amazon_alexa_answers_inaudible_commands/?mt=1504024969000 |url-status=live |archive-url=https://web.archive.org/web/20170902051123/https://www.theregister.co.uk/2017/08/25/amazon_alexa_answers_inaudible_commands/?mt=1504024969000 |archive-date=2 September 2017 |work=The Register |df=dmy-all}} The other adds small, inaudible distortions to other speech or music that are specially crafted to confuse the specific speech recognition system into recognizing music as speech, or to make what sounds like one command to a human sound like a different command to the system.{{Cite web |date=31 January 2018 |title=Attack Targets Automatic Speech Recognition Systems |url=https://www.vice.com/en/article/attack-targets-automatic-speech-recognition-systems/ |url-status=live |archive-url=https://web.archive.org/web/20180303050744/https://motherboard.vice.com/en_us/article/d34nnz/attack-targets-automatic-speech-recognition-systems |archive-date=3 March 2018 |access-date=1 May 2018 |website=vice.com |df=dmy-all}}

Popular speech recognition conferences held each year or two include SpeechTEK and SpeechTEK Europe, ICASSP, Interspeech/Eurospeech, and the IEEE ASRU. Conferences in the field of natural language processing, such as ACL, NAACL, EMNLP, and HLT, are beginning to include papers on speech processing. Important journals include the IEEE Transactions on Speech and Audio Processing (later renamed IEEE Transactions on Audio, Speech and Language Processing and since Sept 2014 renamed IEEE/ACM Transactions on Audio, Speech and Language Processing—after merging with an ACM publication), Computer Speech and Language, and Speech Communication.

Books like "Fundamentals of Speech Recognition" by Lawrence Rabiner can be useful to acquire basic knowledge but may not be fully up to date (1993). Another good source can be "Statistical Methods for Speech Recognition" by Frederick Jelinek and "Spoken Language Processing (2001)" by Xuedong Huang etc., "Computer Speech", by Manfred R. Schroeder, second edition published in 2004, and "Speech Processing: A Dynamic and Optimization-Oriented Approach" published in 2003 by Li Deng and Doug O'Shaughnessey. The updated textbook Speech and Language Processing (2008) by Jurafsky and Martin presents the basics and the state of the art for ASR. Speaker recognition also uses the same features, most of the same front-end processing, and classification techniques as is done in speech recognition. A comprehensive textbook, "Fundamentals of Speaker Recognition" is an in depth source for up to date details on the theory and practice.{{Cite book |last=Beigi |first=Homayoon |url=http://www.fundamentalsofspeakerrecognition.org |title=Fundamentals of Speaker Recognition |publisher=Springer |year=2011 |isbn=978-0-387-77591-3 |location=New York |archive-url=https://web.archive.org/web/20180131140911/http://www.fundamentalsofspeakerrecognition.org/ |archive-date=31 January 2018 |url-status=live |df=dmy-all}} A good insight into the techniques used in the best modern systems can be gained by paying attention to government sponsored evaluations such as those organised by DARPA (the largest speech recognition-related project ongoing as of 2007 is the GALE project, which involves both speech recognition and translation components).

A good and accessible introduction to speech recognition technology and its history is provided by the general audience book "The Voice in the Machine. Building Computers That Understand Speech" by Roberto Pieraccini (2012).

The most recent book on speech recognition is Automatic Speech Recognition: A Deep Learning Approach (Publisher: Springer) written by Microsoft researchers D. Yu and L. Deng and published near the end of 2014, with highly mathematically oriented technical detail on how deep learning methods are derived and implemented in modern speech recognition systems based on DNNs and related deep learning methods.{{Cite journal |last1=Yu |first1=D. |last2=Deng |first2=L. |date=2014 |title=Automatic Speech Recognition: A Deep Learning Approach (Publisher: Springer)}} A related book, published earlier in 2014, "Deep Learning: Methods and Applications" by L. Deng and D. Yu provides a less technical but more methodology-focused overview of DNN-based speech recognition during 2009–2014, placed within the more general context of deep learning applications including not only speech recognition but also image recognition, natural language processing, information retrieval, multimodal processing, and multitask learning.{{Cite journal |last1=Deng |first1=Li |last2=Yu |first2=Dong |year=2014 |title=Deep Learning: Methods and Applications |url=http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf |url-status=live |journal=Foundations and Trends in Signal Processing |volume=7 |issue=3–4 |pages=197–387 |citeseerx=10.1.1.691.3679 |doi=10.1561/2000000039 |archive-url=https://web.archive.org/web/20141022161017/http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf |archive-date=22 October 2014 |df=dmy-all}}

In terms of freely available resources, Carnegie Mellon University's Sphinx toolkit is one place to start to both learn about speech recognition and to start experimenting. Another resource (free but copyrighted) is the HTK book (and the accompanying HTK toolkit). For more recent and state-of-the-art techniques, Kaldi toolkit can be used.Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., ... & Vesely, K. (2011). The Kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding (No. CONF). IEEE Signal Processing Society. In 2017 Mozilla launched the open source project called Common Voice{{Cite web |title=Common Voice by Mozilla |url=https://voice.mozilla.org/ |url-status=dead |archive-url=https://web.archive.org/web/20200227020208/https://voice.mozilla.org/ |archive-date=27 February 2020 |access-date=9 November 2019 |website=voice.mozilla.org}} to gather big database of voices that would help build free speech recognition project DeepSpeech (available free at GitHub),{{Cite web |date=9 November 2019 |title=A TensorFlow implementation of Baidu's DeepSpeech architecture: mozilla/DeepSpeech |url=https://github.com/mozilla/DeepSpeech |via=GitHub |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053949/https://github.com/mozilla/DeepSpeech |url-status=live }} using Google's open source platform TensorFlow.{{Cite web |date=9 November 2019 |title=GitHub - tensorflow/docs: TensorFlow documentation |url=https://github.com/tensorflow/docs |via=GitHub |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053830/https://github.com/tensorflow/docs |url-status=live }} When Mozilla redirected funding away from the project in 2020, it was forked by its original developers as Coqui STT{{Cite web |title=Coqui, a startup providing open speech tech for everyone |url=https://github.com/coqui-ai |access-date=2022-03-07 |website=GitHub |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054614/https://github.com/coqui-ai |url-status=live }} using the same open-source license.{{Cite magazine |last=Coffey |first=Donavyn |date=2021-04-28 |title=Māori are trying to save their language from Big Tech |url=https://www.wired.co.uk/article/maori-language-tech |access-date=2021-10-16 |magazine=Wired UK |language=en-GB |issn=1357-0978 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909053950/https://www.wired.com/story/maori-language-tech/ |url-status=live }}{{Cite web |date=2021-07-07 |title=Why you should move from DeepSpeech to coqui.ai |url=https://discourse.mozilla.org/t/why-you-should-move-from-deepspeech-to-coqui-ai/82798 |access-date=2021-10-16 |website=Mozilla Discourse |language=en-US}}

Google Gboard supports speech recognition on all Android applications. It can be activated through the microphone icon.{{Cite web |title=Type with your voice |url=https://support.google.com/gboard/answer/2781851?hl=en&co=GENIE.Platform%3DAndroid |access-date=9 September 2024 |archive-date=9 September 2024 |archive-url=https://web.archive.org/web/20240909054332/https://support.google.com/gboard/answer/2781851?hl=en&co=GENIE.Platform%3DAndroid |url-status=live }} Speech recognition can be activated in Microsoft Windows operating systems by pressing Windows logo key + Ctrl + S.{{cite web|url=https://support.microsoft.com/en-us/windows/use-voice-recognition-in-windows-83ff75bd-63eb-0b6c-18d4-6fae94050571|title=Use voice recognition in Windows|archive-url=https://web.archive.org/web/20250409223456/https://support.microsoft.com/en-us/windows/use-voice-recognition-in-windows-83ff75bd-63eb-0b6c-18d4-6fae94050571|archive-date=April 9, 2025|url-status=live}}

The commercial cloud based speech recognition APIs are broadly available.

For more software resources, see List of speech recognition software.

{{div col|colwidth=22em}}

• AI effect
• ALPAC
• Application Language Tags for speech recognition
• Articulatory speech recognition
• Audio mining
• Audio-visual speech recognition
• Automatic Language Translator
• Automotive head unit
• Braina
• Cache language model
• Dragon NaturallySpeaking
• Fluency Voice Technology
• Google Voice Search
• IBM ViaVoice
• Keyword spotting
• Kinect
• Mondegreen
• Multimedia information retrieval
• Origin of speech
• Phonetic search technology
• Speaker diarisation
• Speaker recognition
• Speech analytics
• Speech interface guideline
• Speech recognition software for Linux
• Speech synthesis
• Speech verification
• Subtitle (captioning)
• VoiceXML
• VoxForge
• Windows Speech Recognition

; Lists

• List of speech recognition software
• List of emerging technologies
• Outline of artificial intelligence
• Timeline of speech and voice recognition

{{div col end}}

{{Reflist}}

• {{Cite book |title=Survey of the state of the art in human language technology |publisher=Cambridge University Press |year=1997 |isbn=978-0-521-59277-2 |editor-last=Cole |editor-first=Ronald |series=Cambridge Studies in Natural Language Processing |volume=XII–XIII |editor-last2=Mariani |editor-first2=Joseph |editor-link2=Joseph mariani |editor-last3=Uszkoreit |editor-first3=Hans |editor-last4=Varile |editor-first4=Giovanni Battista |editor-last5=Zaenen |editor-first5=Annie |editor-last6=Zampolli |editor-last7=Zue |editor-first7=Victor}}
• {{Cite book |last1=Junqua |first1=J.-C. |title=Robustness in Automatic Speech Recognition: Fundamentals and Applications |last2=Haton |first2=J.-P. |publisher=Kluwer Academic Publishers |year=1995 |isbn=978-0-7923-9646-8}}
• {{Cite book |last1=Karat |first1=Clare-Marie |title=The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (Human Factors and Ergonomics) |last2=Vergo |first2=John |last3=Nahamoo |first3=David |publisher=Lawrence Erlbaum Associates Inc |year=2007 |isbn=978-0-8058-5870-9 |editor-last=Sears |editor-first=Andrew |editor-link=Andrew Sears |chapter=Conversational Interface Technologies |editor-last2=Jacko |editor-first2=Julie A.}}
• {{Cite book |last=Pieraccini |first=Roberto |title=The Voice in the Machine. Building Computers That Understand Speech. |publisher=The MIT Press |year=2012 |isbn=978-0262016858}}
• {{Cite book |title=Advanced algorithms and architectures for speech understanding |publisher=Springer Science & Business Media |year=2013 |isbn=978-3-642-84341-9 |editor-last=Pirani |editor-first=Giancarlo}}
• {{cite conference |last1=Signer |first1=Beat |last2=Hoste |first2=Lode |url=https://www.academia.edu/4685517 |title=SpeeG2: A Speech- and Gesture-based Interface for Efficient Controller-free Text Entry |book-title=Proceedings of ICMI 2013 |conference=15th International Conference on Multimodal Interaction |location=Sydney, Australia |date=December 2013}}
• {{Cite book |last1=Woelfel |first1=Matthias |title=Distant Speech Recognition |last2=McDonough |first2=John |date=2009-05-26 |publisher=Wiley |isbn=978-0470517048}}

{{Natural Language Processing}}

{{Artificial intelligence navbox}}

{{DEFAULTSORT:Speech Recognition}}

Category:Automatic identification and data capture

Category:Computational linguistics

Category:User interface techniques

Category:History of human–computer interaction

Category:Computer accessibility

Category:Machine learning task