Time delay neural network

The TDNN was introduced in the late 1980s and applied to a task of phoneme classification for automatic speech recognition in speech signals where the automatic determination of precise segments or feature boundaries was difficult or impossible. Because the TDNN recognizes phonemes and their underlying acoustic/phonetic features, independent of position in time, it improved performance over static classification.Alexander Waibel, [https://isl.iar.kit.edu/downloads/Pheome_Recognition_Using_Time-Delay_Neural_Networks_SP87-100_6.pdf Phoneme Recognition Using Time-Delay Neural Networks], Procedures of the Institute of Electrical, Information and Communication Engineers (IEICE), December, 1987, Tokyo, Japan. It was also applied to two-dimensional signals (time-frequency patterns in speech,{{cite journal |author=John B. Hampshire |author2=Alex Waibel |url=https://www.researchgate.net/publication/391319411 |title=Connectionist Architectures for Multi-Speaker Phoneme Recognition |journal=Advances in Neural Information Processing Systems |volume=2 |pages=203–210}} and coordinate space pattern in OCR{{cite journal |url=https://www.researchgate.net/publication/220163530 |doi=10.1007/PL00013559 |title=Online handwriting recognition: The NPen++ recognizer |date=2001 |last1=Jaeger |first1=S. |last2=Manke |first2=S. |last3=Reichert |first3=J. |last4=Waibel |first4=A. |journal=International Journal on Document Analysis and Recognition |volume=3 |issue=3 |pages=169–180 }}).

Kunihiko Fukushima published the neocognitron in 1980.{{cite journal |last=Fukushima |first=Kunihiko |year=1980 |title=Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position |url=https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf |url-status=live |journal=Biological Cybernetics |volume=36 |issue=4 |pages=193–202 |doi=10.1007/BF00344251 |pmid=7370364 |s2cid=206775608 |archive-url=https://web.archive.org/web/20140603013137/http://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf |archive-date=3 June 2014 |access-date=16 November 2013}} Max pooling appears in a 1982 publication on the neocognitron{{Cite journal |last1=Fukushima |first1=Kunihiko |last2=Miyake |first2=Sei |date=1982-01-01 |title=Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position |url=https://www.sciencedirect.com/science/article/abs/pii/0031320382900243 |journal=Pattern Recognition |volume=15 |issue=6 |pages=455–469 |doi=10.1016/0031-3203(82)90024-3 |bibcode=1982PatRe..15..455F |issn=0031-3203|url-access=subscription }} and was in the 1989 publication in LeNet-5.{{Cite journal |last1=LeCun |first1=Yann |last2=Boser |first2=Bernhard |last3=Denker |first3=John |last4=Henderson |first4=Donnie |last5=Howard |first5=R. |last6=Hubbard |first6=Wayne |last7=Jackel |first7=Lawrence |date=1989 |title=Handwritten Digit Recognition with a Back-Propagation Network |url=https://proceedings.neurips.cc/paper/1989/hash/53c3bce66e43be4f209556518c2fcb54-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Morgan-Kaufmann |volume=2}}

In 1990, Yamaguchi et al. used max pooling in TDNNs in order to realize a speaker independent isolated word recognition system.{{cite conference |title=A Neural Network for Speaker-Independent Isolated Word Recognition |last1=Yamaguchi |first1=Kouichi |last2=Sakamoto |first2=Kenji |last3=Akabane |first3=Toshio |last4=Fujimoto |first4=Yoshiji |date=November 1990 |location=Kobe, Japan |conference=First International Conference on Spoken Language Processing (ICSLP 90) |url=https://www.isca-speech.org/archive/icslp_1990/i90_1077.html |access-date=2019-09-04 |archive-date=2021-03-07 |archive-url=https://web.archive.org/web/20210307233750/https://www.isca-speech.org/archive/icslp_1990/i90_1077.html |url-status=dead }}

In modern language, the design of TDNN is a 1D convolutional neural network, where the direction of convolution is across the dimension of time. In the original design, there are exactly 3 layers.

The input to the network is a continuous speech signal, preprocessed into a 2D array (a mel scale spectrogram). One dimension is time at 10 ms per frame, and the other dimension is frequency. The time dimension can be arbitrarily long, but the frequency dimension was only 16-long. In the original experiment, they only considered very short speech signals pronouncing single words like "baa", "daa", "gaa". Because of this, the speech signals could be very short, indeed, only 15 frames long (150 ms in time).

In detail, they processed a voice signal as follows:

• Input speech is sampled at 12 kHz, Hamming-windowed.
• Its FFT is computed every 5 ms.
• The mel scale coefficients are computed from the power spectrum by taking log energies in each mel scale energy band.
• Adjacent coefficients in time are soothed over, resulting in one frame every 10 ms.
• For each signal, a human manually detect the onset of the vowel, and the entire speech signal is cut off except 7 frames before and 7 frames after, leaving just 15 frames in total, centered at the onset of the vowel.
• The coefficients are normalized by subtracting the mean, then scaling, so that the signals fall between -1 and +1.

The first layer of the TDNN is a 1D convolutional layer. The layer contains 8 kernels of shape $3\; \backslash times\; 16$. It outputs a tensor of shape $8\; \backslash times\; 13$.

The second layer of the TDNN is a 1D convolutional layer. The layer contains 3 kernels of shape $5\; \backslash times\; 8$. It outputs a tensor of shape $3\; \backslash times\; 9$.

The third layer of the TDNN is not a convolutional layer. Instead, it is simply a fixed layer with 3 neurons. Let the output from the second layer be $x\_\{i,j\}$ where $i\; \backslash in\; 1:3$ and $j\; \backslash in\; 1:9$. The $i$-th neuron in the third layer computes $\backslash sigma(\backslash sum\_\{j\backslash in\; 1:9\}\; x\_\{i,j\})$, where $\backslash sigma$ is the sigmoid function. Essentially, it can be thought of as a convolution layer with 3 kernels of shape $1\; \backslash times\; 9$.

It was trained on ~800 samples for 20000--50000 backpropagation steps. Each steps was computed in a batch over the entire training dataset, i.e. not stochastic. It required the use of an Alliant supercomputer with 4 processors.

In the case of a speech signal, inputs are spectral coefficients over time.

In order to learn critical acoustic-phonetic features (for example formant transitions, bursts, frication, etc.) without first requiring precise localization, the TDNN is trained time-shift-invariantly. Time-shift invariance is achieved through weight sharing across time during training: Time shifted copies of the TDNN are made over the input range (from left to right in Fig.1). Backpropagation is then performed from an overall classification target vector (see TDNN diagram, three phoneme class targets (/b/, /d/, /g/) are shown in the output layer), resulting in gradients that will generally vary for each of the time-shifted network copies. Since such time-shifted networks are only copies, however, the position dependence is removed by weight sharing. In this example, this is done by averaging the gradients from each time-shifted copy before performing the weight update. In speech, time-shift invariant training was shown to learn weight matrices that are independent of precise positioning of the input. The weight matrices could also be shown to detect important acoustic-phonetic features that are known to be important for human speech perception, such as formant transitions, bursts, etc. TDNNs could also be combined or grown by way of pre-training.Alexander Waibel, Hidefumi Sawai, Kiyohiro Shikano, [https://ieeexplore.ieee.org/abstract/document/45535/ Modularity and Scaling in Large Phonemic Neural Networks], IEEE Transactions on Acoustics, Speech, and Signal Processing, December, December 1989.

The precise architecture of TDNNs (time-delays, number of layers) is mostly determined by the designer depending on the classification problem and the most useful context sizes. The delays or context windows are chosen specific to each application. Work has also been done to create adaptable time-delay TDNNs{{cite journal |doi=10.1109/72.809100|s2cid=16813677 |title=An adaptable time-delay neural-network algorithm for image sequence analysis |date=1999 |last1=Wöhler |first1=C. |last2=Anlauf |first2=J.K. |journal=IEEE Transactions on Neural Networks |volume=10 |issue=6 |pages=1531–1536 |pmid=18252656 }} where this manual tuning is eliminated.

TDNN-based phoneme recognizers compared favourably in early comparisons with HMM-based phone models. Modern deep TDNN architectures include many more hidden layers and sub-sample or pool connections over broader contexts at higher layers. They achieve up to 50% word error reduction over GMM-based acoustic models.{{cite book |doi=10.21437/Interspeech.2015-647 |doi-access=free |s2cid=8536162 |chapter=A time delay neural network architecture for efficient modeling of long temporal contexts |title=Interspeech 2015 |date=2015 |last1=Peddinti |first1=Vijayaditya |last2=Povey |first2=Daniel |last3=Khudanpur |first3=Sanjeev |pages=3214–3218 }}David Snyder, Daniel Garcia-Romero, Daniel Povey, [http://danielpovey.com/files/2015_asru_tdnn_ubm.pdf A Time-Delay Deep Neural Network-Based Universal Background Models for Speaker Recognition], Proceedings of ASRU 2015. While the different layers of TDNNs are intended to learn features of increasing context width, they do model local contexts. When longer-distance relationships and pattern sequences have to be processed, learning states and state-sequences is important and TDNNs can be combined with other modelling techniques.{{Cite journal |last1=Haffner |first1=Patrick |last2=Waibel |first2=Alex |date=1991 |title=Multi-State Time Delay Networks for Continuous Speech Recognition |url=https://proceedings.neurips.cc/paper_files/paper/1991/hash/069d3bb002acd8d7dd095917f9efe4cb-Abstract.html |website=proceedings.neurips.cc |volume=4 |publisher=NIPS |pages=135–142}}

TDNNs used to solve problems in speech recognition that were introduced in 1989 and initially focused on shift-invariant phoneme recognition. Speech lends itself nicely to TDNNs as spoken sounds are rarely of uniform length and precise segmentation is difficult or impossible. By scanning a sound over past and future, the TDNN is able to construct a model for the key elements of that sound in a time-shift invariant manner. This is particularly useful as sounds are smeared out through reverberation. Large phonetic TDNNs can be constructed modularly through pre-training and combining smaller networks.

Large vocabulary speech recognition requires recognizing sequences of phonemes that make up words subject to the constraints of a large pronunciation vocabulary. Integration of TDNNs into large vocabulary speech recognizers is possible by introducing state transitions and search between phonemes that make up a word. The resulting Multi-State Time-Delay Neural Network (MS-TDNN) can be trained discriminative from the word level, thereby optimizing the entire arrangement toward word recognition instead of phoneme classification.{{cite book | doi=10.1109/ICASSP.1993.319179 | chapter=Improving connected letter recognition by lipreading | title=IEEE International Conference on Acoustics Speech and Signal Processing | date=1993 | last1=Bregler | first1=C. | last2=Hild | first2=H. | last3=Manke | first3=S. | last4=Waibel | first4=A. | pages=557–560 |volume=1 | isbn=0-7803-0946-4 }}

Two-dimensional variants of the TDNNs were proposed for speaker independence. Here, shift-invariance is applied to the time as well as to the frequency axis in order to learn hidden features that are independent of precise location in time and in frequency (the latter being due to speaker variability).

One of the persistent problems in speech recognition is recognizing speech when it is corrupted by echo and reverberation (as is the case in large rooms and distant microphones). Reverberation can be viewed as corrupting speech with delayed versions of itself. In general, it is difficult, however, to de-reverberate a signal as the impulse response function (and thus the convolutional noise experienced by the signal) is not known for any arbitrary space. The TDNN was shown to be effective to recognize speech robustly despite different levels of reverberation.

TDNNs were also successfully used in early demonstrations of audio-visual speech, where the sounds of speech are complemented by visually reading lip movement. Here, TDNN-based recognizers used visual and acoustic features jointly to achieve improved recognition accuracy, particularly in the presence of noise, where complementary information from an alternate modality could be fused nicely in a neural net.

TDNNs have been used effectively in compact and high-performance handwriting recognition systems. Shift-invariance was also adapted to spatial patterns (x/y-axes) in image offline handwriting recognition.

Video has a temporal dimension that makes a TDNN an ideal solution to analysing motion patterns. An example of this analysis is a combination of vehicle detection and recognizing pedestrians.{{cite journal | doi=10.1016/S0262-8856(01)00040-3 | title=Real-time object recognition on image sequences with the adaptable time delay neural network algorithm — applications for autonomous vehicles | date=2001 | last1=Wöhler | first1=C. | last2=Anlauf | first2=J. K. | journal=Image and Vision Computing | volume=19 | issue=9–10 | pages=593–618 }} When examining videos, subsequent images are fed into the TDNN as input where each image is the next frame in the video. The strength of the TDNN comes from its ability to examine objects shifted in time forward and backward to define an object detectable as the time is altered. If an object can be recognized in this manner, an application can plan on that object to be found in the future and perform an optimal action.

Two-dimensional TDNNs were later applied to other image-recognition tasks under the name of "Convolutional Neural Networks", where shift-invariant training is applied to the x/y axes of an image.

• TDNNs can be implemented in virtually all machine-learning frameworks using one-dimensional convolutional neural networks, due to the equivalence of the methods.
• Matlab: The neural network toolbox has explicit functionality designed to produce a time delay neural network give the step size of time delays and an optional training function. The default training algorithm is a Supervised Learning back-propagation algorithm that updates filter weights based on the Levenberg-Marquardt optimizations. The function is timedelaynet(delays, hidden_layers, train_fnc) and returns a time-delay neural network architecture that a user can train and provide inputs to."[https://www.mathworks.com/help/deeplearning/time-series-and-dynamic-systems.html Time Series and Dynamic Systems - MATLAB & Simulink]". mathworks.com. Retrieved 21 June 2016.
• The Kaldi ASR Toolkit has an implementation of TDNNs with several optimizations for speech recognition.{{cite book |doi=10.1109/ASRU.2015.7404842 |chapter-url=http://danielpovey.com/files/2015_asru_aspire.pdf |chapter=JHU ASpIRE system: Robust LVCSR with TDNNS, iVector adaptation and RNN-LMS |title=2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) |date=2015 |last1=Peddinti |first1=Vijayaditya |last2=Chen |first2=Guoguo |last3=Manohar |first3=Vimal |last4=Ko |first4=Tom |last5=Povey |first5=Daniel |last6=Khudanpur |first6=Sanjeev |pages=539–546 |isbn=978-1-4799-7291-3 }}

• Convolutional neural network{{snd}} a convolutional neural net where the convolution is performed along the time axis of the data is very similar to a TDNN.
• Recurrent neural networks{{snd}} a recurrent neural network also handles temporal data, albeit in a different manner. Instead of a time-varied input, RNNs maintain internal hidden layers to keep track of past (and in the case of Bi-directional RNNs, future) inputs.

{{reflist}}{{Cite journal |last1=Haffner |first1=Patrick |last2=Waibel |date=1991 |orig-date=January 1991 |editor-last=Lippman |editor-first=Richard |editor2-last=Moody |editor2-first=John |title=Multi-State Time Delay Networks for Continuous Speech Recognition |url=https://www.researchgate.net/publication/221618146 |journal=Advances in Neural Information Processing Systems |publisher=Morgan Kaufman |volume=4 |pages=135–142}}

{{Cite journal |last1=Hampshire |first1=John |last2=Waibel |first2=Alex |orig-date=November 30, 1989 |editor-last=Touretzky |editor-first=David |title=Connectionist Architectures for Multi-Speaker Phoneme Recognition |url=http://papers.nips.cc/paper/213-connectionist-architectures-for-multi-speaker-phoneme-recognition |journal=Advances in Neural Information Processing Systems 2 |date=1990 |page=203-210}}

{{Cite journal |last1=Waibel |first1=Alex |last2=Hanazawa |first2=Toshiyuki |last3=Hinton |first3=Geoffrey |last4=Shikano |first4=Kiyohiro |last5=Lang |first5=Kevin |date=April 1989 |title=Phoneme recognition using time-delay neural networks |url=https://www.researchgate.net/publication/391037926 |journal= IEEE Transactions on Acoustics, Speech, and Signal Processing|volume=37 |issue=3 |pages=328–339 |doi=10.1109/29.21701}}

{{Cite journal |last=Waibel |first=Alex |date=1987 |orig-date=December |title=Phoneme Recognition Using Time-Delay Neural Networks |url=https://www.researchgate.net/publication/391037926 |journal=Conference: Meeting of the Institute of Electrical, Information and Communication Engineers (IEICE) |location=Japan}}

Category:Neural network architectures