tensor (machine learning)

A tensor is by definition a multilinear map. In mathematics, this may express a multilinear relationship between sets of algebraic objects. In physics, tensor fields, considered as tensors at each point in space, are useful in expressing mechanics such as stress or elasticity. In machine learning, the exact use of tensors depends on the statistical approach being used.

In 2001, the field of signal processing and statistics were making use of tensor methods. Pierre Comon surveys the early adoption of tensor methods in the fields of telecommunications, radio surveillance, chemometrics and sensor processing. Linear tensor rank methods (such as, Parafac/CANDECOMP) analyzed M-way arrays ("data tensors") composed of higher order statistics that were employed in blind source separation problems to compute a linear model of the data. He noted several early limitations in determining the tensor rank and efficient tensor rank decomposition.{{citation| first1=Pierre| last1=Comon| title=Tensor Decompositions: State of the Art and Applications|work=Mathematics in Signal Processing| publisher=Oxford University Press| year=2001}}

In the early 2000s, multilinear tensor methods{{citation| first1=M.A.O. |last1=Vasilescu|first2=Amnon| last2=Shashua| title=Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics Tutorial| publisher=International Conference on Machine Learning| url=http://www.media.mit.edu/~maov/classes/icml2007/index.html|date=2007}} crossed over into computer vision, computer graphics and machine learning with papers by Vasilescu or in collaboration with Terzopoulos, such as Human Motion Signatures,{{citation| first1=M.A.O. | last1=Vasilescu| title=Human Motion Signatures: Analysis, Synthesis, Recognition| publisher=Proceedings of International Conference on Pattern Recognition (ICPR 2002)| volume=3| pages=456–460| date=2002}}{{citation| first1=M.A.O. |last1=Vasilescu| title=An Algorithm for Extracting Human Motion Signatures| publisher=Computer Vision and Pattern Recognition CVPR 2001 Technical Sketches| date=2001}} TensorFaces TensorTexures{{citation|first1=M.A.O. |last1=Vasilescu|first2=D. |last2=Terzopoulos| title=TensorTextures: Multilinear Image-Based Rendering| work=ACM Transactions on Graphics| volume=23| pages=336–342| date=2004| issue=3 |url=http://www.media.mit.edu/~maov/tensortextures/Vasilescu_siggraph04.pdf|doi=10.1145/1015706.1015725}} and Multilinear Projection.{{cite conference| title=Multilinear Projection for Face Recognition via Canonical Decomposition |first=M.A.O.| last=Vasilescu| publisher=In Proc. Face and Gesture Conference (FG'11)|pages=476–483|date=2011}}{{citation| title="Multilinear Projection for Appearance-Based Recognition in the Tensor Framework"| first1=M.A.O.| last1=Vasilescu| first2=D.| last2=Terzopoulos| publisher=Proc. Eleventh IEEE International Conf. on Computer Vision (ICCV'07)| date=2007| pages=1–8}} Multilinear algebra, the algebra of higher-order tensors, is a suitable and transparent framework for analyzing the multifactor structure of an ensemble of observations and for addressing the difficult problem of disentangling the causal factors based on second order{{cite book| first1=M.A.O.| last1=Vasilescu| first2=D.| last2=Terzopoulos| url=http://www.cs.toronto.edu/~maov/tensorfaces/Springer%20ECCV%202002_files/eccv02proceeding_23500447.pdf| title=Multilinear Analysis of Image Ensembles: TensorFaces| series=Lecture Notes in Computer Science 2350; (Presented at Proc. 7th European Conference on Computer Vision (ECCV'02), Copenhagen, Denmark)|publisher=Springer, Berlin, Heidelberg|doi=10.1007/3-540-47969-4_30| isbn=9783540437451| year=2002}} or higher order statistics associated with each causal factor.{{citation| first1=M.A.O. |last1=Vasilescu| first2=D. |last2=Terzopoulos| url=http://www.media.mit.edu/~maov/mica/mica05.pdf|title=Multilinear Independent Component Analysis| series=LProceedings of the 2005 IEEE Computer Vision and Pattern Recognition Conference (CVPR 2005), San Diego, CA|year=2005}}

Tensor (multilinear) factor analysis disentangles and reduces the influence of different causal factors with multilinear subspace learning.{{citation| first1=M.A.O.| last1=Vasilescu| first2=D.| last2=Terzopoulos| title=Multilinear Subspace Learning of Image Ensembles|work=2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings.| date=2003| url=https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=f50a23e011df6563afc9d97e2ede96c81ebc965f}}

When treating an image or a video as a 2- or 3-way array, i.e., "data matrix/tensor", tensor methods reduce spatial or time redundancies as demonstrated by Wang and Ahuja.{{citation| first1=H| last1=Wang| first2=Ahuja| last2=Narendra|title=Rank-R approximation of tensors using image-as-matrix representation| publisher=IEEE Computer Society Conference on Computer Vision and Pattern Recognition| year=2005}}

Yoshua Bengio,{{cite arXiv |last1=Desjardins |first1=Guillaume |last2=Courvile |first2=Aaron |last3=Bengio |first3=Yoshua |title=Disentangling Factors of Variation via Generative Entangling |year=2012 |class=stat.ML |eprint=1210.5474 }}{{cite journal |last1=Bengio |first1=Yoshua |last2=Courville |first2=Aaron |title=Disentangling Factors of Variation via Generative Entangling |journal=Handbook on Neural Information Processing |date=2013 |pages=1–28 |isbn=9783642366574 }}

Geoff Hinton

{{cite journal |last1=Tang |first1=Yichuan |last2=Salakhutdinov |first2=Ruslan |last3=Hinton |first3=Geoffrey |title=Tensor Analyzers |journal=30 Th International Conference on Machine Learning |date=2013 |url=http://proceedings.mlr.press/v28/tang13.pdf}}{{cite journal |last1=Memisevic |first1=Roland |last2=Hinton |first2=Geoffrey |title=Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines |journal=Neural Computation |date=2010 |volume=22|issue=6 |pages=1473–1492 |doi=10.1162/neco.2010.01-09-953 |pmid=20141471 |s2cid=1413690 |url=https://www.cs.toronto.edu/~hinton/absps/rolandNC.pdf}} and their collaborators briefly discuss the relationship between deep neural networks and tensor factor analysis beyond the use of M-way arrays ("data tensors") as inputs. One of the early uses of tensors for neural networks appeared in natural language processing. A single word can be expressed as a vector via Word2vec. Thus a relationship between two words can be encoded in a matrix. However, for more complex relationships such as subject-object-verb, it is necessary to build higher-dimensional networks. In 2009, the work of Sutskever introduced Bayesian Clustered Tensor Factorization to model relational concepts while reducing the parameter space.{{cite journal| last1=Sutskever| first1=I| title=Modeling Relational Data using Bayesian Clustered Tensor Factorization| journal=Advances in Neural Information Processing Systems| year=2009| volume=22}} From 2014 to 2015, tensor methods become more common in convolutional neural networks (CNNs). Tensor methods organize neural network weights in a "data tensor", analyze and reduce the number of neural network weights.{{citation| first1=A| last1=Novikov| first2=P| last2=Dmitry| first3=A| last3=Osokin| first4=D| last4=Vetrov| title=Tensorizing Neural Networks|journal=Neural Information Processing Systems|year=2015|arxiv=1509.06569 }}{{cite arXiv|last1=Kossaifi| first1=Jean| title=T-Net: Parameterizing Fully Convolutional Nets with a Single High-Order Tensor| year=2019| class=cs.CV | eprint=1904.02698 }} Lebedev et al. accelerated CNN networks for character classification (the recognition of letters and digits in images) by using 4D kernel tensors.{{citation| first1=Vadim| last1=Lebedev| title=Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition| year=2014| arxiv=1412.6553 }}

Let $\backslash mathbb\; F$ be a field such as the real numbers $\backslash mathbb\; R$ or the complex numbers $\backslash mathbb\; C$. A tensor $\{\backslash mathcal\; T\}\; \backslash in\; \{\backslash mathbb\; F\}^\{I\_0\; \backslash times\; I\_2\; \backslash times\; \backslash ldots\; \backslash times\; I\_C\}$ is a multilinear transformation from a set of domain vector spaces to a range vector space:

$\{\backslash mathcal\; T\}:\backslash \{\{\backslash mathbb\; F\}^\{I\_1\}\; \backslash times\; \{\backslash mathbb\; F\}^\{I\_2\}\; \backslash times\; \backslash ldots\; \{\backslash mathbb\; F\}^\{I\_C\}\backslash \}\backslash mapsto\{\backslash mathbb\; F\}^\{I\_0\}$

Here, $C$ and $I\_0,\; I\_1,\; \backslash ldots,\; I\_C$ are positive integers, and $(C+1)$ is the number of modes of a tensor (also known as the number of ways of a multi-way array). The dimensionality of mode $c$ is $I\_c$, for $0\backslash le\; c\backslash le\; C$ .{{citation|last1=Vasilescu|first1=M. Alex O.|title= A Multilinear (Tensor) Framework for Computer Graphics, Computer Vision and Machine Learning"|year=2009|url=https://web.cs.ucla.edu/~dt/theses/vasilescu-thesis.pdf}}{{cite arXiv|last1=Rabanser|first1=Stephan|title=Introduction to Tensor Decompositions and their Applications in Machine Learning|year=2017|class=stat.ML |eprint=1711.10781}}

File:TensorFaces16 (1).png

In statistics and machine learning, an image is vectorized when viewed as a single observation, and a collection of vectorized images is organized as a "data tensor". For example, a set of facial images $\backslash \{\{\backslash mathbb\; d\}\_\{i\_p,i\_e,i\_l,i\_v\}\backslash in\; \{\backslash mathbb\; R\}^\{I\_X\}\backslash \}$ with $I\_X$ pixels that are the consequences of multiple causal factors, such as a facial geometry $i\_p\; (1\backslash le\; i\_p\backslash le\; I\_P)$, an expression $i\_e\; (1\backslash le\; i\_e\backslash le\; I\_E)$, an illumination condition $i\_l\; (1\backslash le\; i\_l\backslash le\; I\_L)$, and a viewing condition $i\_v\; (1\backslash le\; i\_v\backslash le\; I\_V)$ may be organized into a data tensor (ie. multiway array) $\{\backslash mathcal\; D\}\backslash in\; \{\backslash mathbb\; R\}^\{I\_X\backslash times\; I\_P\; \backslash times\; I\_E\backslash times\; I\_L\; \backslash times\; V\}$ where $I\_P$ are the total number of facial geometries, $I\_E$ are the total number of expressions, $I\_L$ are the total number of illumination conditions, and $I\_V$ are the total number of viewing conditions. Tensor factorizations methods such as TensorFaces and multilinear (tensor) independent component analysis factorizes the data tensor into a set of vector spaces that span the causal factor representations, where an image is the result of tensor transformation $\{\backslash mathcal\; T\}$ that maps a set of causal factor representations to the pixel space.

File:M mode svd image as matrix.png

Another approach to using tensors in machine learning is to embed various data types directly. For example, a grayscale image, commonly represented as a discrete 2-way array $\{\backslash mathbf\; D\}\backslash in\; \{\backslash mathbb\; R\}^\{I\_\{RX\}\backslash times\; I\_\{CX\}\}$ with dimensionality $I\_\{RX\}\; \backslash times\; I\_\{CX\}$ where $I\_\{RX\}$ are the number of rows and $I\_\{CX\}$ are the number of columns. When an image is treated as 2-way array or 2nd order tensor (i.e. as a collection of column/row observations), tensor factorization methods compute the image column space, the image row space and the normalized PCA coefficients or the ICA coefficients.

Similarly, a color image with RGB channels, $\backslash mathcal\{D\}\backslash in\; \backslash mathbb\{R\}^\{N\; \backslash times\; M\; \backslash times\; 3\}.$ may be viewed as a 3rd order data tensor or 3-way array.--------

In natural language processing, a word might be expressed as a vector $v$ via the Word2vec algorithm. Thus $v$ becomes a mode-1 tensor

: $v\; \backslash mapsto\; \backslash mathcal\{A\}\backslash in\; \backslash mathbb\{R\}^N.$

The embedding of subject-object-verb semantics requires embedding relationships among three words. Because a word is itself a vector, subject-object-verb semantics could be expressed using mode-3 tensors

: $v\_a\; \backslash times\; v\_b\; \backslash times\; v\_c\; \backslash mapsto\; \backslash mathcal\{A\}\backslash in\; \backslash mathbb\{R\}^\{N\; \backslash times\; N\; \backslash times\; N\}.$

In practice the neural network designer is primarily concerned with the specification of embeddings, the connection of tensor layers, and the operations performed on them in a network. Modern machine learning frameworks manage the optimization, tensor factorization and backpropagation automatically.

File:Tensor Units.jpg

Tensors may be used as the unit values of neural networks which extend the concept of scalar, vector and matrix values to multiple dimensions.

The output value of single layer unit $y\_m$ is the sum-product of its input units and the connection weights filtered through the activation function $f$:

: $y\_m\; =\; f\backslash left(\backslash sum\_n\; x\_n\; u\_\{m,n\}\backslash right),$

where

:$y\_m\; \backslash in\; \backslash mathbb\{R\}.$

File:Typical cnn.png

If each output element of $y\_m$ is a scalar, then we have the classical definition of an artificial neural network. By replacing each unit component with a tensor, the network is able to express higher dimensional data such as images or videos:

: $y\_m\; \backslash in\; \backslash mathbb\{R\}^\{I\_0\; \backslash times\; I\_1\; \backslash times\; ..\; \backslash times\; I\_C\}.$

This use of tensors to replace unit values is common in convolutional neural networks where each unit might be an image processed through multiple layers. By embedding the data in tensors such network structures enable learning of complex data types.

File:Fully connected neural network and it's expression as a tensor product.jpg

Tensors may also be used to compute the layers of a fully connected neural network, where the tensor is applied to the entire layer instead of individual unit values.

The output value of single layer unit $y\_m$ is the sum-product of its input units and the connection weights filtered through the activation function $f$:

: $y\_m\; =\; f\backslash left(\backslash sum\_n\; x\_n\; u\_\{m,n\}\backslash right).$

The vectors $x$ and $y$ of output values can be expressed as a mode-1 tensors, while the hidden weights can be expressed as a mode-2 tensor. In this example the unit values are scalars while the tensor takes on the dimensions of the network layers:

: $x\_n\; \backslash mapsto\; \backslash mathcal\{X\}\backslash in\; \backslash mathbb\{R\}^\{1\; \backslash times\; N\},$

: $y\_n\; \backslash mapsto\; \backslash mathcal\{Y\}\backslash in\; \backslash mathbb\{R\}^\{M\; \backslash times\; 1\},$

: $u\_n\; \backslash mapsto\; \backslash mathcal\{U\}\backslash in\; \backslash mathbb\{R\}^\{N\; \backslash times\; M\}.$

In this notation, the output values can be computed as a tensor product of the input and weight tensors:

: $\backslash mathcal\{Y\}\; =\; f\; (\; \backslash mathcal\{X\}\; \backslash mathcal\{U\}\; ).$

which computes the sum-product as a tensor multiplication (similar to matrix multiplication).

This formulation of tensors enables the entire layer of a fully connected network to be efficiently computed by mapping the units and weights to tensors.

A different reformulation of neural networks allows tensors to express the convolution layers of a neural network. A convolutional layer has multiple inputs, each of which is a spatial structure such as an image or volume. The inputs are convolved by filtering before being passed to the next layer. A typical use is to perform feature detection or isolation in image recognition.

Convolution is often computed as the multiplication of an input signal $g$ with a filter kernel $f$. In two dimensions the discrete, finite form is:

: $(f*g)\_\{x,y\}\; =\; \backslash sum\_\{j=-w\}^w\; \backslash sum\_\{k=-w\}^w\; f\_\{j,k\}\; g\_\{x+j,y+k\},$

where $w$ is the width of the kernel.

This definition can be rephrased as a matrix-vector product in terms of tensors that express the kernel, data and inverse transform of the kernel.{{cite arXiv|last1=Bedden|first1=David|title=Deep Tensor Convolution on Multicores|year=2017|class=cs.CV |eprint=1611.06565 }}

: $\backslash mathcal\{Y\}\; =\; \backslash mathcal\{A\}[(Cg)\; \backslash odot\; (Bd)],$

where $\backslash mathcal\{A\},\; \backslash mathcal\{B\}$ and $\backslash mathcal\{C\}$ are the inverse transform, data and kernel. The derivation is more complex when the filtering kernel also includes a non-linear activation function such as sigmoid or ReLU.

The hidden weights of the convolution layer are the parameters to the filter. These can be reduced with a pooling layer which reduces the resolution (size) of the data, and can also be expressed as a tensor operation.

An important contribution of tensors in machine learning is the ability to factorize tensors to decompose data into constituent factors or reduce the learned parameters. Data tensor modeling techniques stem from the linear tensor decomposition (CANDECOMP/Parafac decomposition) and the multilinear tensor decompositions (Tucker).

File:Tensor tucker.jpg

Tucker decomposition, for example, takes a 3-way array $\backslash mathcal\{X\}\; \backslash in\; \backslash mathbb\{R\}^\{I\; \backslash times\; J\; \backslash times\; K\}$

and decomposes the tensor into three matrices $\backslash mathcal\{A,B,C\}$ and a smaller tensor $\backslash mathcal\{G\}$. The shape of the matrices and new tensor are such that the total number of elements is reduced. The new tensors have shapes

: $\backslash mathcal\{A\}\; \backslash in\; \backslash mathbb\{R\}^\{I\; \backslash times\; P\},$

: $\backslash mathcal\{B\}\; \backslash in\; \backslash mathbb\{R\}^\{J\; \backslash times\; Q\},$

: $\backslash mathcal\{C\}\; \backslash in\; \backslash mathbb\{R\}^\{K\; \backslash times\; R\},$

: $\backslash mathcal\{G\}\; \backslash in\; \backslash mathbb\{R\}^\{P\; \backslash times\; Q\; \backslash times\; R\}.$

Then the original tensor can be expressed as the tensor product of these four tensors:

: $\backslash mathcal\{X\}\; =\; \backslash mathcal\{G\}\; \backslash times\; \backslash mathcal\{A\}\; \backslash times\; \backslash mathcal\{B\}\; \backslash times\; \backslash mathcal\{C\}.$

In the example shown in the figure, the dimensions of the tensors are

: $\backslash mathcal\{X\}$: I=8, J=6, K=3, $\backslash mathcal\{A\}$: I=8, P=5, $\backslash mathcal\{B\}$: J=6, Q=4, $\backslash mathcal\{C\}$: K=3, R=2, $\backslash mathcal\{G\}$: P=5, Q=4, R=2.

The total number of elements in the Tucker factorization is

: $|\backslash mathcal\{A\}|+|\backslash mathcal\{B\}|+|\backslash mathcal\{C\}|+|\backslash mathcal\{G\}|\; =$

: $(I\; \backslash times\; P)\; +\; (J\; \backslash times\; Q)\; +\; (K\; \backslash times\; R)\; +\; (P\; \backslash times\; Q\; \backslash times\; R)\; =\; 8\backslash times5\; +\; 6\backslash times4\; +\; 3\backslash times2\; +\; 5\backslash times4\backslash times2\; =\; 110.$

The number of elements in the original $\backslash mathcal\{X\}$ is 144, resulting in a data reduction from 144 down to 110 elements, a reduction of 23% in parameters or data size. For much larger initial tensors, and depending on the rank (redundancy) of the tensor, the gains can be more significant.

The work of Rabanser et al. provides an introduction to tensors with more details on the extension of Tucker decomposition to N-dimensions beyond the mode-3 example given here.

Another technique for decomposing tensors rewrites the initial tensor as a sequence (train) of smaller sized tensors. A tensor-train (TT) is a sequence of tensors of reduced rank, called canonical factors. The original tensor can be expressed as the sum-product of the sequence.

: $\backslash mathcal\{X\}\; =\; \backslash mathcal\{G\_1\}\; \backslash mathcal\{G\_2\}\; \backslash mathcal\{G\_3\}\; ..\; \backslash mathcal\{G\_d\}$

Developed in 2011 by Ivan Oseledts, the author observes that Tucker decomposition is "suitable for small dimensions, especially for the three-dimensional case. For large d it is not suitable."{{cite journal|title=Tensor-Train Decomposition|last1=Oseledets|first1=Ivan|journal=SIAM Journal on Scientific Computing|volume=33|number=5|pages=2295–2317|year=2011|doi=10.1137/090752286 |bibcode=2011SJSC...33.2295O |s2cid=207059098 }} Thus tensor-trains can be used to factorize larger tensors in higher dimensions.

The unified data architecture and automatic differentiation of tensors has enabled higher-level designs of machine learning in the form of tensor graphs. This leads to new architectures, such as tensor-graph convolutional networks (TGCN), which identify highly non-linear associations in data, combine multiple relations, and scale gracefully, while remaining robust and performant.{{cite journal| last1=Ioannidis| first1=Vassilis| title=Tensor Graph Convolutional Networks for Multi-Relational and Robust Learning| journal=IEEE Transactions on Signal Processing| volume=68| year=2020| pages=6535–6546| doi=10.1109/TSP.2020.3028495 | arxiv=2003.07729 |bibcode=2020ITSP...68.6535I |s2cid=212736801 }}

These developments are impacting all areas of machine learning, such as text mining and clustering, time varying data, and neural networks wherein the input data is a social graph and the data changes dynamically.{{cite journal|title=Tensor-based Graph Modularity for Text Data Clustering|last1=Boutalbi|first1=Rafika|year=2022|journal=Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval}}{{cite journal|title=Tensor Graph Neural Networks for Learning on Time Varying Graphs|last1=Malik|first1=Osman|journal=2019 Conference on Neural Information Processing (NeurIPS)|year=2019}}{{cite journal|arxiv=1704.06199|doi=10.1016/j.patcog.2019.107000 |title=Dynamic graph convolutional networks |year=2020 |last1=Manessi |first1=Franco |last2=Rozza |first2=Alessandro |last3=Manzo |first3=Mario |journal=Pattern Recognition |volume=97 |page=107000 |bibcode=2020PatRe..9707000M |s2cid=16745566 }}{{cite web|title=Dynamic Graph Convolutional Networks Using the Tensor M-Product|last1=Malik|first1=Osman|url=https://par.nsf.gov/servlets/purl/10285968}}

Tensors provide a unified way to train neural networks for more complex data sets. However, training is expensive to compute on classical CPU hardware.

In 2014, Nvidia developed [https://developer.nvidia.com/cudnn cuDNN], CUDA Deep Neural Network, a library for a set of optimized primitives written in the parallel CUDA language.{{cite web|title=Nvidia Introduces cuDNN, a CUDA-based library for Deep Neural Networks|url=https://www.infoq.com/news/2014/09/cudnn/|last1=Serrano|first1=Jerome|year=2014}} CUDA and thus cuDNN run on dedicated GPUs that implement unified massive parallelism in hardware. These GPUs were not yet dedicated chips for tensors, but rather existing hardware adapted for parallel computation in machine learning.

In the period 2015–2017 Google invented the Tensor Processing Unit (TPU).{{cite journal |title=Motivation for and evaluation of the first tensor processing unit |last1=Jouppi |first1=Norman |last2=Young |last3=Patil |last4=Patterson |year=2018 |journal=IEEE Micro |volume=38 |issue=3 |pages=10–19 |doi=10.1109/MM.2018.032271057 |s2cid=21657842 }} TPUs are dedicated, fixed function hardware units that specialize in the matrix multiplications needed for tensor products. Specifically, they implement an array of 65,536 multiply units that can perform a 256x256 matrix sum-product in just one global instruction cycle.{{cite journal |title=First In-Depth Look at Google's TPU Architecture|year=2017 |last1=Hemsoth |first1=Nicole |journal=The Next Platform |url=https://www.nextplatform.com/2017/04/05/first-depth-look-googles-tpu-architecture/}}

Later in 2017, Nvidia released its own Tensor Core with the Volta GPU architecture. Each Tensor Core is a microunit that can perform a 4x4 matrix sum-product. There are eight tensor cores for each shared memory (SM) block.{{cite web |title=NVIDIA Tesla V100 GPU Architecture |year=2017 |url=https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf}} The first GV100 GPU card has 108 SMs resulting in 672 tensor cores. This device accelerated machine learning by 12x over the previous Tesla GPUs.{{cite web|title=On Tensors, Tensorflow, And Nvidia's Latest 'Tensor Cores' |last1=Armasu |first1=Lucian |year=2017 |url=https://www.tomshardware.com/news/nvidia-tensor-core-tesla-v100,34384.html}} The number of tensor cores scales as the number of cores and SM units continue to grow in each new generation of cards.

The development of GPU hardware, combined with the unified architecture of tensor cores, has enabled the training of much larger neural networks. In 2022, the largest neural network was Google's PaLM with 540 billion learned parameters (network weights){{Cite web |title=Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance |url=https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html |access-date=2023-03-29 |website=ai.googleblog.com |language=en}} (the older GPT-3 language model has over 175 billion learned parameters that produces human-like text; size isn't everything, Stanford's much smaller 2023 Alpaca model claims to be better,{{Cite web |title=Alpaca: A Strong, Replicable Instruction-Following Model |url=https://crfm.stanford.edu/2023/03/13/alpaca.html |access-date=2023-03-29 |website=crfm.stanford.edu}} having learned from Meta/Facebook's 2023 model LLaMA, the smaller 7 billion parameter variant). The widely popular chatbot ChatGPT is built on top of GPT-3.5 (and after an update GPT-4) using supervised and reinforcement learning.

{{reflist}}

*

Category:Machine learning

Category:Tensors