PyTorch

{{Infobox software

| name = PyTorch

| logo = PyTorch logo black.svg

| screenshot =

| caption =

| collapsible =

| author = {{Unbulleted list|Adam Paszke|Sam Gross|Soumith Chintala|Gregory Chanan}}

| developer = Meta AI

| released = {{Start date and age|2016|9|df=yes}}{{cite web|url=https://github.com/pytorch/pytorch/releases/tag/v0.1.1|title=PyTorch Alpha-1 release|last=Chintala|first=Soumith|website=GitHub|date=1 September 2016|access-date=19 August 2020|archive-date=29 August 2021|archive-url=https://web.archive.org/web/20210829055231/https://github.com/pytorch/pytorch/releases/tag/v0.1.1|url-status=live}}

| latest release version = {{wikidata|property|edit|reference|P348}}

| latest release date = {{start date and age|{{wikidata|qualifier|P348|P577}}}}

| repo = {{URL|github.com/pytorch/pytorch}}

| programming language = {{Unbulleted list|Python|C++|CUDA}}

| operating system = {{Unbulleted list|Linux|macOS|Windows}}

| platform = IA-32, x86-64, ARM64

| language = English

| genre = Library for machine learning and deep learning

| license = BSD-3{{cite web |last=Claburn |first=Thomas |date=12 September 2022 |title=PyTorch gets lit under The Linux Foundation |url=https://www.theregister.com/2022/09/12/pytorch_meta_linux_foundation/ |work=The Register |access-date=18 October 2022 |archive-date=18 October 2022 |archive-url=https://web.archive.org/web/20221018040848/https://www.theregister.com/2022/09/12/pytorch_meta_linux_foundation/ |url-status=live }}

| website = {{URL|https://pytorch.org/}}

}}

PyTorch is a machine learning library based on the Torch library,{{cite news|url=https://www.infoworld.com/article/3159120/artificial-intelligence/facebook-brings-gpu-powered-machine-learning-to-python.html|title=Facebook brings GPU-powered machine learning to Python|last=Yegulalp|first=Serdar|date=19 January 2017|work=InfoWorld|access-date=11 December 2017|archive-date=12 July 2018|archive-url=https://web.archive.org/web/20180712054543/https://www.infoworld.com/article/3159120/artificial-intelligence/facebook-brings-gpu-powered-machine-learning-to-python.html|url-status=live}}{{cite web|url=https://www.oreilly.com/ideas/why-ai-and-machine-learning-researchers-are-beginning-to-embrace-pytorch|title=Why AI and machine learning researchers are beginning to embrace PyTorch|last=Lorica|first=Ben|date=3 August 2017|publisher=O'Reilly Media|access-date=11 December 2017|archive-date=17 May 2019|archive-url=https://web.archive.org/web/20190517055218/https://www.oreilly.com/ideas/why-ai-and-machine-learning-researchers-are-beginning-to-embrace-pytorch|url-status=live}}{{Cite book|title=Deep Learning with Python|last=Ketkar|first=Nikhil|date=2017|publisher=Apress, Berkeley, CA|isbn=9781484227657|pages=195–208|language=en|doi=10.1007/978-1-4842-2766-4_12|chapter=Introduction to PyTorch}} used for applications such as computer vision and natural language processing,{{Cite web|url=https://www.datacamp.com/tutorial/nlp-with-pytorch-a-comprehensive-guide|title=NLP with PyTorch: A Comprehensive Guide|author=Moez Ali|date=Jun 2023|website=datacamp.com|language=en|access-date=1 April 2024|archive-date=1 April 2024|archive-url=https://web.archive.org/web/20240401214813/https://www.datacamp.com/tutorial/nlp-with-pytorch-a-comprehensive-guide|url-status=live}} originally developed by Meta AI and now part of the Linux Foundation umbrella.{{Cite news|url=https://www.oreilly.com/ideas/when-two-trends-fuse-pytorch-and-recommender-systems|title=When two trends fuse: PyTorch and recommender systems|last=Patel|first=Mo|date=7 December 2017|work=O'Reilly Media|access-date=18 December 2017|language=en|archive-date=30 March 2019|archive-url=https://web.archive.org/web/20190330131436/https://www.oreilly.com/ideas/when-two-trends-fuse-pytorch-and-recommender-systems|url-status=live}}{{Cite news|url=https://techcrunch.com/2017/09/07/facebook-and-microsoft-collaborate-to-simplify-conversions-from-pytorch-to-caffe2/|title=Facebook and Microsoft collaborate to simplify conversions from PyTorch to Caffe2|last=Mannes|first=John|work=TechCrunch|access-date=18 December 2017|language=en|quote=FAIR is accustomed to working with PyTorch – a deep learning framework optimized for achieving state of the art results in research, regardless of resource constraints. Unfortunately in the real world, most of us are limited by the computational capabilities of our smartphones and computers.|archive-date=6 July 2020|archive-url=https://web.archive.org/web/20200706115906/https://techcrunch.com/2017/09/07/facebook-and-microsoft-collaborate-to-simplify-conversions-from-pytorch-to-caffe2/|url-status=live}}{{Cite web|url=https://venturebeat.com/2017/11/29/tech-giants-are-using-open-source-frameworks-to-dominate-the-ai-community/|title=Tech giants are using open source frameworks to dominate the AI community|last=Arakelyan|first=Sophia|date=29 November 2017|website=VentureBeat|language=en-US|access-date=18 December 2017|archive-date=30 March 2019|archive-url=https://web.archive.org/web/20190330131432/https://venturebeat.com/2017/11/29/tech-giants-are-using-open-source-frameworks-to-dominate-the-ai-community/|url-status=live}}{{Cite web |title=PyTorch strengthens its governance by joining the Linux Foundation |url=https://pytorch.org/blog/PyTorchfoundation/ |access-date=13 September 2022 |website=pytorch.org |language=en}} It is one of the most popular deep learning frameworks, alongside others such as TensorFlow,{{Cite web|url=https://github.com/cncf/velocity|title=Top 30 Open Source Projects.|website=Open Source Project Velocity by CNCF|access-date=12 October 2023|archive-date=3 September 2023|archive-url=https://web.archive.org/web/20230903024925/https://github.com/cncf/velocity|url-status=live}} offering free and open-source software released under the modified BSD license. Although the Python interface is more polished and the primary focus of development, PyTorch also has a C++ interface.{{Cite web|url=https://pytorch.org/cppdocs/frontend.html|title=The C++ Frontend|website=PyTorch Master Documentation|access-date=29 July 2019|archive-date=29 July 2019|archive-url=https://web.archive.org/web/20190729202037/https://pytorch.org/cppdocs/frontend.html|url-status=live}}

A number of pieces of deep learning software are built on top of PyTorch, including Tesla Autopilot,{{Cite web|last=Karpathy|first=Andrej|title=PyTorch at Tesla - Andrej Karpathy, Tesla|website=YouTube|date=6 November 2019|url=https://www.youtube.com/watch?v=oBklltKXtDE|access-date=2 June 2020|archive-date=24 March 2023|archive-url=https://web.archive.org/web/20230324144838/https://www.youtube.com/watch?v=oBklltKXtDE|url-status=live}} Uber's Pyro,{{Cite news|url=https://eng.uber.com/pyro/|title=Uber AI Labs Open Sources Pyro, a Deep Probabilistic Programming Language|date=3 November 2017|work=Uber Engineering Blog|access-date=18 December 2017|language=en-US|archive-date=25 December 2017|archive-url=https://web.archive.org/web/20171225034106/https://eng.uber.com/pyro/|url-status=live}} Hugging Face's Transformers,{{Citation|title=PYTORCH-TRANSFORMERS: PyTorch implementations of popular NLP Transformers|date=1 December 2019|url=https://pytorch.org/hub/huggingface_pytorch-transformers/|publisher=PyTorch Hub|access-date=1 December 2019|archive-date=11 June 2023|archive-url=https://web.archive.org/web/20230611061047/https://pytorch.org/hub/huggingface_pytorch-transformers/|url-status=live}}{{Cite web|url=https://pytorch.org/ecosystem/|title=Ecosystem Tools|website=pytorch.org|language=en|access-date=18 June 2020|archive-date=18 July 2023|archive-url=https://web.archive.org/web/20230718105354/https://pytorch.org/ecosystem/|url-status=live}} and Catalyst.{{Citation|title=GitHub - catalyst-team/catalyst: Accelerated DL & RL|date=5 December 2019|url=https://github.com/catalyst-team/catalyst|publisher=Catalyst-Team|access-date=5 December 2019|archive-date=22 December 2019|archive-url=https://web.archive.org/web/20191222162045/https://github.com/catalyst-team/catalyst|url-status=live}}{{Cite web|url=https://pytorch.org/ecosystem/|title=Ecosystem Tools|website=pytorch.org|language=en|access-date=4 April 2020|archive-date=18 July 2023|archive-url=https://web.archive.org/web/20230718105354/https://pytorch.org/ecosystem/|url-status=live}}

PyTorch provides two high-level features:{{cite web |url=https://pytorch.org/about/ |title=PyTorch – About |website=pytorch.org |access-date=11 June 2018 |archive-url=https://web.archive.org/web/20180615190804/https://pytorch.org/about/ |archive-date=15 June 2018 |url-status=dead }}

Tensor computing (like NumPy) with strong acceleration via graphics processing units (GPU)
Deep neural networks built on a tape-based automatic differentiation system

History

Meta (formerly known as Facebook) operates both PyTorch and Convolutional Architecture for Fast Feature Embedding (Caffe2), but models defined by the two frameworks were mutually incompatible. The Open Neural Network Exchange (ONNX) project was created by Meta and Microsoft in September 2017 for converting models between frameworks. Caffe2 was merged into PyTorch at the end of March 2018.{{cite web|url=https://medium.com/@Synced/caffe2-merges-with-pytorch-a89c70ad9eb7|title=Caffe2 Merges With PyTorch|date=2 April 2018|access-date=2 January 2019|archive-date=30 March 2019|archive-url=https://web.archive.org/web/20190330143500/https://medium.com/@Synced/caffe2-merges-with-pytorch-a89c70ad9eb7|url-status=live}} In September 2022, Meta announced that PyTorch would be governed by the independent PyTorch Foundation, a newly created subsidiary of the Linux Foundation.{{cite web |url=https://arstechnica.com/information-technology/2022/09/meta-spins-off-pytorch-foundation-to-make-ai-framework-vendor-neutral/ |title=Meta spins off PyTorch Foundation to make AI framework vendor neutral |date=12 September 2022 |website=Ars Technica |last=Edwards |first=Benj |access-date=13 September 2022 |archive-date=13 September 2022 |archive-url=https://web.archive.org/web/20220913034850/https://arstechnica.com/information-technology/2022/09/meta-spins-off-pytorch-foundation-to-make-ai-framework-vendor-neutral/ |url-status=live }}

PyTorch 2.0 was released on 15 March 2023, introducing TorchDynamo, a Python-level compiler that makes code run up to 2x faster, along with significant improvements in training and inference performance across major cloud platforms.{{cite web|title=Dynamo Overview |url=https://pytorch.org/docs/stable/torch.compiler_dynamo_overview.html }}{{cite news |title=PyTorch 2.0 brings new fire to open-source machine learning |url=https://venturebeat.com/ai/pytorch-2-0-brings-new-fire-to-open-source-machine-learning/ |access-date=16 March 2023 |work=VentureBeat |date=15 March 2023 |archive-date=16 March 2023 |archive-url=https://web.archive.org/web/20230316004808/https://venturebeat.com/ai/pytorch-2-0-brings-new-fire-to-open-source-machine-learning/ |url-status=live }}

PyTorch tensors

PyTorch defines a class called Tensor (torch.Tensor) to store and operate on homogeneous multidimensional rectangular arrays of numbers. PyTorch Tensors are similar to NumPy Arrays, but can also be operated on a CUDA-capable NVIDIA GPU. PyTorch has also been developing support for other GPU platforms, for example, AMD's ROCm{{cite web|url=https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html|title=Installing PyTorch for ROCm|date=9 February 2024|website=rocm.docs.amd.com}} and Apple's Metal Framework.{{Cite web |title=Introducing Accelerated PyTorch Training on Mac |url=https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/ |access-date=4 June 2022 |website=pytorch.org |language=en |archive-date=29 January 2024 |archive-url=https://web.archive.org/web/20240129141050/https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/ |url-status=live }}

PyTorch supports various sub-types of Tensors.{{cite web |url=https://www.analyticsvidhya.com/blog/2018/02/pytorch-tutorial/ |title=An Introduction to PyTorch – A Simple yet Powerful Deep Learning Library |website=analyticsvidhya.com |access-date=11 June 2018 |date=22 February 2018 |archive-date=22 October 2019 |archive-url=https://web.archive.org/web/20191022200858/https://www.analyticsvidhya.com/blog/2018/02/pytorch-tutorial/ |url-status=live }}

Note that the term "tensor" here does not carry the same meaning as tensor in mathematics or physics. The meaning of the word in machine learning is only superficially related to its original meaning as a certain kind of object in linear algebra. Tensors in PyTorch are simply multi-dimensional arrays.

PyTorch neural networks

PyTorch defines a module called nn (torch.nn) to describe neural networks and to support training. This module offers a comprehensive collection of building blocks for neural networks, including various layers and activation functions, enabling the construction of complex models. Networks are built by inheriting from the torch.nn module and defining the sequence of operations in the forward() function.

Example

The following program shows the low-level functionality of the library with a simple example.

import torch

dtype = torch.float

device = torch.device("cpu") # Execute all calculations on the CPU

device = torch.device("cuda:0") # Executes all calculations on the GPU

Create a tensor and fill it with random numbers

a = torch.randn(2, 3, device=device, dtype=dtype)

print(a)

Output: tensor([[-1.1884, 0.8498, -1.7129],
[-0.8816, 0.1944, 0.5847]])

b = torch.randn(2, 3, device=device, dtype=dtype)

print(b)

Output: tensor([[ 0.7178, -0.8453, -1.3403],
[ 1.3262, 1.1512, -1.7070]])

print(a * b)

Output: tensor([[-0.8530, -0.7183, 2.58],
[-1.1692, 0.2238, -0.9981]])

print(a.sum())

Output: tensor(-2.1540)

print(a[1,2]) # Output of the element in the third column of the second row (zero based)

Output: tensor(0.5847)

print(a.max())

Output: tensor(0.8498)

The following code-block defines a neural network with linear layers using the nn module.

from torch import nn # Import the nn sub-module from PyTorch

class NeuralNetwork(nn.Module): # Neural networks are defined as classes

def __init__(self): # Layers and variables are defined in the __init__ method

super().__init__() # Must be in every network.

self.flatten = nn.Flatten() # Construct a flattening layer.

self.linear_relu_stack = nn.Sequential( # Construct a stack of layers.

nn.Linear(28*28, 512), # Linear Layers have an input and output shape

nn.ReLU(), # ReLU is one of many activation functions provided by nn

nn.Linear(512, 512),

nn.ReLU(),

nn.Linear(512, 10),

)

def forward(self, x): # This function defines the forward pass.

x = self.flatten(x)

logits = self.linear_relu_stack(x)

return logits

References

External links

{{Official website|https://pytorch.org}}

Category:Deep learning software

Category:Facebook software

Category:Free science software

Category:Free software programmed in C

Category:Free software programmed in Python

Category:Open-source artificial intelligence

Category:Python (programming language) scientific libraries

Category:Software using the BSD license