OpenAI Five

Development on the algorithms used for the bots began in November 2016. OpenAI decided to use Dota 2, a competitive five-on-five video game, as a base due to it being popular on the live streaming platform Twitch, having native support for Linux, and had an application programming interface (API) available.{{cite web |author=OpenAI|url=https://openai.com/five/#timeline |title=OpenAI Five |website=openai.com/five |access-date=10 October 2018 |archive-url=https://web.archive.org/web/20180901172612/https://openai.com/five/#timeline |archive-date=1 September 2018 |url-status=live |df=dmy-all}} Before becoming a team of five, the first public demonstration occurred at The International 2017 in August, the annual premiere championship tournament for the game, where Dendi, a Ukrainian professional player, lost against an OpenAI bot in a live one-on-one matchup.{{cite web |last1=Savov |first1=Vlad |title=My favorite game has been invaded by killer AI bots and Elon Musk hype |url=https://www.theverge.com/2017/8/14/16141938/dota-2-openai-bots-elon-musk-artificial-intelligence |website=The Verge |date=14 August 2017 |access-date=25 June 2018 |archive-url=https://web.archive.org/web/20180626030145/https://www.theverge.com/2017/8/14/16141938/dota-2-openai-bots-elon-musk-artificial-intelligence |archive-date=26 June 2018 |url-status=live |df=dmy-all }}{{cite web|last1=Frank|first1=Blair Hanley|title=OpenAI's bot beats top Dota 2 player so badly that he quits|url=https://venturebeat.com/2017/08/11/openais-bot-beats-top-dota-2-player-so-badly-that-he-quits/|website=Venture Beat|access-date=12 August 2017|url-status=dead|archive-url=https://web.archive.org/web/20170812065202/https://venturebeat.com/2017/08/11/openais-bot-beats-top-dota-2-player-so-badly-that-he-quits/|archive-date=12 August 2017}} After the match, CTO Greg Brockman explained that the bot had learned by playing against itself for two weeks of real time, and that the learning software was a step in the direction of creating software that can handle complex tasks "like being a surgeon".{{cite web|author=OpenAI|title=Dota 2|url=https://blog.openai.com/dota-2/|website=blog.openai.com|date=11 August 2017 |access-date=12 August 2017|archive-url=https://web.archive.org/web/20170811235617/https://blog.openai.com/dota-2/|archive-date=11 August 2017|url-status=live|df=dmy-all}}{{cite web|author=OpenAI|title=More on Dota 2|url=https://blog.openai.com/more-on-dota-2/|website=blog.openai.com|date=16 August 2017 |access-date=16 August 2017|archive-url=https://web.archive.org/web/20170816161246/https://blog.openai.com/more-on-dota-2/|archive-date=16 August 2017|url-status=live|df=dmy-all}} OpenAI used a methodology called reinforcement learning, as the bots learn over time by playing against itself hundreds of times a day for months, in which they are rewarded for actions such as killing an enemy and destroying towers.{{cite magazine |last1=Simonite |first1=Tom |date=25 June 2018 |title=Can Bots Outwit Humans in One of the Biggest Esports Games? |url=https://www.wired.com/story/can-bots-outwit-humans-in-one-of-the-biggest-esports-games/ |magazine=Wired |access-date=25 June 2018 |archive-url=https://web.archive.org/web/20180625213810/https://www.wired.com/story/can-bots-outwit-humans-in-one-of-the-biggest-esports-games/ |archive-date=25 June 2018 |url-status=live |df=dmy-all }}{{cite news |last1=Kahn |first1=Jeremy |title=A Bot Backed by Elon Musk Has Made an AI Breakthrough in Video Game World |url=https://www.bloomberg.com/news/articles/2018-06-25/musk-backed-bot-conquers-e-gamer-teams-in-ai-breakthrough |newspaper=Bloomberg.com |date=25 June 2018 |access-date=27 June 2018 |archive-url=https://web.archive.org/web/20180627144300/https://www.bloomberg.com/news/articles/2018-06-25/musk-backed-bot-conquers-e-gamer-teams-in-ai-breakthrough |archive-date=27 June 2018 |url-status=live |df=dmy-all }}

By June 2018, the ability of the bots expanded to play together as a full team of five and were able to defeat teams of amateur and semi-professional players.{{cite web |author=OpenAI |title=OpenAI Five Benchmark |url=https://blog.openai.com/openai-five-benchmark/ |website=blog.openai.com |date=18 July 2018 |access-date=25 August 2018 |archive-url=https://web.archive.org/web/20180826043935/https://blog.openai.com/openai-five-benchmark/ |archive-date=26 August 2018 |url-status=live |df=dmy-all }}{{cite web |last1=Vincent |first1=James |title=AI bots trained for 180 years a day to beat humans at Dota 2 |url=https://www.theverge.com/2018/6/25/17492918/openai-dota-2-bot-ai-five-5v5-matches |website=The Verge |date=25 June 2018 |access-date=25 June 2018 |archive-url=https://web.archive.org/web/20180625183203/https://www.theverge.com/2018/6/25/17492918/openai-dota-2-bot-ai-five-5v5-matches |archive-date=25 June 2018 |url-status=live |df=dmy-all }}{{cite web |last1=Savov |first1=Vlad |title=The OpenAI Dota 2 bots just defeated a team of former pros |url=https://www.theverge.com/2018/8/6/17655086/dota2-openai-bots-professional-gaming-ai |website=The Verge |date=6 August 2018 |access-date=7 August 2018 |archive-url=https://web.archive.org/web/20180807113227/https://www.theverge.com/2018/8/6/17655086/dota2-openai-bots-professional-gaming-ai |archive-date=7 August 2018 |url-status=live |df=dmy-all }} At The International 2018, OpenAI Five played in two games against professional teams, one against the Brazilian-based paiN Gaming and the other against an all-star team of former Chinese players.{{cite magazine |last1=Simonite |first1=Tom |title=Pro Gamers Fend off Elon Musk-Backed AI Bots—for Now |url=https://www.wired.com/story/pro-gamers-fend-off-elon-musks-ai-bots/ |magazine=Wired |access-date=25 August 2018 |archive-url=https://web.archive.org/web/20180824120523/https://www.wired.com/story/pro-gamers-fend-off-elon-musks-ai-bots/ |archive-date=24 August 2018 |url-status=live |df=dmy-all }}{{cite web |last1=Quach |first1=Katyanna |title=Game over, machines: Humans defeat OpenAI bots once again at video games Olympics |url=https://www.theregister.co.uk/2018/08/24/openai_bots_eliminated_dota_2/ |website=The Register |access-date=25 August 2018 |archive-url=https://web.archive.org/web/20180825110329/https://www.theregister.co.uk/2018/08/24/openai_bots_eliminated_dota_2/ |archive-date=25 August 2018 |url-status=live |df=dmy-all }} Although the bots lost both matches, OpenAI still considered it a successful venture, stating that playing against some of the best players in Dota 2 allowed them to analyze and adjust their algorithms for future games.{{cite web |author=OpenAI |title=The International 2018: Results |url=https://blog.openai.com/the-international-2018-results/ |website=blog.openai.com |date=24 August 2018 |access-date=25 August 2018 |archive-url=https://web.archive.org/web/20180824131639/https://blog.openai.com/the-international-2018-results/ |archive-date=24 August 2018 |url-status=live |df=dmy-all }} The bots' final public demonstration occurred in April 2019, where they won a best-of-three series against The International 2018 champions OG at a live event in San Francisco.{{cite web |last1=Wiggers |first1=Kyle |title=OpenAI Five defeats professional Dota 2 team, twice |url=https://venturebeat.com/2019/04/13/openai-five-defeats-a-team-of-professional-dota-2-players/ |website=Venture Beat |date=13 April 2019 |access-date=13 April 2019 |archive-url=https://web.archive.org/web/20190413210513/https://venturebeat.com/2019/04/13/openai-five-defeats-a-team-of-professional-dota-2-players/ |archive-date=13 April 2019 |url-status=live |df=dmy-all }} A four-day online event to play against the bots, open to the public, occurred the same month.{{cite news |last1=Statt |first1=Nick |title=OpenAI's Dota 2 AI steamrolls world champion e-sports team with back-to-back victories |url=https://www.theverge.com/2019/4/13/18309459/openai-five-dota-2-finals-ai-bot-competition-og-e-sports-the-international-champion |url-status=live |access-date=15 April 2019 |work=The Verge |publisher=Vox Media |date=13 April 2019 |archive-url=https://web.archive.org/web/20190415011925/https://www.theverge.com/2019/4/13/18309459/openai-five-dota-2-finals-ai-bot-competition-og-e-sports-the-international-champion |archive-date=15 April 2019 |df=dmy-all}} There, the bots played in 42,729 public games, winning 99.4% of those games.{{cite web |last1=Wiggers |first1=Kyle |title=OpenAI's Dota 2 bot defeated 99.4% of players in public matches |url=https://venturebeat.com/2019/04/22/openais-dota-2-bot-defeated-99-4-of-players-in-public-matches/ |website=Venture Beat |date=22 April 2019 |access-date=22 April 2019}}

Each OpenAI Five bot is a neural network containing a single layer with a 4096-unit{{cite web|url=https://cdn.openai.com/dota-2.pdf|title=Dota 2 with Large Scale Deep Reinforcement Learning|work=OpenAI|access-date=29 September 2024|archive-url=https://web.archive.org/web/20240926092329/https://cdn.openai.com/dota-2.pdf|archive-date=26 September 2024|url-status=live|df=dmy-all}} LSTM that observes the current game state extracted from the Dota developer's API. The neural network conducts actions via numerous possible action heads (no human data involved), and every head has meaning. For instance, the number of ticks to delay an action, what action to select – the X or Y coordinate of this action in a grid around the unit. In addition, action heads are computed independently. The AI system observes the world as a list of 20,000 numbers and takes an action by conducting a list of eight enumeration values. Also, it selects different actions and targets to understand how to encode every action and observe the world.

OpenAI Five has been developed as a general-purpose reinforcement learning training system on the "Rapid" infrastructure. Rapid consists of two layers: it spins up thousands of machines and helps them ‘talk’ to each other and a second layer runs software. By 2018, OpenAI Five had played around 180 years worth of games in reinforcement learning running on 256 GPUs and 128,000 CPU cores,{{cite web|url=https://qz.com/1348177/why-are-ai-researchers-so-obsessed-with-games|title=Why are AI researchers so obsessed with games?|work=QUARTZ|date=4 August 2018 |access-date=4 August 2018|archive-url=https://web.archive.org/web/20180804131643/https://qz.com/1348177/why-are-ai-researchers-so-obsessed-with-games/|archive-date=4 August 2018|url-status=live|df=dmy-all}} using Proximal Policy Optimization, a policy gradient method.{{Cite arXiv|title=Proximal Policy Optimization Algorithms|eprint = 1707.06347|last1 = Schulman|first1 = John|last2 = Wolski|first2 = Filip|last3 = Dhariwal|first3 = Prafulla|last4 = Radford|first4 = Alec|last5 = Klimov|first5 = Oleg|class = cs.LG|year = 2017}}

Prior to OpenAI Five, other AI versus human experiments and systems have been successfully used before, such as Jeopardy! with Watson, chess with Deep Blue, and Go with AlphaGo.{{cite news|url=https://www.theguardian.com/technology/2011/feb/17/ibm-computer-watson-wins-jeopardy|title=IBM computer Watson wins Jeopardy clash|journal=The Guardian|access-date=17 February 2011|date=17 February 2011|last1=Gabbatt|first1=Adam|archive-url=https://web.archive.org/web/20130921060018/http://www.theguardian.com/technology/2011/feb/17/ibm-computer-watson-wins-jeopardy|archive-date=21 September 2013|url-status=live|df=dmy-all}}

{{cite web|url=https://www.businessinsider.com/garry-kasparov-talks-about-artificial-intelligence-2017-12|title=Chess grandmaster Garry Kasparov on what happens when machines 'reach the level that is impossible for humans to compete'|work=Business Insider|access-date=29 December 2017|archive-url=https://web.archive.org/web/20171229122316/http://www.businessinsider.com/garry-kasparov-talks-about-artificial-intelligence-2017-12|archive-date=29 December 2017|url-status=live|df=dmy-all}}

{{cite web|url=https://www.theverge.com/2017/10/18/16495548/deepmind-ai-go-alphago-zero-self-taught|title=DeepMind's Go-playing AI doesn't need human help to beat us anymore|work=Verge|access-date=18 October 2017|date=18 October 2017|archive-url=https://web.archive.org/web/20171018195425/https://www.theverge.com/2017/10/18/16495548/deepmind-ai-go-alphago-zero-self-taught|archive-date=18 October 2017|url-status=live|df=dmy-all}} In comparison with other games that have used AI systems to play against human players, Dota 2 differs as explained below:{{cite web|author=OpenAI|url=https://blog.openai.com/openai-five/|title=OpenAI Five|website=blog.openai.com|access-date=25 June 2018|date=25 June 2018|archive-url=https://web.archive.org/web/20180625140124/https://blog.openai.com/openai-five/|archive-date=25 June 2018|url-status=live|df=dmy-all}}

Long run view: The bots run at 30 frames per second for an average match time of 45 minutes, which results in 80,000 ticks per game. OpenAI Five observes every fourth frame, generating 20,000 moves. By comparison, chess usually ends before 40 moves, while Go ends before 150 moves.

Partially observed state of the game: Players and their allies can only see the map directly around them. The rest of it is covered in a fog of war which hides enemies units and their movements. Thus, playing Dota 2 requires making inferences based on this incomplete data, as well as predicting what their opponent could be doing at the same time. By comparison, Chess and Go are "full-information games", as they do not hide elements from the opposing player.

Continuous action space: Each playable character in a Dota 2 game, known as a hero, can take dozens of actions that target either another unit or a position. The OpenAI Five developers allow the space into 170,000 possible actions per hero. Without counting the perpetual aspects of the game, there are an average of ~1,000 valid actions each tick. By comparison, the average number of actions in chess is 35 and 250 in Go.

Continuous observation space: Dota 2 is played on a large map with ten heroes, five on each team, along with dozens of buildings and non-player character (NPC) units. The OpenAI system observes the state of a game through developers’ bot API, as 20,000 numbers that constitute all information a human is allowed to get access to. A chess board is represented as about 70 lists, whereas a Go board has about 400 enumerations.

OpenAI Five have received acknowledgement from the AI, tech, and video game community at large. Microsoft founder Bill Gates called it a "big deal", as their victories "required teamwork and collaboration".{{cite web|url=https://www.cnbc.com/2018/06/27/bill-gates-openai-robots-beating-humans-at-dota-2-is-ai-milestone.html|title=Bill Gates says gamer bots from Elon Musk-backed nonprofit are 'huge milestone' in A.I.|work=CNBC|access-date=28 June 2018|date=28 June 2018|archive-url=https://web.archive.org/web/20180628231125/https://www.cnbc.com/2018/06/27/bill-gates-openai-robots-beating-humans-at-dota-2-is-ai-milestone.html|archive-date=28 June 2018|url-status=live|df=dmy-all}}{{cite web|url=https://www.businessinsider.com/bill-gates-hails-huge-milestone-for-ai-as-bots-beat-humans-at-dota-2-2018-6|title=Bill Gates hails 'huge milestone' for AI as bots work in a team to destroy humans at video game 'Dota 2'|work=Business Insider|access-date=27 June 2018|archive-url=https://web.archive.org/web/20180627131504/http://www.businessinsider.com/bill-gates-hails-huge-milestone-for-ai-as-bots-beat-humans-at-dota-2-2018-6|archive-date=27 June 2018|url-status=live|df=dmy-all}} Chess champion Garry Kasparov, who lost against the Deep Blue AI in 1997, stated that despite their losing performance at The International 2018, the bots would eventually "get there, and sooner than expected".{{cite web|url=https://twitter.com/Kasparov63/status/1033108573151092736|title=Garry Kasparov's Twitter|access-date=24 August 2018|date=24 August 2018}}

In a conversation with MIT Technology Review, AI experts also considered OpenAI Five system as a significant achievement, as they noted that Dota 2 was an "extremely complicated game", so even beating non-professional players was impressive.{{cite web|last1=Knight|first1=Will|date=25 June 2018|url=https://www.technologyreview.com/s/611536/a-team-of-ai-algorithms-just-crushed-expert-humans-in-a-complex-computer-game|title=A team of AI algorithms just crushed humans in a complex computer game|work= MIT Tech Review|access-date=25 June 2018}} PC Gamer wrote that their wins against professional players was a significant event in machine learning.{{cite news |last1=Park |first1=Morgan |title=How the OpenAI Five tore apart a team of Dota 2 pros |url=https://www.pcgamer.com/how-the-openai-five-tore-apart-a-team-of-dota-2-pros/ |access-date=25 May 2020 |work=PC Gamer |date=11 August 2018}} In contrast, Motherboard wrote that the victory was "basically cheating" due to the simplified hero pools on both sides, as well as the fact that bots were given direct access to the API, as opposed to using computer vision to interpret pixels on the screen.{{cite news |last1=Gault |first1=Matthew |title=OpenAI Is Beating Humans at 'Dota 2' Because It's Basically Cheating |url=https://www.vice.com/en/article/ai-beat-humans-at-dota-2/ |access-date=25 May 2020 |work=Vice |date=17 August 2018 |language=en}} The Verge wrote that the bots were evidence that the company's approach to reinforcement learning and its general philosophy about AI was "yielding milestones".

In 2019, DeepMind unveiled a similar bot for Starcraft II, AlphaStar. Like OpenAI Five, AlphaStar used reinforcement learning and self-play. The Verge reported that "the goal with this type of AI research is not just to crush humans in various games just to prove it can be done. Instead, it’s to prove that — with enough time, effort, and resources — sophisticated AI software can best humans at virtually any competitive cognitive challenge, be it a board game or a modern video game." They added that the DeepMind and OpenAI victories were also a testament to the power of certain uses of reinforcement learning.{{cite news |last1=Statt |first1=Nick |title=DeepMind's StarCraft 2 AI is now better than 99.8 percent of all human players |url=https://www.theverge.com/2019/10/30/20939147/deepmind-google-alphastar-starcraft-2-research-grandmaster-level |access-date=25 May 2020 |work=The Verge |date=30 October 2019 |language=en}}

It was OpenAI's hope that the technology could have applications outside of the digital realm. In 2018, they were able to reuse the same reinforcement learning algorithms and training code from OpenAI Five for Dactyl, a human-like robot hand with a neural network built to manipulate physical objects.{{cite arXiv |author1=OpenAI |last2=Andrychowicz |first2=Marcin |last3=Baker |first3=Bowen |last4=Chociej |first4=Maciek |last5=Józefowicz |first5=Rafał |last6=McGrew |first6=Bob |last7=Pachocki |first7=Jakub |last8=Petron |first8=Arthur |last9=Plappert |first9=Matthias |last10=Powell |first10=Glenn |last11=Ray |first11=Alex |last12=Schneider |first12=Jonas |last13=Sidor |first13=Szymon |last14=Tobin |first14=Josh |last15=Welinder |first15=Peter |last16=Weng |first16=Lilian |last17=Zaremba |first17=Wojciech |date=2019 |title=Learning Dexterous In-Hand Manipulation |eprint=1808.00177v5 |class=cs.LG}} In 2019, Dactyl solved the Rubik's Cube.{{cite arXiv |author1=OpenAI |last2=Akkaya |first2=Ilge |last3=Andrychowicz |first3=Marcin |last4=Chociej |first4=Maciek |last5=Litwin |first5=Mateusz |last6=McGrew |first6=Bob |last7=Petron |first7=Arthur |last8=Paino |first8=Alex |last9=Plappert |first9=Matthias |last10=Powell |first10=Glenn |last11=Ribas |first11=Raphael |date=2019 |title=Solving Rubik's Cube with a Robot Hand |eprint=1910.07113v1 |class=cs.LG}}

{{Reflist}}

{{Portal|Video games}}

• {{Official website|https://openai.com/five/}}
• {{Official website|https://blog.openai.com/openai-five/|Official blog}}

{{Dota}}

{{OpenAI navbox}}

{{Artificial intelligence navbox}}

Category:2017 software

Category:Applied machine learning

Category:Dota

Category:OpenAI