Llama (language model)#LLaMA

After the release of large language models such as GPT-3, a focus of research was up-scaling models which in some instances showed major increases in emergent capabilities.{{cite web |title=Examining Emergent Abilities in Large Language Models |url=https://hai.stanford.edu/news/examining-emergent-abilities-large-language-models |website=hai.stanford.edu |language=en |date=13 September 2022}} The release of ChatGPT and its surprise success caused an increase in attention to large language models.{{cite web |title=The inside story of how ChatGPT was built from the people who made it |url=https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/ |website=MIT Technology Review |language=en |access-date=2024-10-20 |archive-date=2023-03-03 |archive-url=https://web.archive.org/web/20230303093219/https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/ |url-status=live }}

Compared with other responses to ChatGPT, Meta's Chief AI scientist Yann LeCun stated that large language models are best for aiding with writing.{{cite web |title=ChatGPT is 'not particularly innovative,' and 'nothing revolutionary', says Meta's chief AI scientist |url=https://www.zdnet.com/article/chatgpt-is-not-particularly-innovative-and-nothing-revolutionary-says-metas-chief-ai-scientist/ |website=ZDNET |language=en |access-date= |archive-date=2023-02-17 |first = Tiernan|last = Ray|date = 23 January 2023|archive-url=https://web.archive.org/web/20230217163917/https://www.zdnet.com/article/chatgpt-is-not-particularly-innovative-and-nothing-revolutionary-says-metas-chief-ai-scientist/ |url-status=live }}{{cite web |last1=Badminton |first1=Nik |title=Meta's Yann LeCun on auto-regressive Large Language Models (LLMs) |url=https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-large-language-models-llms/ |website=Futurist.com |date=13 February 2023 |access-date=20 October 2024 |archive-date=22 July 2024 |archive-url=https://web.archive.org/web/20240722082109/https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-large-language-models-llms/ |url-status=live }}{{cite web |title=Yann LeCun on LinkedIn: My unwavering opinion on current (auto-regressive) LLMs |url=https://www.linkedin.com/feed/update/urn:li:activity:7030921081876029443/ |url-status=live |archive-url=https://web.archive.org/web/20240917092533/https://www.linkedin.com/feed/update/urn:li:activity:7030921081876029443/ |archive-date=2024-09-17 |access-date=2024-10-20 |website=LinkedIn |language=en}}{{cite web |title=Meta's Yann LeCun Asks How AIs will Match — and Exceed — Human-level Intelligence |date=23 October 2024 |url=https://www.engineering.columbia.edu/about/news/metas-yann-lecun-asks-how-ais-will-match-and-exceed-human-level-intelligence}}

An empirical investigation of the Llama series was the scaling laws. It was observed that the Llama 3 models showed that when a model is trained on data that is more than the "Chinchilla-optimal" amount, the performance continues to scale log-linearly. For example, the Chinchilla-optimal dataset for Llama 3 8B is 200 billion tokens, but performance continued to scale log-linearly to the 75-times larger dataset of 15 trillion tokens.

LLaMA was announced on February 24, 2023, via a blog post and a paper describing the model's training, architecture, and performance. The inference code used to run the model was publicly released under the open-source GPLv3 license. Access to the model's weights was managed by an application process, with access to be granted "on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world".

Llama was trained on only publicly available information, and was trained at various model sizes, with the intention to make it more accessible to different hardware. The model was exclusively a foundation model, although the paper contained examples of instruction fine-tuned versions of the model.

Meta AI reported the 13B parameter model performance on most NLP benchmarks exceeded that of the much larger GPT-3 (with 175B parameters), and the largest 65B model was competitive with state of the art models such as PaLM and Chinchilla.

On March 3, 2023, a torrent containing LLaMA's weights was uploaded, with a link to the torrent shared on the 4chan imageboard and subsequently spread through online AI communities. That same day, a pull request on the main LLaMA repository was opened, requesting to add the magnet link to the official documentation.{{cite news |last1=VK |first1=Anirudh |title=Meta's LLaMA Leaked to the Public, Thanks To 4chan |url=https://analyticsindiamag.com/metas-llama-leaked-to-the-public-thanks-to-4chan/ |access-date=17 March 2023 |work=Analytics India Magazine |date=6 March 2023 |archive-date=26 March 2023 |archive-url=https://web.archive.org/web/20230326020443/https://analyticsindiamag.com/metas-llama-leaked-to-the-public-thanks-to-4chan/ |url-status=live }}{{cite web |title=Save bandwidth by using a torrent to distribute more efficiently by ChristopherKing42 · Pull Request #73 · facebookresearch/llama |url=https://github.com/facebookresearch/llama/pull/73 |website=GitHub |access-date=25 March 2023 |language=en |archive-date=10 April 2023 |archive-url=https://web.archive.org/web/20230410000618/https://github.com/facebookresearch/llama/pull/73 |url-status=live }} On March 4, a pull request was opened to add links to HuggingFace repositories containing the model.{{cite web |title=Download weights from hugging face to help us save bandwidth by Jainam213 · Pull Request #109 · facebookresearch/llama |url=https://github.com/facebookresearch/llama/pull/109 |website=GitHub |access-date=17 March 2023 |language=en |archive-date=21 March 2023 |archive-url=https://web.archive.org/web/20230321172220/https://github.com/facebookresearch/llama/pull/109 |url-status=live }} On March 6, Meta filed takedown requests to remove the HuggingFace repositories linked in the pull request, characterizing it as "unauthorized distribution" of the model. HuggingFace complied with the requests.{{cite news |last1=Cox |first1=Joseph |title=Facebook's Powerful Large Language Model Leaks Online |url=https://www.vice.com/en/article/facebooks-powerful-large-language-model-leaks-online-4chan-llama/ |access-date=17 March 2023 |work=Vice |date=7 March 2023 |language=en |archive-date=6 April 2023 |archive-url=https://web.archive.org/web/20230406135000/https://www.vice.com/en/article/xgwqgw/facebooks-powerful-large-language-model-leaks-online-4chan-llama |url-status=live }} On March 20, Meta filed a DMCA takedown request for copyright infringement against a repository containing a script that downloaded LLaMA from a mirror, and GitHub complied the next day.{{cite web |author1=OpSec Online LLC |title=github/dmca - Notice of Claimed Infringement via Email |url=https://github.com/github/dmca/blob/master/2023/03/2023-03-21-meta.md |publisher=GitHub |access-date=25 March 2023 |date=21 March 2023 |archive-date=10 April 2023 |archive-url=https://web.archive.org/web/20230410032303/https://github.com/github/dmca/blob/master/2023/03/2023-03-21-meta.md |url-status=live }}

Reactions to the leak varied. Some speculated that the model would be used for malicious purposes, such as more sophisticated spam. Some have celebrated the model's accessibility, as well as the fact that smaller versions of the model can be run relatively cheaply, suggesting that this will promote the flourishing of additional research developments. Multiple commentators, such as Simon Willison, compared LLaMA to Stable Diffusion, a text-to-image model which, unlike comparably sophisticated models which preceded it, was openly distributed, leading to a rapid proliferation of associated tools, techniques, and software.

On July 18, 2023, in partnership with Microsoft, Meta announced LLaMa 2, the next generation of Llama. Meta trained and released Llama 2 in three model sizes: 7, 13, and 70 billion parameters.{{cite web |title=Meta and Microsoft Introduce the Next Generation of LLaMA |url=https://about.fb.com/news/2023/07/llama-2/ |website=Meta |access-date=21 July 2023 |date=18 July 2023 |archive-date=14 September 2023 |archive-url=https://web.archive.org/web/20230914132306/https://about.fb.com/news/2023/07/llama-2/ |url-status=live }} The model architecture remains largely unchanged from that of LLaMA-1 models, but 40% more data was used to train the foundational models.{{cite arXiv|last1=Touvron |first1=Hugo|last2=Martin |first2=Louis |title=LLaMA-2: Open Foundation and Fine-Tuned Chat Models|date=18 Jul 2023|eprint=2307.09288|class=cs.CL|display-authors=etal}} The accompanying preprint also mentions a model with 34B parameters that might be released in the future upon satisfying safety targets.

LLaMa 2 includes foundation models and models fine-tuned for chat. In a further departure from the original version of LLaMa, all models are released with weights and may be used for many commercial use cases. However, because LLaMa's license enforces an acceptable use policy that prohibits Llama from being used for some purposes, Meta's use of the term open source to describe Llama has been disputed by the Open Source Initiative (which maintains The Open Source Definition) and others.{{Cite web |last=Edwards |first=Benj |date=2023-07-18 |title=Meta launches LLaMA-2, a source-available AI model that allows commercial applications [Updated] |url=https://arstechnica.com/information-technology/2023/07/meta-launches-llama-2-an-open-source-ai-model-that-allows-commercial-applications/ |access-date=2023-08-08 |website=Ars Technica |language=en-us |archive-date=2023-11-07 |archive-url=https://web.archive.org/web/20231107082612/https://arstechnica.com/information-technology/2023/07/meta-launches-llama-2-an-open-source-ai-model-that-allows-commercial-applications/ |url-status=live }}{{cite web |last1=Thomas |first1=Prasanth Aby |title=Meta offers Llama AI to US government for national security |url=https://www.cio.com/article/3599448/meta-offers-llama-ai-to-us-government-for-national-security.html |website=CIO |access-date=9 December 2024 |language=en |date=5 November 2024}}

Code Llama is a fine-tune of LLaMa 2 with code specific datasets. 7B, 13B, and 34B versions were released on August 24, 2023, with the 70B releasing on the January 29, 2024.{{cite web |title=Introducing Code Llama, a state-of-the-art large language model for coding |url=https://ai.meta.com/blog/code-llama-large-language-model-coding/ |website=ai.meta.com |language=en |access-date=2024-10-20 |archive-date=2024-09-27 |archive-url=https://web.archive.org/web/20240927091138/https://ai.meta.com/blog/code-llama-large-language-model-coding/ |url-status=live }} Starting with the foundation models from LLaMa 2, Meta AI would train an additional 500B tokens of code datasets, before an additional 20B token of long-context data, creating the Code Llama foundation models. This foundation model was further trained on 5B instruction following token to create the instruct fine-tune. Another foundation model was created for Python code, which trained on 100B tokens of Python-only code, before the long-context data.{{cite arXiv |last1=Rozière |first1=Baptiste |title=Code Llama: Open Foundation Models for Code |date=2024-01-31 |eprint=2308.12950 |last2=Gehring |first2=Jonas |last3=Gloeckle |first3=Fabian |last4=Sootla |first4=Sten |last5=Gat |first5=Itai |last6=Tan |first6=Xiaoqing Ellen |last7=Adi |first7=Yossi |last8=Liu |first8=Jingyu |last9=Sauvestre |first9=Romain|class=cs.CL }}

File:A Representation of Meta AI and Llama (Meta AI Imagine).webp

On April 18, 2024, Meta released Llama-3 with two sizes: 8B and 70B parameters.{{Cite web |date=April 18, 2024 |title=Introducing Meta Llama 3: The most capable openly available LLM to date |url=https://ai.meta.com/blog/meta-llama-3/ |access-date=2024-04-21 |website=ai.meta.com |language=en |archive-date=2024-05-15 |archive-url=https://web.archive.org/web/20240515023523/https://ai.meta.com/blog/meta-llama-3/ |url-status=live }} The models have been pre-trained on approximately 15 trillion tokens of text gathered from “publicly available sources” with the instruct models fine-tuned on “publicly available instruction datasets, as well as over 10M human-annotated examples". Meta AI's testing showed in April 2024 that Llama 3 70B was beating Gemini Pro 1.5 and Claude 3 Sonnet on most benchmarks. Meta also announced plans to make Llama 3 multilingual and multimodal, better at coding and reasoning, and to increase its context window.{{cite web |last1=Wiggers |first1=Kyle |date=18 April 2024 |title=Meta releases Llama 3, claims it's among the best open models available |url=https://techcrunch.com/2024/04/18/meta-releases-llama-3-claims-its-among-the-best-open-models-available/ |website=TechCrunch |access-date=20 October 2024 |archive-date=18 September 2024 |archive-url=https://web.archive.org/web/20240918202013/https://techcrunch.com/2024/04/18/meta-releases-llama-3-claims-its-among-the-best-open-models-available/ |url-status=live }}{{cite web |last1=Mann |first1=Tobias |date=April 19, 2024 |title=Meta debuts third-generation Llama large language model |url=https://www.theregister.com/2024/04/19/meta_debuts_llama3_llm/ |website=The Register |language=en |access-date=October 20, 2024 |archive-date=August 25, 2024 |archive-url=https://web.archive.org/web/20240825145130/https://www.theregister.com/2024/04/19/meta_debuts_llama3_llm/ |url-status=live }}

During an interview with Dwarkesh Patel, Mark Zuckerberg said that the 8B version of Llama 3 was nearly as powerful as the largest Llama 2. Compared to previous models, Zuckerberg stated the team was surprised that the 70B model was still learning even at the end of the 15T tokens training. The decision was made to end training to focus GPU power elsewhere.{{Cite web |last=Patel |first=Dwarkesh |date=2024-07-24 |title=Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus |url=https://www.dwarkeshpatel.com/p/mark-zuckerberg |access-date=2024-08-01 |website=www.dwarkeshpatel.com |language=en |quote=the 8 billion is nearly as powerful as the biggest version of Llama 2 that we released [...] even by the end, it was... still learning right it's like we probably could have fed it more tokens and it would have gotten somewhat better but i mean at some point you know you're running a company you need to do these meta reasoning questions of [...] how do I want to spend our GPUs |archive-date=2024-07-16 |archive-url=https://web.archive.org/web/20240716152236/https://www.dwarkeshpatel.com/p/mark-zuckerberg |url-status=live }}

Llama-3.1 was released on July 23, 2024, with three sizes: 8B, 70B, and 405B parameters.{{Cite web |date=July 23, 2024 |title=Introducing Llama 3.1: Our most capable models to date |url=https://ai.meta.com/blog/meta-llama-3-1/ |url-status=live |archive-url=https://web.archive.org/web/20240723153909/https://ai.meta.com/blog/meta-llama-3-1/ |archive-date=2024-07-23 |access-date=2024-07-23 |website=ai.meta.com |language=en}}{{Citation |last1=Dubey |first1=Abhimanyu |title=The Llama 3 Herd of Models |date=2024-07-31 |arxiv=2407.21783 |last2=Jauhri |first2=Abhinav |last3=Pandey |first3=Abhinav |last4=Kadian |first4=Abhishek |last5=Al-Dahle |first5=Ahmad |last6=Letman |first6=Aiesha |last7=Mathur |first7=Akhil |last8=Schelten |first8=Alan |last9=Yang |first9=Amy}}

The Llama-4 series was released in 2025. The architecture was changed to a mixture of experts. They are multimodal (text and image input, text output) and multilingual (12 languages). Specifically, on 5 April 2025, the following were released both as base and instruction-tuned versions:{{Cite web |title=The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation |url=https://ai.meta.com/blog/llama-4-multimodal-intelligence/ |archive-url=http://web.archive.org/web/20250405185132/https://ai.meta.com/blog/llama-4-multimodal-intelligence/ |archive-date=2025-04-05 |access-date=2025-04-05 |website=ai.meta.com |language=en}}

• Scout: 17 billion active parameter model with 16 experts, context window of 10M, with 109B parameters in total.
• Maverick: 17 billion active parameter model with 128 experts, context window of 1M, with 400B parameters in total.

Also claimed was Behemoth (not yet released): 288 billion active parameter model with 16 experts and around 2T parameters in total. The Behemoth version was still in training at that time. The Scout was trained from scratch. The Maverick was "codistilled" from Behemoth. Note that the Scout was trained for longer and had a longer context length than Maverick.

The training data included publicly available data, licensed data, and Meta-proprietary data such as publicly shared posts from Instagram and Facebook and people’s interactions with Meta AI. The data cutoff was August 2024.

Meta claimed in its release announcement that Llama 4 bested GPT-4o's score on the LMArena AI benchmark.{{cite web |last1=Robison |first1=Kylie |title=Meta got caught gaming AI benchmarks |url=https://www.theverge.com/meta/645012/meta-llama-4-maverick-benchmarks-gaming |website=The Verge |access-date=8 April 2025 |date=8 April 2025}} The company also stated that Llama 4's benchmark score was achieved using an unreleased "experimental chat version" of the model that was "optimized for conversationality", which differed from the version of Llama 4 released to the public.{{cite web |last1=Wiggers |first1=Kyle |title=Meta's benchmarks for its new AI models are a bit misleading |url=https://techcrunch.com/2025/04/06/metas-benchmarks-for-its-new-ai-models-are-a-bit-misleading/ |website=TechCrunch |access-date=8 April 2025 |date=6 April 2025}} LMArena indicated that it would change its policies to prevent this incident from reoccurring, and responded, "Meta's interpretation of our policy did not match what we expect from model providers. Meta should have made it clearer that 'Llama-4-Maverick-03-26-Experimental' was a customized model to optimize for human preference." Some users criticized Meta on social media for its use of a separate model version tailored for benchmarking, and some additionally accused Meta of training Llama 4 on test sets to further boost its benchmark scores—which Meta denied.{{cite web |last1=Franzen |first1=Carl |title=Meta defends Llama 4 release against 'reports of mixed quality,' blames bugs |url=https://venturebeat.com/ai/meta-defends-llama-4-release-against-reports-of-mixed-quality-blames-bugs/ |website=VentureBeat |access-date=8 April 2025 |date=8 April 2025}}

For the training cost column, only the largest model's cost is written by default. So for example, "21,000" is the training cost of Llama 2 69B in units of petaFLOP-day. Also, 1 petaFLOP-day = 1 petaFLOP/sec × 1 day = 8.64E19 FLOP. "T" means "trillion" and "B" means "billion".

The following table lists the main model versions of Llama, describing the significant changes included with each version:{{Cite web |title=Llama Models |url=https://www.llama.com/docs/model-cards-and-prompt-formats/ |url-status=live |archive-url=https://web.archive.org/web/20250409095851/https://www.llama.com/docs/model-cards-and-prompt-formats/ |archive-date=April 9, 2025 |access-date=April 20, 2025 |website=www.llama.com}}

{{Quote box

| title = "recommendation letter for the Magic Unicorn Corporation"

| quote = Here is the recommendation letter that I wrote for an application to a dragon feeder position at the Magic Unicorn Corporation:

Dear recruiter,

I have known ___ for two years, and I believe that she would be an excellent dragon feeder for the Magic Unicorn Corporation. ___ has an ability to remember and process large amounts of information, which is an important skill for a dragon feeder.

___, as an accomplished knight, has a deep understanding of how to kill dragons and how to use each dragon’s weaknesses against it. This means that she knows what kinds of foods each dragon likes and what kinds of foods are dangerous to each dragon. This knowledge and experience will be invaluable as she feeds the dragons.

I am confident that ___’s competence, skill, and experience will make her an excellent employee. Please contact me at (___) ___-___ if you have any questions. I look forward to hearing from you.

Best regards,

Honorable Knight

Sir George

| source = – Output of 65 billion parameter LLaMA model before instruction tuning, given the prompt (in bold)

| align = right

| width = 350px

}}

Like GPT-3, the Llama series of models are autoregressive decoder-only Transformers, but there are some minor differences:

• SwiGLU{{Cite arXiv |eprint=2002.05202 |class=cs.CL |first=Noam |last=Shazeer |title=GLU Variants Improve Transformer |date=2020-02-01}} activation function instead of GeLU;
• rotary positional embeddings (RoPE){{Cite arXiv |last1=Su |first1=Jianlin |last2=Lu |first2=Yu |last3=Pan |first3=Shengfeng |last4=Murtadha |first4=Ahmed |last5=Wen |first5=Bo |last6=Liu |first6=Yunfeng |date=2021-04-01 |title=RoFormer: Enhanced Transformer with Rotary Position Embedding |class=cs.CL |eprint=2104.09864}} instead of absolute positional embedding;
• RMSNorm{{Cite arXiv|last1=Zhang |first1=Biao |last2=Sennrich |first2=Rico |date=2019-10-01 |title=Root Mean Square Layer Normalization |class=cs.LG |eprint=1910.07467}} instead of layer normalization;{{Cite arXiv|last1=Lei Ba |first1=Jimmy |last2=Kiros |first2=Jamie Ryan |last3=Hinton |first3=Geoffrey E. |date=2016-07-01 |title=Layer Normalization |class=stat.ML |eprint=1607.06450}}

LLaMA's developers focused their effort on scaling the model's performance by increasing the volume of training data, rather than the number of parameters, reasoning that the dominating cost for LLMs is from doing inference on the trained model rather than the computational cost of the training process.

LLaMA 1 foundational models were trained on a data set with 1.4 trillion tokens, drawn from publicly available data sources, including:

• Webpages scraped by CommonCrawl
• Open source repositories of source code from GitHub
• Wikipedia in 20 languages
• Public domain books from Project Gutenberg
• Books3 books dataset
• The LaTeX source code for scientific papers uploaded to ArXiv
• Questions and answers from Stack Exchange websites

On April 17, 2023, TogetherAI launched a project named RedPajama to reproduce and distribute an open source version of the LLaMA dataset. The dataset has approximately 1.2 trillion tokens and is publicly available for download.

Llama 2 foundational models were trained on a data set with 2 trillion tokens. This data set was curated to remove Web sites that often disclose personal data of people. It also upsamples sources considered trustworthy. Llama 2 - Chat was additionally fine-tuned on 27,540 prompt-response pairs created for this project, which performed better than larger but lower-quality third-party datasets. For AI alignment, reinforcement learning with human feedback (RLHF) was used with a combination of 1,418,091 Meta examples and seven smaller datasets. The average dialog depth was 3.9 in the Meta examples, 3.0 for Anthropic Helpful and Anthropic Harmless sets, and 1.0 for five other sets, including OpenAI Summarize, StackExchange, etc.

Llama 3 consists of mainly English data, with over 5% in over 30 other languages. Its dataset was filtered by a text-quality classifier, and the classifier was trained by text synthesized by Llama 2.

In a lawsuit brought by Richard Kadrey and others against Meta Platforms, CEO Mark Zuckerberg was alleged to have authorized the use of copyrighted content from Library Genesis to train Llama AI models and conceal its actions by removing copyright markers from the data.{{Cite web |last=Wiggers |first=Kyle |date=January 9, 2025 |title=Mark Zuckerberg gave Meta's Llama team the OK to train on copyrighted works, filing claims |url=https://techcrunch.com/2025/01/09/mark-zuckerberg-gave-metas-llama-team-the-ok-to-train-on-copyrighted-works-filing-claims/ |access-date=January 12, 2025 |website=Techcrunch}}

Llama 1 models are only available as foundational models with self-supervised learning and without fine-tuning. Llama 2 – Chat models were derived from foundational Llama 2 models. Unlike GPT-4 which increased context length during fine-tuning, Llama 2 and Code Llama - Chat have the same context length of 4K tokens. Supervised fine-tuning used an autoregressive loss function with token loss on user prompts zeroed out. The batch size was 64.

For AI alignment, human annotators wrote prompts and then compared two model outputs (a binary protocol), giving confidence levels and separate safety labels with veto power. Two separate reward models were trained from these preferences for safety and helpfulness using Reinforcement learning from human feedback (RLHF). A major technical contribution is the departure from the exclusive use of Proximal Policy Optimization (PPO) for RLHF – a new technique based on Rejection sampling was used, followed by PPO.

Multi-turn consistency in dialogs was targeted for improvement, to make sure that "system messages" (initial instructions, such as "speak in French" and "act like Napoleon") are respected during the dialog. This was accomplished using the new "Ghost attention" technique during training, which concatenates relevant instructions to each new user message but zeros out the loss function for tokens in the prompt (earlier parts of the dialog).

The Stanford University Institute for Human-Centered Artificial Intelligence (HAI) Center for Research on Foundation Models (CRFM) released Alpaca, a training recipe based on the LLaMA 7B model that uses the "Self-Instruct" method of instruction tuning to acquire capabilities comparable to the OpenAI GPT-3 series text-davinci-003 model at a modest cost.{{cite web |url=https://crfm.stanford.edu/2023/03/13/alpaca.html |title=Alpaca: A Strong, Replicable Instruction-Following Model |date=13 March 2023 |first1=Rohan |last1=Taori |first2=Ishaan |last2=Gulrajani |first3=Tianyi |last3=Zhang |first4=Yann |last4=Dubois |first5=Xuechen |last5=Li |first6=Carlos |last6=Guestrin |first7=Percy |last7=Liang |first8=Tatsunori B. |last8=Hashimoto |website= |publisher=Stanford Center for Research on Foundation Models |access-date= |archive-date=6 April 2023 |archive-url=https://web.archive.org/web/20230406082332/https://crfm.stanford.edu/2023/03/13/alpaca.html |url-status=live }}{{cite arXiv | eprint=2212.10560 | last1=Wang | first1=Yizhong | last2=Kordi | first2=Yeganeh | last3=Mishra | first3=Swaroop | last4=Liu | first4=Alisa | last5=Smith | first5=Noah A. | last6=Khashabi | first6=Daniel | last7=Hajishirzi | first7=Hannaneh | title=Self-Instruct: Aligning Language Models with Self-Generated Instructions | year=2022 | class=cs.CL }}{{cite web |title=Stanford CRFM |url=https://crfm.stanford.edu/2023/03/13/alpaca.html |website=crfm.stanford.edu |access-date=2023-03-20 |archive-date=2023-04-06 |archive-url=https://web.archive.org/web/20230406082332/https://crfm.stanford.edu/2023/03/13/alpaca.html |url-status=live }} The model files were officially removed on March 21, 2023, over hosting costs and safety concerns, though the code and paper remain online for reference.{{cite web |last1=Quach |first1=Katyanna |title=Stanford takes costly, risky Alpaca AI model offline |url=https://www.theregister.com/2023/03/21/stanford_ai_alpaca_taken_offline/ |website=www.theregister.com |language=en}}{{cite web |title=Stanford Researchers Take Down Alpaca AI Over Cost and Hallucinations |url=https://gizmodo.com/stanford-ai-alpaca-llama-facebook-taken-down-chatgpt-1850247570 |website=Gizmodo |language=en |date=21 March 2023 |access-date=20 October 2024 |archive-date=12 May 2024 |archive-url=https://web.archive.org/web/20240512075506/https://gizmodo.com/stanford-ai-alpaca-llama-facebook-taken-down-chatgpt-1850247570 |url-status=live }}

Meditron is a family of Llama-based finetuned on a corpus of clinical guidelines, PubMed papers, and articles. It was created by researchers at École Polytechnique Fédérale de Lausanne School of Computer and Communication Sciences, and the Yale School of Medicine. It shows increased performance on medical-related benchmarks such as MedQA and MedMCQA.{{cite web |title=Meditron: An LLM suite for low-resource medical settings leveraging Meta Llama |url=https://ai.meta.com/blog/llama-2-3-meditron-yale-medicine-epfl-open-source-llm/ |website=ai.meta.com |language=en}}{{cite web |last1=Petersen |first1=Tanya |title=EPFL's new Large Language Model for Medical Knowledge |url=https://actu.epfl.ch/news/epfl-s-new-large-language-model-for-medical-knowle/ |language=en |date=28 November 2023 |access-date=20 October 2024 |archive-date=17 September 2024 |archive-url=https://web.archive.org/web/20240917180520/https://actu.epfl.ch/news/epfl-s-new-large-language-model-for-medical-knowle/ |url-status=live }}{{cite web |title=epfLLM/meditron |url=https://github.com/epfLLM/meditron |publisher=epfLLM |date=11 May 2024 |access-date=20 October 2024 |archive-date=27 September 2024 |archive-url=https://web.archive.org/web/20240927092256/https://github.com/epfLLM/meditron |url-status=live }}

Zoom used Meta Llama 2 to create an AI Companion that can summarize meetings, provide helpful presentation tips, and assist with message responses. This AI Companion is powered by multiple models, including Meta Llama 2.{{cite web |title=How Companies Are Using Meta Llama |url=https://about.fb.com/news/2024/05/how-companies-are-using-meta-llama/ |website=Meta |date=7 May 2024 |access-date=20 October 2024 |archive-date=27 September 2024 |archive-url=https://web.archive.org/web/20240927181724/https://about.fb.com/news/2024/05/how-companies-are-using-meta-llama/ |url-status=live }}

Reuters reported in 2024 that many Chinese foundation models relied on Llama models for their training.{{Cite news |date=May 9, 2024 |title=How dependent is China on US artificial intelligence technology? |url=https://www.reuters.com/technology/how-dependent-is-china-us-artificial-intelligence-technology-2024-05-09/ |work=Reuters}}

{{Main|llama.cpp}}

Software developer Georgi Gerganov released llama.cpp as open-source on March 10, 2023. It's a re-implementation of LLaMA in C++, allowing systems without a powerful GPU to run the model locally.{{Cite web |last=Edwards |first=Benj |date=2023-03-13 |title=You can now run a GPT-3-level AI model on your laptop, phone, and Raspberry Pi |url=https://arstechnica.com/information-technology/2023/03/you-can-now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-raspberry-pi/ |access-date=2024-01-04 |website=Ars Technica |language=en-us |archive-date=2024-01-09 |archive-url=https://web.archive.org/web/20240109194611/https://arstechnica.com/information-technology/2023/03/you-can-now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-raspberry-pi/ |url-status=live }} The llama.cpp project introduced the GGUF file format, a binary format that stores both tensors and metadata.{{cite web |title=GGUF |url=https://huggingface.co/docs/hub/gguf |website=huggingface.co |access-date=9 May 2024}} The format focuses on supporting different quantization types, which can reduce memory usage, and increase speed at the expense of lower model precision.{{cite web |last1=Labonne |first1=Maxime |title=Quantize Llama models with GGUF and llama.cpp |url=https://towardsdatascience.com/quantize-llama-models-with-ggml-and-llama-cpp-3612dfbcc172 |website=Medium |publisher=Towards Data Science |access-date=9 May 2024 |language=en |date=29 November 2023 |archive-date=9 May 2024 |archive-url=https://web.archive.org/web/20240509081605/https://towardsdatascience.com/quantize-llama-models-with-ggml-and-llama-cpp-3612dfbcc172 |url-status=live }}

llamafile created by Justine Tunney is an open-source tool that bundles llama.cpp with the model into a single executable file. Tunney et al. introduced new optimized matrix multiplication kernels for x86 and ARM CPUs, improving prompt evaluation performance for FP16 and 8-bit quantized data types.{{cite web |last1=Connatser |first1=Matthew |title=Llamafile LLM driver project boosts performance on CPU cores |url=https://www.theregister.com/2024/04/03/llamafile_performance_gains/ |website=www.theregister.com |access-date=10 May 2024 |language=en |archive-date=10 May 2024 |archive-url=https://web.archive.org/web/20240510232003/https://www.theregister.com/2024/04/03/llamafile_performance_gains/ |url-status=live }}

In 2024, researchers from the People's Liberation Army Academy of Military Sciences (top military academy of China) were reported to have developed a military tool using Llama, which Meta Platforms stated was unauthorized due to Llama's license prohibiting the use of the model for military purposes.{{Cite web |last=Cheung |first=Sunny |date=October 31, 2024 |title=PRC Adapts Meta's Llama for Military and Security AI Applications |url=https://jamestown.org/program/prcs-adaptation-of-open-source-llm-for-military-and-security-purposes/ |access-date=2024-11-03 |website=Jamestown Foundation |language=en-US}}{{Cite news |last1=Pomfret |first1=James |last2=Pang |first2=Jessie |date=November 1, 2024 |title=Chinese researchers develop AI model for military use on back of Meta's Llama |url=https://www.reuters.com/technology/artificial-intelligence/chinese-researchers-develop-ai-model-military-use-back-metas-llama-2024-11-01/ |access-date=November 1, 2024 |work=Reuters}} Meta granted the US government and US military contractors permission to use Llama in November 2024, but continued to prohibit military use by non-US entities.{{cite web |last1=Smith |first1=Matthew S. |title=Meta Opens Its AI Model for the U.S. Military - IEEE Spectrum |url=https://spectrum.ieee.org/ai-used-by-military |website=IEEE Spectrum |access-date=9 December 2024 |language=en |date=17 November 2024}}

Wired describes the 8B parameter version of Llama 3 as being "surprisingly capable" given its size.{{cite magazine |last1=Knight |first1=Will |title=Meta's Open Source Llama 3 Is Already Nipping at OpenAI's Heels |url=https://www.wired.com/story/metas-open-source-llama-3-nipping-at-openais-heels/ |magazine=Wired |access-date=2024-10-20 |archive-date=2024-09-27 |archive-url=https://web.archive.org/web/20240927073830/https://www.wired.com/story/metas-open-source-llama-3-nipping-at-openais-heels/ |url-status=live }}

The response to Meta's integration of Llama into Facebook was mixed, with some users confused after Meta AI told a parental group that it had a child.{{cite web |title=Meta's amped-up AI agents confusing Facebook users |url=https://www.abc.net.au/news/2024-04-19/meta-releases-llama-3-ai-model/103744538 |website=ABC News |language=en-AU |date=19 April 2024 |access-date=2024-10-20 |archive-date=2024-09-17 |archive-url=https://web.archive.org/web/20240917102930/https://www.abc.net.au/news/2024-04-19/meta-releases-llama-3-ai-model/103744538 |url-status=live }}

According to the Q4 2023 Earnings transcript, Meta adopted the strategy of open weights to improve on model safety, iteration speed, increase adoption among developers and researchers, and to become the industry standard. Llama 5, 6, and 7 are planned for the future.{{Cite web |url=https://s21.q4cdn.com/399680738/files/doc_financials/2023/q4/META-Q4-2023-Earnings-Call-Transcript.pdf |title=Archived copy |access-date=2024-10-20 |archive-date=2024-09-17 |archive-url=https://web.archive.org/web/20240917115531/https://s21.q4cdn.com/399680738/files/doc_financials/2023/q4/META-Q4-2023-Earnings-Call-Transcript.pdf |url-status=live }}

The release of Llama models has sparked significant debates on the benefits and misuse risks of open weight models. Such models can be fine-tuned to remove safeguards, notably by cyber criminals, until they comply with harmful requests. Some experts contend that future models may facilitate causing damage more than defending against it, for example by making it relatively easy to engineer advanced bioweapons without specialized knowledge. Conversely, open-weight models can be useful for a wide variety of purposes, including for safety research.{{Cite magazine |last=Knight |first=Will |title=Meta's New Llama 3.1 AI Model Is Free, Powerful, and Risky |url=https://www.wired.com/story/meta-ai-llama-3/ |access-date=2024-08-04 |magazine=Wired |language=en-US |issn=1059-1028 |archive-date=2024-08-03 |archive-url=https://web.archive.org/web/20240803201314/https://www.wired.com/story/meta-ai-llama-3/ |url-status=live }}

Open Source Initiative head Stefano Maffulli criticized Meta for describing Llama as open source, saying that it was causing confusion among users and "polluting" the term.{{Cite news|url=https://www.ft.com/content/397c50d8-8796-4042-a814-0ac2c068361f|title=Meta under fire for 'polluting' open-source|last=Waters|first=Richard|date=October 17, 2024|work=Financial Times}}

• GPT-4o
• IBM Granite, an open-source LLM made by IBM
• Mistral AI, a French open-source AI company

{{reflist|refs=

{{cite web

|work=Meta AI

|title=Introducing LLaMA: A foundational, 65-billion-parameter large language model

|date=24 February 2023

|url=https://ai.facebook.com/blog/large-language-model-llama-meta-ai/

|access-date=16 March 2023

|archive-date=3 March 2023

|archive-url=https://web.archive.org/web/20230303112302/https://ai.facebook.com/blog/large-language-model-llama-meta-ai/

|url-status=live

}}

{{cite arXiv

|eprint=2302.13971

|last1=Touvron

|first1=Hugo

|last2=Lavril

|first2=Thibaut

|last3=Izacard

|first3=Gautier

|last4=Martinet

|first4=Xavier

|last5=Lachaux

|first5=Marie-Anne

|last6=Lacroix

|first6=Timothée

|last7=Rozière

|first7=Baptiste

|last8=Goyal

|first8=Naman

|last9=Hambro

|first9=Eric

|last10=Azhar

|first10=Faisal

|last11=Rodriguez

|first11=Aurelien

|last12=Joulin

|first12=Armand

|last13=Grave

|first13=Edouard

|last14=Lample

|first14=Guillaume

|title=LLaMA: Open and Efficient Foundation Language Models

|year=2023

|class=cs.CL

}}

{{cite web

|work=The Verge

|title=Meta's powerful AI language model has leaked online — what happens now?

|last=Vincent

|first=James

|date=8 March 2023

|url=https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse

|access-date=16 March 2023

|archive-date=3 November 2023

|archive-url=https://web.archive.org/web/20231103161046/https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse

|url-status=live

}}

{{cite web

|title=llama

|work=GitHub

|access-date=16 March 2023

|url=https://github.com/facebookresearch/llama

|archive-date=15 March 2023

|archive-url=https://web.archive.org/web/20230315183955/https://github.com/facebookresearch/llama/

|url-status=live

}}

{{cite web

|work=Simon Willison's Weblog

|last=Willison

|first=Simon

|title=Large language models are having their Stable Diffusion moment

|date=11 March 2023

|url=https://simonwillison.net/2023/Mar/11/llama/

|access-date=16 March 2023

|archive-date=16 March 2023

|archive-url=https://web.archive.org/web/20230316201253/https://simonwillison.net/2023/Mar/11/llama/

|url-status=live

}}

{{cite web

|title=alpaca-lora

|work=GitHub

|access-date=5 April 2023

|url=https://github.com/tloen/alpaca-lora

|archive-date=4 April 2023

|archive-url=https://web.archive.org/web/20230404210345/https://github.com/tloen/alpaca-lora

|url-status=live

}}

{{cite web

|title=RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training dataset

|url=https://github.com/togethercomputer/RedPajama-Data

|website=GitHub

|publisher=Together

|access-date=4 May 2023

|archive-date=7 November 2023

|archive-url=https://web.archive.org/web/20231107223503/https://github.com/togethercomputer/RedPajama-Data

|url-status=live

}}

{{cite web

|title=RedPajama-Data-1T

|url=https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T

|website=Hugging Face

|publisher=Together

|access-date=4 May 2023

|archive-date=3 November 2023

|archive-url=https://web.archive.org/web/20231103013716/https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T

|url-status=live

}}

{{cite web

|last1=Peters

|first1=Jay

|last2=Vincent

|first2=James

|title=Meta has a new machine learning language model to remind you it does AI too

|url=https://www.theverge.com/2023/2/24/23613512/meta-llama-ai-research-large-language-model

|website=The Verge

|language=en

|date=24 February 2023}}

}}

{{refbegin}}

• {{Cite web |last1=Huang |first1=Kalley |last2=O'Regan |first2=Sylvia Varnham |date=September 5, 2023 |title=Inside Meta's AI Drama: Internal Feuds Over Compute Power |url=https://www.theinformation.com/articles/inside-metas-ai-drama-internal-feuds-over-compute-power |url-access=limited |url-status=live |archive-url=https://web.archive.org/web/20230905174145/https://www.theinformation.com/articles/inside-metas-ai-drama-internal-feuds-over-compute-power |archive-date=September 5, 2023 |access-date=September 6, 2023 |website=The Information}}

{{refend}}

• {{Official website}}
• {{Official website|https://huggingface.co/meta-llama|name=Official Hugging Face organization for Llama, Llama Guard, and Prompt Guard models}}

{{Generative AI}}

{{Artificial intelligence navbox}}

Category:2023 software

Category:Internet leaks

Category:Large language models

Category:Meta Platforms