Common Crawl

Amazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.{{cite news |title=Common Crawl to Add New Data in Amazon Web Services Bucket |author=Jennifer Zaino |publisher=Semantic Web |date=March 13, 2012 |access-date=July 31, 2014 |url=http://semanticweb.com/common-crawl-to-add-new-data-in-amazon-web-services-bucket_b27341 |archive-url=https://web.archive.org/web/20140701235708/http://semanticweb.com/common-crawl-to-add-new-data-in-amazon-web-services-bucket_b27341 |archive-date=July 1, 2014 |url-status=dead}}

The organization began releasing metadata files and the text output of the crawlers alongside .arc files in July 2012.{{cite news |author=Jennifer Zaino |date=July 16, 2012 |title=Common Crawl Corpus Update Makes Web Crawl Data More Efficient, Approachable for Users to Explore |url=http://semanticweb.com/common-crawl-corpus-update-makes-web-crawl-data-more-efficient-approachable-for-users-to-explore_b30771 |archive-url=https://web.archive.org/web/20140812101154/http://semanticweb.com/common-crawl-corpus-update-makes-web-crawl-data-more-efficient-approachable-for-users-to-explore_b30771 |archive-date=August 12, 2014 |url-status=dead |publisher=Semantic Web |access-date=July 31, 2014}} Common Crawl's archives had only included .arc files previously.

In December 2012, blekko donated to Common Crawl search engine metadata blekko had gathered from crawls it conducted from February to October 2012.{{cite news |author=Jennifer Zaino |date=December 18, 2012 |title=Blekko Data Donation Is s Big Benefit to Common Crawl |url=http://semanticweb.com/blekko-data-donation-is-a-big-benefit-to-common-crawl_b34177 |archive-url=https://web.archive.org/web/20140812101151/http://semanticweb.com/blekko-data-donation-is-a-big-benefit-to-common-crawl_b34177 |archive-date=August 12, 2014 |url-status=dead |publisher=Semantic Web |access-date=July 31, 2014}} The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."

In 2013, Common Crawl began using the Apache Software Foundation's Nutch webcrawler instead of a custom crawler.{{cite web |author=Jordan Mendelson |date=February 20, 2014 |title=Common Crawl's Move to Nutch |url=http://commoncrawl.org/common-crawl-move-to-nutch/ |publisher=Common Crawl |access-date=July 31, 2014}} Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.{{cite web |author=Jordan Mendelson |date=November 27, 2013 |title=New Crawl Data Available! |url=http://commoncrawl.org/new-crawl-data-available/ |publisher=Common Crawl |access-date=July 31, 2014}}

A filtered version of Common Crawl was used to train OpenAI's GPT-3 language model, announced in 2020.{{Cite arXiv |last1=Brown |first1=Tom |last2=Mann |first2=Benjamin |last3=Ryder |first3=Nick |last4=Subbiah |first4=Melanie |last5=Kaplan |first5=Jared |last6=Dhariwal |first6=Prafulla |last7=Neelakantan |first7=Arvind |last8=Shyam |first8=Pranav |last9=Sastry |first9=Girish |last10=Askell |first10=Amanda |last11=Agarwal |first11=Sandhini |date=2020-06-01 |title=Language Models Are Few-Shot Learners |page=14 |eprint=2005.14165 |class=cs.CL |quote=the majority of our data is derived from raw Common Crawl with only quality-based filtering.}}

The following data have been collected from the official Common Crawl Blog{{Cite web|url=https://commoncrawl.org/blog/|title=Blog – Common Crawl}}

and Common Crawl's API.{{Cite web|url=https://index.commoncrawl.org/collinfo.json|title=Collection info - Common Crawl}}

In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux.{{cite web |title=The Norvig Web Data Science Award|author=Lisa Green|publisher=Common Crawl|date=November 15, 2012|access-date=July 31, 2014|url=http://commoncrawl.org/the-norvig-web-data-science-award/}}{{cite web|title=Norvig Web Data Science Award 2014|publisher=Dutch Techcentre for Life Sciences|access-date=July 31, 2014|url=http://www.dtls.nl/dtl/news/norvig-web-data-science-award-2014.html|archive-url=https://web.archive.org/web/20140815035946/http://www.dtls.nl/dtl/news/norvig-web-data-science-award-2014.html|archive-date=August 15, 2014|url-status=dead}} The award is named for Peter Norvig who also chairs the judging committee for the award.

{{Anchor|Colossal Clean Crawled Corpus}}

Google's version of the Common Crawl is called the Colossal Clean Crawled Corpus, or C4 for short. It was constructed for the training of the T5 language model series in 2019.{{Cite journal |last1=Raffel |first1=Colin |last2=Shazeer |first2=Noam |last3=Roberts |first3=Adam |last4=Lee |first4=Katherine |last5=Narang |first5=Sharan |last6=Matena |first6=Michael |last7=Zhou |first7=Yanqi |last8=Li |first8=Wei |last9=Liu |first9=Peter J. |date=2020 |title=Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |url=http://jmlr.org/papers/v21/20-074.html |journal=Journal of Machine Learning Research |volume=21 |issue=140 |pages=1–67 |arxiv=1910.10683 |issn=1533-7928}} There are some concerns over copyrighted content in the C4.{{Cite news |last=Hern |first=Alex |date=2023-04-20 |title=Fresh concerns raised over sources of training material for AI systems |url=https://www.theguardian.com/technology/2023/apr/20/fresh-concerns-training-material-ai-systems-facist-pirated-malicious |access-date=2023-04-21 |work=The Guardian |language=en-GB |issn=0261-3077}}

{{reflist}}

• [http://commoncrawl.org/ Common Crawl] in California, United States
• [https://github.com/commoncrawl/ Common Crawl GitHub Repository] with the crawler, libraries and example code
• [https://groups.google.com/forum/?fromgroups#!forum/common-crawl Common Crawl Discussion Group]
• [https://commoncrawl.org/blog/ Common Crawl Blog]

Category:Internet-related organizations

Category:Web archiving

Category:Web archiving initiatives