Open Archives Initiative Protocol for Metadata Harvesting

In the late 1990s, Herbert Van de Sompel (Ghent University) was working with researchers and librarians at Los Alamos National Laboratory (US) and called a meeting to address difficulties related to interoperability issues of e-print servers and digital repositories. The meeting was held in Santa Fe, New Mexico, in October 1999.{{Cite journal|last=Marshall|first=E.|year=1999|title=Researchers plan free global preprint archive|url=https://www.science.org/doi/10.1126/science.286.5441.887a|journal=Science|volume=286|issue=5441|pages=887a–887|doi=10.1126/science.286.5441.887a|pmid=10577235|s2cid=178990556 |via=|url-access=subscription}} A key development from the meeting was the definition of an interface that permitted e-print servers to expose metadata for the papers it held in a structured fashion so other repositories could identify and copy papers of interest with each other. This interface/protocol was named the "Santa Fe Convention".{{Cite web |title=The Santa Fe Convention by the Open Archives Initiative |url=http://www.openarchives.org/sfc/sfc_entry.htm |date=February 15, 2000 |website=Open Archives Initiative |access-date=May 29, 2022}}

Several workshops were held in 2000 at the ACM Digital Libraries conference,{{Cite web|title=The Santa Fe Convention of the Open Archives Initiative|url=https://dspace.library.uu.nl/bitstream/handle/1874/3142/VandeSompelDLib2000SantaFe.htm?sequence=2&isAllowed=y|access-date=2021-02-10|website=dspace.library.uu.nl}} at the 1st ACM/IEEE-CS joint conference on Digital libraries{{Cite book|editor1=Edward A. Fox|editor2= Christine L. Borgman|language=en|location=Roanoke, Virginia, United States|publisher=ACM Press|volume=|pages=|doi=10.1145/379437|isbn=978-1-58113-345-5 |title=Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries |date=2001 }}{{Cite book|last1=Lagoze|first1=Carl|last2=Van de Sompel|first2=Herbert|title=Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries |chapter=The open archives initiative |date=2001 |citeseerx=10.1.1.161.6800|chapter-url=http://portal.acm.org/citation.cfm?doid=379437.379449|language=en|location=Roanoke, Virginia, United States|publisher=ACM Press|volume=|pages=54–62|doi=10.1145/379437.379449|isbn=978-1-58113-345-5|s2cid=1315824 |via=}} and elsewhere to share the ideas from the Santa Fe Convention.{{Cite journal|last1=Van de Sompel|first1=Herbert|last2=Lagoze|first2=Carl|year=2000|title=The Santa Fe Convention of the Open Archives Initiative|url=http://www.dlib.org/dlib/february00/vandesompel-oai/02vandesompel-oai.html|journal=D-Lib Magazine|language=en|volume=6|issue=2|pages=|doi=10.1045/february2000-vandesompel-oai|doi-access=free|issn=1082-9873|via=}} It was discovered at the workshops that the problems faced by the e-print community were also shared by libraries, museums, journal publishers, and others who needed to share distributed resources. To address these needs, the Coalition for Networked Information{{cite web |url=http://www.cni.org/ |title=Homepage |website=Coalition for Networked Information |access-date=May 29, 2022}} and the Digital Library Federation{{cite web |url=http://www.diglib.org/ |title=Homepage |website=Digital Library Federation |access-date=May 29, 2022}} provided funding to establish an Open Archives Initiative (OAI) secretariat managed by Herbert Van de Sompel and Carl Lagoze. The OAI held a meeting at Cornell University (Ithaca, New York) in September 2000 aimed to improve the interface developed at the Santa Fe Convention.{{Cite web|title=OAi-tech Meeting, Cornell University, September 7-8 2000|url=http://www.openarchives.org/meetings/tech-Cornell/oai-tech-cornell.htm|access-date=2021-02-10|website=www.openarchives.org}} The specifications were refined over e-mail.

OAI-PMH version 1.0 was introduced to the public in January 2001 at a workshop in Washington D.C.,{{Cite web|last=|first=|date=|title=The Open Archives Initiative: Open Meeting Renaissance Hotel, Washington DC January 23, 2001|url=https://www.openarchives.org/meetings/DC2001/OpenMeeting.html|access-date=2021-02-10|website=www.openarchives.org}} and another in February in Berlin, Germany.{{Cite web|last=|first=|date=|title=The Open Archives Initiative: Open Meeting Staatsbibliothek zu Berlin, Germany February 26, 2001|url=https://www.openarchives.org/meetings/Berlin2001/OpenMeeting-agenda.html|access-date=2021-02-10|website=www.openarchives.org}} Subsequent modifications to the XML standard by the W3C required making minor modifications to OAI-PMH resulting in version 1.1. The current version, 2.0, was released in June 2002. It contained several technical changes and enhancements and is not backward compatible.{{Cite journal|last1=Van de Sompel|first1=Herbert|last2=Young|first2=Jeffrey A.|last3=Hickey|first3=Thomas B.|year=2003|title=Using the OAI-PMH ... Differently|url=http://www.dlib.org/dlib/july03/young/07young.html|doi-access=free|journal=D-Lib Magazine|volume=9|issue=7/8|pages=|doi=10.1045/july2003-young|issn=1082-9873|via=}}

From 2001 CERN, and later in collaboration with University of Geneva, has organized bi-annual OAI workshops,{{Cite web |title=Previous OAI Workshops – OAI |url=https://oai.events/previous-oai-workshops/ |access-date=2023-01-13 |website=The Geneva Workshop on Innovations in Scholarly Communication |language=en-US}} which over time have developed to cover most aspects of open science. Since 2021 the workshop series is named the Geneva Workshop on Innovations in Scholarly Communication, with the nick name OAI reflecting its origin.{{Cite web |last=Azwa |first=Adnan Siti Norfateha |title=Library Guide: Open Access Guide: The Latest on OA |url=https://umlibguides.um.edu.my/OpenAccess/OANews |access-date=2023-01-13 |website=umlibguides.um.edu.my |language=en}}

Some commercial search engines use OAI-PMH to acquire more resources. Google initially included support for OAI-PMH when launching sitemaps, however decided to support only the standard XML Sitemaps format in May 2008.{{cite web |url=http://googlewebmastercentral.blogspot.com/2008/04/retiring-support-for-oai-pmh-in.html |title=Retiring Support for OAI-PMH in Sitemaps |date=April 23, 2008 |website=Google Search Central Blog |access-date=May 29, 2022}} In 2004, Yahoo! acquired content from OAIster (University of Michigan) that was obtained through metadata harvesting with OAI-PMH. Wikimedia uses an OAI-PMH repository to provide feeds of Wikipedia and related site updates for search engines and other bulk analysis/republishing endeavors.{{Cite web |title=Wikimedia update feed service |url=http://meta.wikimedia.org/wiki/Wikimedia_update_feed_service |publisher=Wikimedia Meta-Wiki |access-date=14 July 2013}} Especially when dealing with thousands of files being harvested every day, OAI-PMH can help in reducing the network traffic and other resource usage by doing incremental harvesting.{{cite web |url=http://www.dlxs.org/docs/13/ancil/harvest.html |title=OAI Harvesting System |website=DLXS |access-date=May 29, 2022}} NASA's Mercury metadata search system uses OAI-PMH to index thousands of metadata records from Global Change Master Directory (GCMD) every day.{{Cite journal |title=Data sharing and retrieval uses OAI-PMH |author1=R. Devarakonda |author2=G. Palanisamy |author3=J. Green |author4=B. Wilson |year=2010 |journal=Earth Science Informatics |volume=4 |issue=1 |pages=1–5 |publisher=Springer Berlin / Heidelberg |doi=10.1007/s12145-010-0073-0|s2cid=46330319 }}

The mod_oai project is using OAI-PMH to expose content to web crawlers that is accessible from Apache Web servers.

OAI-PMH has later been applied to sharing of scientific data.{{Cite journal|last1=Devarakonda|first1=Ranjeet|last2=Palanisamy|first2=Giri|last3=Green|first3=James M.|last4=Wilson|first4=Bruce E.|year=2011|title=Data sharing and retrieval using OAI-PMH|url=http://link.springer.com/10.1007/s12145-010-0073-0|journal=Earth Science Informatics|language=en|volume=4|issue=1|pages=1–5|doi=10.1007/s12145-010-0073-0|s2cid=46330319 |issn=1865-0473|via=|url-access=subscription}}

OAI-PMH is based on a client–server architecture, in which "harvesters" request information on updated records from "repositories". Requests for data can be based on a datestamp range, and can be restricted to named sets defined by the provider. Data providers are required to provide XML metadata in Dublin Core format, and may also provide it in other XML formats.

A number of software systems support the OAI-PMH, including Fedora, EThOS from the British Library, GNU EPrints from the University of Southampton, Open Journal Systems from the Public Knowledge Project, Desire2Learn, DSpace from MIT, HyperJournal from the University of Pisa, Digibib from Digibis, MyCoRe, Koha, Primo, DigiTool, Rosetta and MetaLib from Ex Libris, ArchivalWare from PTFS, DOOR {{cite web |url=http://door.sourceforge.net/index.html |title=Overview |website=DOOR |access-date=May 29, 2022}} from the eLab{{cite web |url=http://www.elearninglab.org/ |title=eLab |website=Universita della Svizzera italiana |language=Italian |access-date=May 29, 2022}} in Lugano, Switzerland, panFMP from the PANGAEA data library,{{cite web|url=http://www.panFMP.org/|title=PANGAEA® Framework for Metadata Portals|website=panfmp.org}} SimpleDL from Roaring Development, and jOAI from the National Center for Atmospheric Research.{{Cite web |url=https://github.com/NCAR/joai-project|title=NCAR/joai-project|website=Github.com|date=31 May 2022 }}

A number of large archives support the protocol including arXiv and the CERN Document Server.

• Data format management
• Digital curation
• Digital preservation
• File format
• Dublin Core, an ISO metadata standard
• National Digital Information Infrastructure and Preservation Program (NDIIPP)
• National Digital Library Program (NDLP)
• Metadata Encoding and Transmission Standard (METS) maintained by the Library of Congress
• Preservation Metadata: Implementation Strategies (PREMIS)
• LOCKSS
• Search as a service
• Web archiving
• Object Reuse and Exchange (OAI-ORE)
• [https://oai.events Geneva Workshop on Innovations in Scholarly Communication]

{{Reflist|colwidth=35em}}

• [https://web.archive.org/web/20100314214736/http://oai.sdu.edu.tr/ Suleyman Demirel University Open Archives Harvester]
• [http://www.openarchives.org/OAI/openarchivesprotocol.html Protocol specification]
• [https://www.loc.gov/library/libarch-digital.html National Library of Congress, Digital Collections and Programs]
• [http://www.digitalpreservation.gov/ Library of Congress, National Digital Information Infrastructure and Preservation Program]
• [https://www.loc.gov/webcapture/ Library of Congress, Web Capture]

{{open access navbox}}

Category:Online archives

Category:Internet protocols

Category:Metadata

Category:Open access projects

Category:Archival science

de:OAI-PMH