Singularity (software)

{{short description|Free, cross-platform and open-source computer program}}

{{Infobox software

| name = Singularity

| screenshot = Singularity-software-build and run example-screenshot.png

| caption = Singularity running a hello world container from the command line

| author = Gregory Kurtzer (gmk), et al.

| developer = Community
Gregory Kurtzer

| latest_release_version = 3.8.7

| latest release date = {{Start date and age|df=yes|2022|03|17}}{{cite web

| url = https://www.github.com/apptainer/apptainer/releases

| title = Releases · apptainer/singularity

| website = github.com

| access-date = 29 June 2022

}}

| programming language = Go{{cite web|url=https://github.com/apptainer/apptainer/search?l=go|title=Singularity+GoLang|website=GitHub|access-date=3 December 2021 }}

| operating system = Linux

| genre = Operating-system-level virtualization

| license = 3-clause BSD License{{cite web |url=https://github.com/apptainer/apptainer/blob/main/LICENSE.md |title=Singularity License |website=Apptainer.org |publisher=Singularity Team |access-date=3 December 2021 }}

| website = {{Official URL}}

}}

Singularity is a free and open-source computer program that performs operating-system-level virtualization also known as containerization.{{Cite web|url=https://archive.fosdem.org/2017/schedule/event/singularityhpc/|title=Singularity presentation at FOSDEM 17|website=archive.fosdem.org}}

One of the main uses of Singularity is to bring containers and reproducibility to scientific computing and the high-performance computing (HPC) world.{{cite journal |title=Singularity: Scientific Containers for Mobility of Compute |journal=PLOS ONE |volume=12 |issue=5 |pages=e0177459 |doi=10.1371/journal.pone.0177459 |pmid=28494014 |pmc=5426675 |date=2017 |last1=Kurtzer |first1=Gregory M. |last2=Sochat |first2=Vanessa |last3=Bauer |first3=Michael W. |bibcode=2017PLoSO..1277459K |doi-access=free }}

The need for reproducibility requires the ability to use containers to move applications from system to system.{{cite web|url=http://www.admin-magazine.com/HPC/Articles/Singularity-A-Container-for-HPC|title=Singularity, a container for HPC|date=24 April 2016|publisher=admin-magazine.com}}

Using Singularity containers, developers can work in reproducible environments of their choosing and design, and these complete environments can easily be copied and executed on other platforms.{{cite web|url=https://singularityware.github.io/user-guide.html|title=Singularity Manual: Mobility of Compute|work=Singularity User Guide - Version 2.5.2}}

In 2021 the Singularity open source project split into two projects called Apptainer and SingularityCE.

File:Singularity container workflow.png

History

Singularity began as an open-source project in 2015, when a team of researchers at Lawrence Berkeley National Laboratory, led by Gregory Kurtzer, developed the initial version written in the C programming language and released it{{Cite web|url=https://www.top500.org/news/sylabs-brings-singularity-containers-into-commercial-hpc/|title=Sylabs Brings Singularity Containers into Commercial HPC |website=top500.org}} under the BSD license.{{cite web|url=http://singularity.lbl.gov/docs-license |title=Singularity License |work=singularity.lbl.gov |publisher=Singularity Team |date=19 March 2018 |access-date=2018-03-19}}

By the end of 2016, many developers from different research facilities joined forces with the team at Lawrence Berkeley National Laboratory to further the development of Singularity.{{cite web|url=https://github.com/singularityware/singularity/commit/56ee6e80e158dda288609ca0d342a153c7ddfe19#diff-a8b2df7fd69554249528b6ce108f533b|title= Changes to the AUTHORS.md file in Singularity source code made in April 2017|website=GitHub }}

Singularity quickly attracted the attention of computing-heavy scientific institutions worldwide:{{cite web|url=http://newscenter.lbl.gov/2017/06/07/berkeley-labs-open-source-spinoff-serves-science/|title=Berkeley Lab's Open-Source Spinoff Serves Science|date=7 June 2017}}

Stanford University Research Computing Center deployed Singularity on their XStream{{Cite web|url=http://xstream.stanford.edu/docs/singularity/|title=XStream online user manual, section on Singularity|website=xstream.stanford.edu}}{{Cite web|url=https://portal.xsede.org/stanford-xstream|title=XStream cluster overview|access-date=10 April 2018|archive-date=24 October 2020|archive-url=https://web.archive.org/web/20201024231800/https://portal.xsede.org/stanford-xstream|url-status=dead}} and Sherlock{{Cite web|url=https://srcc.stanford.edu/news/sherlock-whats-new-containers-and-deep-learning-tools|title=Sherlock: What's New, Containers and Deep Learning Tools |website= Stanford Research Computing Center}} clusters
National Institutes of Health installed Singularity on Biowulf,{{Cite web|url=https://hpc.nih.gov/apps/singularity.html|title=NIH HPC online user manual, section on Singularity |website=hpc.nih.gov }} their 95,000+ core/30 PB Linux cluster{{Cite web|url=https://hpc.nih.gov/systems/|title=NIH HPC Systems|website=hpc.nih.gov}}
Various sites of the Open Science Grid Consortium including Fermilab started adopting Singularity;{{Cite web|url=http://fife.fnal.gov/singularity-on-the-osg/|title=Singularity on the OSG}} by April 2017, Singularity was deployed on 60% of the Open Science Grid network.{{cite web|url=https://indico.cern.ch/event/612601/contributions/2495602/attachments/1424434/2184476/Singularity-in-CMS-v2.pdf|title=Singularity in CMS: Over a million containers served}}

For two years in a row, in 2016 and 2017, Singularity was recognized by HPCwire editors as "One of five new technologies to watch".{{Cite web|url=https://www.hpcwire.com/off-the-wire/hpcwire-reveals-winners-2016-readers-editors-choice-awards-sc16-conference-salt-lake-city/|title=HPCwire Reveals Winners of the 2016 Readers' and Editors' Choice Awards at SC16 Conference in Salt Lake City|website=HPCwire}}{{Cite web|url=https://www.hpcwire.com/off-the-wire/hpcwire-reveals-winners-2017-readers-editors-choice-awards-sc17-conference-denver/|title=HPCwire Reveals Winners of the 2017 Readers' and Editors' Choice Awards at SC17 Conference in Denver|website=HPCwire}}

In 2017 Singularity also won the first place for the category "Best HPC Programming Tool or Technology".

{{As of|2018|post=,}} based on the data entered on a voluntary basis in a public registry, Singularity user base was estimated to be greater than 25,000 installations{{Cite web|url=https://docs.google.com/spreadsheets/d/1Vc_1prq_1WHGf0LWtpUBY-tfKdLLM_TErjnCe1mY5m0/pub?gid=1407658660&single=true&output=pdf|title=Voluntary registry of Singularity installations}} and included users at academic institutions such as Ohio State University and Michigan State University, as well as top HPC centers like Texas Advanced Computing Center, San Diego Supercomputer Center, and Oak Ridge National Laboratory.

In February 2018 the Sylabs{{cite web|url=https://sylabs.io|title=Sylabs home page|access-date=29 June 2022}} company, founded by the Singularity author, was announced {{cite press release|url=https://www.globenewswire.com/news-release/2018/02/08/1336384/0/en/Sylabs-Emerges-from-Stealth-to-Bring-Singularity-Container-Technology-to-Enterprise-Performance-Computing.html|title=Sylabs Emerges from Stealth to Bring Singularity Container Technology to Enterprise Performance Computing|date=8 February 2018 |access-date=29 June 2022}} to provide commercial support for Singularity. In October of that year Sylabs released version 3.0.0{{Cite web|url=https://github.com/apptainer/singularity/releases/tag/v3.0.0|title=Singularity 3.0.0 |website=GitHub}} which was a rewrite in the Go programming language.

= Apptainer / Singularity split =

In May 2020 Gregory Kurtzer left Sylabs but retained leadership of the Singularity open source project.{{cite web|url=https://groups.google.com/a/lbl.gov/g/singularity/c/HAcb_l1WUgg/m/LPdAOJbXBwAJ|title=Singularity repository move and company updates|access-date=29 June 2022}} In May 2021 Sylabs made a fork of the project{{cite web|url=https://groups.google.com/a/lbl.gov/g/singularity/c/UbywVHXD_co/m/bGa8M4QAAQAJ|title=Sylabs fork of Singularity|access-date=29 June 2022|archive-url=https://archive.today/2025.05.20-122806/https://groups.google.com/a/lbl.gov/g/singularity/c/UbywVHXD_co/m/bGa8M4QAAQAJ|archive-date=2025-05-20}} and called it SingularityCE.{{cite web|url=https://sylabs.io/singularity/|title=SingularityCE|access-date=30 June 2022}}{{Cite web|url=https://github.com/sylabs/singularity|title=SingularityCE|date=28 October 2022|via=GitHub}} In November 2021 the Singularity open source project joined the Linux Foundation{{cite web|url=https://groups.google.com/a/lbl.gov/g/singularity/c/lRWR3vJGvks/m/bMvv9OfYBAAJ|title=Singularity has joined the Linux Foundation!|access-date=29 June 2022}} and was renamed to Apptainer.{{cite web|url=https://apptainer.org|title=Apptainer website|access-date=15 February 2023}}

Features

Singularity is able to support natively high-performance interconnects, such as InfiniBand{{cite web|url=https://www.intel.com/content/dam/www/public/us/en/documents/presentation/hpc-containers-singularity-advanced.pdf|title=Intel Advanced Tutorial: HPC Containers & Singularity – Advanced Tutorial – Intel}} and Intel Omni-Path Architecture (OPA).{{cite web|url=https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Build_Containers_for_Intel_OPA_AN_J57474_v4_0.pdf|title=Intel Application Note: Building Containers for Intel Omni-Path Fabrics using Docker and Singularity }}

Similar to the support for InfiniBand and Intel OPA devices, Singularity can support any PCIe-attached device within the compute node, such as graphic accelerators.{{cite web|url=http://singularity.lbl.gov/docs-exec#a-gpu-example|title=Singularity Manual: A GPU example}}

Singularity also has native support for Open MPI library by utilizing a hybrid MPI container approach where OpenMPI exists both inside and outside the container.

These features make Singularity increasingly useful in areas such as machine learning, deep learning and most data-intensive workloads where the applications benefit from the high bandwidth and low latency characteristics of these technologies.{{cite book |volume=10724 |pages=3–21 |doi=10.1007/978-3-319-72971-8_1 |series=Lecture Notes in Computer Science |date=2018 |last1=Tallent |first1=Nathan R. |last2=Gawande |first2=Nitin |last3=Siegel |first3=Charles |last4=Vishnu |first4=Abhinav |last5=Hoisie |first5=Adolfy |title=High Performance Computing Systems. Performance Modeling, Benchmarking, and Simulation |chapter=Evaluating On-Node GPU Interconnects for Deep Learning Workloads |isbn=978-3-319-72970-1 |s2cid=1674152 }}

Integration

HPC systems traditionally already have resource management and job scheduling systems in place, so the container runtime environments must be integrated into the existing system resource manager.

Using other enterprise container solutions like Docker in HPC systems would require modifications to the software.{{cite web|url=https://cug.org/proceedings/cug2017_proceedings/includes/files/pap164s2-file1.pdf|title=HPC Containers in use|author=Jonathan Sparks, Cray Inc.|year=2017}}

Docker containers can be automatically converted to stand-alone singularity files which can then be submitted to HPC resource managers.{{cite web|url=https://apptainer.org/user-docs/3.8/singularity_and_docker.html|title=Singularity and Docker|access-date=December 3, 2021}}

Singularity seamlessly integrates with many resource managers{{cite web|url=http://singularity.lbl.gov/user-guide#support-on-existing-traditional-hpc|title=Support on existing traditional HPC}} including:

HTCondor{{Cite web|url=https://htcondor.readthedocs.io/en/stable/admin-manual/singularity-support.html|title=HTCondor Stable Release Manual : Singularity Support|access-date=4 February 2020|archive-date=4 February 2020|archive-url=https://web.archive.org/web/20200204215032/https://htcondor.readthedocs.io/en/stable/admin-manual/singularity-support.html|url-status=dead}}
Oracle Grid Engine (SGE)
SLURM (Simple Linux Utility for Resource Management)
TORQUE (Terascale Open-source Resource and QUEue Manager)
PBS Pro (PBS Professional)
[https://www.nomadproject.io/docs/drivers/external/singularity HashiCorp Nomad] (A simple and flexible workload orchestrator)
[https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=containers-lsf-singularity IBM Platform LSF]