Cgroups

{{Redirect|cgroup|other uses|C group (disambiguation){{!}}C group}}

{{Infobox software

| name = cgroups

| title = cgroups

| logo =

| logo caption =

| logo_size =

| logo_alt =

| screenshot =

| caption =

| screenshot_size =

| screenshot_alt =

| collapsible =

| author = v1: Paul Menage, Rohit Seth, Memory Controller by Balbir Singh, CPU controller by Srivatsa Vaddagiri
v2: Tejun Heo

| developer = Tejun Heo, Johannes Weiner, Michal Hocko, Waiman Long, Roman Gushchin, Chris Down et al.

| released = {{Start date and age|2007}}

| discontinued =

| latest release version =

| latest release date =

| latest preview version =

| latest preview date =

| status =

| programming language = C

| operating system = Linux

| platform =

| size =

| language =

| language count =

| language footnote =

| genre = System software

| license = GPL and LGPL

| website = {{URL|https://docs.kernel.org/admin-guide/cgroup-v1/|Cgroup v1}}, {{URL|https://docs.kernel.org/admin-guide/cgroup-v2.html|Cgroup v2}}

}}

cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, etc.{{Cite web |title=Control Group v2 — The Linux Kernel documentation |url=https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#controllers |access-date=2023-06-24 |website=www.kernel.org}}) of a collection of processes.

Engineers at Google started the work on this feature in 2006 under the name "process containers".{{Cite news |date=29 May 2007 |author= Jonathan Corbet |title= Process containers |publisher= LWN.net |url= //lwn.net/Articles/236038/ }} In late 2007, the nomenclature changed to "control groups" to avoid confusion caused by multiple meanings of the term "container" in the Linux kernel context, and the control groups functionality was merged into the Linux kernel mainline in kernel version 2.6.24, which was released in January 2008.{{Cite news |date=29 October 2007 |author=Jonathan Corbet |title=Notes from a container |publisher=LWN.net |url=//lwn.net/Articles/256389/ | access-date=2015-04-14 | quote=The original 'containers' name was considered to be too generic – this code is an important part of a container solution, but it's far from the whole thing. So containers have now been renamed 'control groups' (or 'cgroups') and merged for 2.6.24.}} Since then, developers have added many new features and controllers, such as support for kernfs in 2014,{{cite web |url=https://lkml.org/lkml/2014/1/28/502 |title=cgroup: convert to kernfs |publisher=Linux kernel mailing list|date=2014-01-28}} firewalling,{{cite web|url=http://www.spinics.net/lists/netdev/msg264727.html |title=netfilter: x_tables: lightweight process control group matching |date=2014-04-23 |url-status=dead |archive-url=https://web.archive.org/web/20140424095421/http://www.spinics.net/lists/netdev/msg264727.html |archive-date=24 April 2014 |df=dmy }} and unified hierarchy.{{cite web |url=https://lkml.org/lkml/2014/3/13/503 |title=cgroup: prepare for the default unified hierarchy |date=2014-03-13}} cgroup v2 was merged in Linux kernel 4.5 with significant changes to the interface and internal functionality.{{Cite web|url=https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#multiple-hierarchies|title=cgroup v2: Multiple hierarchies}}

Versions

{{Cleanup|section|reason=incomplete descriptions and lack of references|date=June 2016}}

There are two versions of cgroups.

Cgroups was originally written by Paul Menage and Rohit Seth, and merged into the mainline Linux kernel in 2007. Afterwards this is called cgroups version 1.{{cite web |url=https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/diff/Documentation/cgroup-v1?id=v4.5&id2=v4.4 |title=diff between Linux kernel 4.4 and 4.5 |date=2016-03-14}}

Development and maintenance of cgroups was then taken over by Tejun Heo. Tejun Heo redesigned and rewrote cgroups. This rewrite is now called version 2, the documentation of cgroup-v2 first appeared in Linux kernel 4.5 released on 14 March 2016.{{cite web |url=https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/diff/Documentation/cgroup-v2.txt?id=v4.5&id2=v4.4 |title=Documentation/cgroup-v2.txt as appeared in Linux kernel 4.5 |date=2016-03-14}}

Unlike v1, cgroup v2 has only a single process hierarchy and discriminates between processes, not threads.

Features

One of the design goals of cgroups is to provide a unified interface to many different use cases, from controlling single processes (by using nice, for example) to full operating system-level virtualization (as provided by OpenVZ, Linux-VServer or LXC, for example). Cgroups provides:

; Resource limiting

: groups can be set not to exceed a configured memory limit, which also includes the file system cache,{{Cite news |date=31 July 2007 |author= Jonathan Corbet | title = Controlling memory use in containers | publisher=LWN |url = //lwn.net/Articles/243795/}}{{cite web |title=Containers: Challenges with the memory resource controller and its performance | author= Balbir Singh, Vaidynathan Srinivasan | date = July 2007| publisher = Ottawa Linux Symposium | url = http://www.kernel.org/doc/ols/2007/ols2007v2-pages-209-222.pdf }} I/O bandwidth limit,{{Cite web |last=Carvalho |first=André |title=Using cgroups to limit I/O |url=https://andrestc.com/post/cgroups-io/ |access-date=2022-09-12 |website=andrestc.com|date=18 October 2017 }} CPU quota limit,{{Cite web |last=Luu |first=Dan |title=The container throttling problem |url=https://danluu.com/cgroup-throttling/ |access-date=2022-09-12 |website=danluu.com}} CPU set limit,{{Cite web |last=Derr |first=Simon |date=2004 |title=CPUSETS |url=https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt |access-date=2022-09-12}} or maximum open files.{{Cite web |title=setrlimit(2) — Arch manual pages |url=https://man.archlinux.org/man/setrlimit.2.en |access-date=2023-11-27 |website=man.archlinux.org}}

; Prioritization

: some groups may get a larger share of CPU utilization{{Cite news | date = 23 October 2007 | author = Jonathan Corbet | title = Kernel space: Fair user scheduling for Linux | publisher = Network World | url = http://www.networkworld.com/news/2007/101207-kernel.html | access-date = 2012-08-22 | archive-url = https://web.archive.org/web/20131019123524/http://www.networkworld.com/news/2007/101207-kernel.html | archive-date = 19 October 2013 | url-status = dead | df = dmy-all }} or disk I/O throughput{{cite conference | date = 19 November 2008 | author = Kamkamezawa Hiroyu | title = Cgroup and Memory Resource Controller | format = PDF presentation slides | publisher = Japan Linux Symposium | url = http://www.linuxfoundation.jp/jp_uploads/seminar20081119/CgroupMemcgMaster.pdf | url-status = dead | archive-url = https://web.archive.org/web/20110722113016/http://www.linuxfoundation.jp/jp_uploads/seminar20081119/CgroupMemcgMaster.pdf | archive-date = 22 July 2011 | df = dmy-all }}

; Accounting

: measures a group's resource usage, which may be used, for example, for billing purposes{{cite conference | vauthors=Hansen D, ((IBM Linux Technology Center)) | title = Resource Management | format = PDF presentation slides | publisher = Linux Foundation | url = https://events.static.linuxfound.org/slides/lfcs09_hansen2.pdf | date = 2009}}

; Control

: freezing groups of processes, their checkpointing and restarting

{{Anchor|USAGE}}Use

File:Linux kernel and daemons with exclusive access.svg

A control group (abbreviated as cgroup) is a collection of processes that are bound by the same criteria and associated with a set of parameters or limits. These groups can be hierarchical, meaning that each group inherits limits from its parent group. The kernel provides access to multiple controllers (also called subsystems) through the cgroup interface; for example, the "memory" controller limits memory use, "cpuacct" accounts CPU usage, etc.

Control groups can be used in multiple ways:

By accessing the cgroup virtual file system manually.
By creating and managing groups on the fly using tools like cgcreate, cgexec, and cgclassify (from libcgroup).
Through the "rules engine daemon" that can automatically move processes of certain users, groups, or commands to cgroups as specified in its configuration.
Indirectly through other software that uses cgroups, such as Docker, Firejail, LXC,{{Cite news |author=Matt Helsley |date=3 February 2009 |title=LXC: Linux container tools |publisher=IBM developerWorks |url=http://www.ibm.com/developerworks/linux/library/l-lxc-containers/ }} libvirt, systemd, Open Grid Scheduler/Grid Engine,{{cite web |url=http://blogs.scalablelogic.com/2012/05/grid-engine-cgroups-integration.html |title=Grid Engine cgroups Integration| date=2012-05-22 |publisher=Scalable Logic}} and Google's developmentally defunct lmctfy.

The Linux kernel documentation contains some technical details of the setup and use of control groups version 1{{cite web |url=http://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt |title=cgroups |publisher=kernel.org}} and version 2.{{Cite web|url=https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/cgroup-v2.rst|title = Torvalds/Linux|website = GitHub|date = 13 February 2022}} systemd-cgtop{{Cite web|url=https://www.freedesktop.org/software/systemd/man/systemd-cgtop.html|title = Systemd-cgtop}} command can be used to show top control groups by their resource usage.

Redesign

Redesign of cgroups started in 2013,{{cite web |url=https://www.linuxfoundation.org/blog/2013/08/all-about-the-linux-kernel-cgroups-redesign/ |title=All About the Linux Kernel: Cgroup's Redesign |work=Linux.com |date=2013-08-15 |access-date=2014-05-19 |archive-date=28 April 2019 |archive-url=https://web.archive.org/web/20190428203713/https://www.linuxfoundation.org/blog/2013/08/all-about-the-linux-kernel-cgroups-redesign/ |url-status=dead }} with additional changes brought by versions 3.15 and 3.16 of the Linux kernel.{{cite web |url=//lwn.net/Articles/601840/ |title=The unified control group hierarchy in 3.16 |publisher=LWN.net |date=2014-06-11}}{{cite web |url=http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=32d01dc7be4e725ab85ce1d74e8f4adc02ad68dd |title=Pull cgroup updates for 3.15 from Tejun Heo |publisher=kernel.org |date=2014-04-03}}{{cite web |url=http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=14208b0ec56919f5333dd654b1a7d10765d0ad05 |title=Pull cgroup updates for 3.16 from Tejun Heo |publisher=kernel.org |date=2014-06-09}}

= {{Anchor|NAMESPACE-ISOLATION}}Namespace isolation =

While not technically part of the cgroups work, a related feature of the Linux kernel is namespace isolation, where groups of processes are separated such that they cannot "see" resources in other groups. For example, a PID namespace provides a separate enumeration of process identifiers within each namespace. Also available are mount, user, UTS (Unix Time Sharing), network and SysV IPC namespaces.

The PID namespace provides isolation for the allocation of process identifiers (PIDs), lists of processes and their details. While the new namespace is isolated from other siblings, processes in its "parent" namespace still see all processes in child namespaces—albeit with different PID numbers.{{Cite news |author=Pavel Emelyanov, Kir Kolyshkin |date=19 November 2007 |title=PID namespaces in the 2.6.24 kernel |publisher=LWN.net |url=//lwn.net/Articles/259217/ }}
Network namespace isolates the network interface controllers (physical or virtual), iptables firewall rules, routing tables etc. Network namespaces can be connected with each other using the "veth" virtual Ethernet device.{{Cite news |date=30 January 2007 |author=Jonathan Corbet |title=Network namespaces |publisher=LWN.net |url=//lwn.net/Articles/219794/ }}
"UTS" namespace allows changing the hostname.
Mount namespace allows creating a different file system layout, or making certain mount points read-only.{{Cite news |author=Serge E. Hallyn, Ram Pai |date=17 September 2007 |title=Applying mount namespaces |publisher=IBM developerWorks |url=http://www.ibm.com/developerworks/linux/library/l-mount-namespaces.html }}
IPC namespace isolates the System V inter-process communication between namespaces.
User namespace isolates the user IDs between namespaces.{{Cite news | date= 27 February 2013| author=Michael Kerrisk|title=Namespaces in operation, part 5: User namespaces|publisher=lwn.net Linux Info from the Source|url=//lwn.net/Articles/532593/}}
Cgroup namespace{{Cite web|url=https://lkml.org/lkml/2016/3/26/132/|title=LKML: Linus Torvalds: Linux 4.6-rc1}}

Namespaces are created with the "unshare" command or syscall, or as "new" flags in a "clone" syscall.{{cite web |date=11 January 2006 |author=Janak Desai |title=Linux kernel documentation on unshare |url=http://www.mjmwired.net/kernel/Documentation/unshare.txt }}

The "ns" subsystem was added early in cgroups development to integrate namespaces and control groups. If the "ns" cgroup was mounted, each namespace would also create a new group in the cgroup hierarchy. This was an experiment that was later judged to be a poor fit for the cgroups API, and removed from the kernel.

Linux namespaces were inspired by the more general namespace functionality used heavily throughout Plan 9 from Bell Labs.{{cite web | title=The Use of Name Spaces in Plan 9 | date=1992 | url=http://www.cs.bell-labs.com/sys/doc/names.html | access-date=15 February 2015 | archive-url=https://web.archive.org/web/20140906153815/http://www.cs.bell-labs.com/sys/doc/names.html | archive-date=6 September 2014 | url-status=dead | df=dmy-all }}

= Unified hierarchy =

Kernfs was introduced into the Linux kernel with version 3.14 in March 2014, the main author being Tejun Heo.{{cite web |url=//lwn.net/Articles/584019/ |title=kernfs, sysfs, driver-core: implement synchronous self-removal |publisher=LWN.net |date=2014-02-03 |access-date=2014-04-07}} One of the main motivators for a separate kernfs is the cgroups file system. Kernfs is basically created by splitting off some of the sysfs logic into an independent entity, thus easing for other kernel subsystems the implementation of their own virtual file system with handling for device connect and disconnect, dynamic creation and removal, and other attributes. Redesign continued into version 3.15 of the Linux kernel.{{cite web

| url = https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=2bd59d48ebfb3df41ee56938946ca0dd30887312

| title = Linux kernel source tree: kernel/git/torvalds/linux.git: cgroups: convert to kernfs

| date = 2014-02-11 | access-date = 2014-05-23

| publisher = kernel.org

}}

= {{Anchor|Kmemcg}}Kernel memory control groups (kmemcg) =

Kernel memory control groups (kmemcg) were merged into version 3.8 ({{release date and age|df=yes|18|02|2013}}) of the Linux kernel mainline.{{cite web |url= http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7ae1e1d0f8ac2927ed7e3ca6d15e42d485903459 |title= memcg: kmem controller infrastructure |publisher=kernel.org source code|date=2012-12-18 }}{{cite web |url=http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=510fc4e11b772fd60f2c545c64d4c55abd07ce36 |title= memcg: kmem accounting basic infrastructure |publisher=kernel.org source code |date=2012-12-18}}{{cite web |url= http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d5bdae7d59451b9d63303f7794ef32bb76ba6330 |title= memcg: add documentation about the kmem controller |publisher=kernel.org |date=2012-12-18}} The kmemcg controller can limit the amount of memory that the kernel can utilize to manage its own internal processes.

= cgroup awareness of OOM killer =

Linux Kernel 4.19 (October 2018) introduced cgroup awareness of OOM killer implementation which adds an ability to kill a cgroup as a single unit and so guarantee the integrity of the workload.{{cite web | url=https://kernelnewbies.org/Linux_4.19#Memory_management | title=Linux_4.19 - Linux Kernel Newbies }}

{{Anchor|ADOPTION}}Adoption

Various projects use cgroups as their basis, including CoreOS, Docker (in 2013), Hadoop, Jelastic, Kubernetes,{{cite web |url=https://mesosphere.com/blog/2014/07/10/mesosphere-announces-kubernetes-on-mesos/ |title=Mesosphere to Bring Google's Kubernetes to Mesos |publisher=Mesosphere.io |date=2014-07-10 |access-date=2014-07-13 |archive-url=https://web.archive.org/web/20150906165229/https://mesosphere.com/blog/2014/07/10/mesosphere-announces-kubernetes-on-mesos/ |archive-date=6 September 2015 |url-status=dead }} lmctfy (Let Me Contain That For You), LXC (Linux Containers), systemd, Mesos and Mesosphere, and HTCondor.

Major Linux distributions also adopted it such as Red Hat Enterprise Linux (RHEL) 6.0 in November 2010, three years before adoption by the mainline Linux kernel.{{cite web|url=https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/pdf/6.0_release_notes/red_hat_enterprise_linux-6-6.0_release_notes-en-us.pdf|title=6.0 Release Notes|website=redhat.com|access-date=12 September 2023}}

On 29 October 2019, the Fedora Project modified Fedora 31 to use CgroupsV2 by default{{Cite web|url=https://bugzilla.redhat.com/show_bug.cgi?id=1732114|title=1732114 – Modify Fedora 31 to use CgroupsV2 by default}}

References

External links

[https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/index.html Official Linux kernel documentation on cgroups v1] and [https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html cgroups v2]
[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01 Red Hat Resource Management Guide on cgroups]
[http://manpages.ubuntu.com/manpages/cosmic/man7/cgroups.7.html Ubuntu manpage on cgroups] {{Webarchive|url=https://web.archive.org/web/20210809192233/http://manpages.ubuntu.com/manpages/cosmic/man7/cgroups.7.html |date=9 August 2021 }}
[http://www.haifux.org/lectures/299/netLec7.pdf Linux kernel Namespaces and cgroups by Rami Rosen] (2013)
[http://www.netdevconf.org/1.1/proceedings/slides/rosen-namespaces-cgroups-lxc.pdf Namespaces and cgroups, the basis of Linux containers (including cgroups v2)], slides of a talk by Rami Rosen, Netdev 1.1, Seville, Spain, 2016
[//lwn.net/Articles/679786 Understanding the new control groups API], LWN.net, by Rami Rosen, March 2016
[https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/43438.pdf Large-scale cluster management at Google with Borg], April 2015, by Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune and John Wilkes
[https://msdn.microsoft.com/en-us/library/windows/desktop/ms684161(v=vs.85).aspx Job Objects], similar feature on Windows