Zstd

Zstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest){{Cite web |title=Release Zstandard v1.3.4 - faster everything · facebook/zstd |url=https://github.com/facebook/zstd/releases/tag/v1.3.4 |access-date=2024-03-27 |website=GitHub |language=en |archive-date=11 September 2021 |archive-url=https://web.archive.org/web/20210911024159/https://github.com/facebook/zstd/releases/tag/v1.3.4 |url-status=live }} to 22 (slowest in compression speed, but best compression ratio).

Starting from version 1.3.2 (October 2017), zstd optionally implements very-long-range search and deduplication ({{code|--long}}, 128 MiB window) similar to rzip or lrzip.{{cite web |title=Command Line Interface for Zstandard library |url=https://github.com/facebook/zstd/blob/dev/programs/README.md |website=GitHub |date=28 October 2021 |language=en}}

Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.{{cite web

|url=http://www.open-zfs.org/w/images/b/b3/03-OpenZFS_2017_-_ZStandard_in_ZFS.pdf

|title=ZStandard in ZFS

|date=2017

|access-date=2019-04-20

|website=open-zfs.org

|archive-date=18 December 2019

|archive-url=https://web.archive.org/web/20191218102535/http://open-zfs.org/w/images/b/b3/03-OpenZFS_2017_-_ZStandard_in_ZFS.pdf

|url-status=live

}} The Zstandard command-line has an "adaptive" ({{code|--adapt}}) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.

Zstd at its maximum compression level gives a compression ratio close to lzma, lzham, and ppmx, and {{synthesis span|performs better{{vague|date=January 2023}} than lza or bzip2.|date=January 2023}}{{cite web

|url=http://mattmahoney.net/dc/silesia.html

|title=Silesia Open Source Compression Benchmark

|author=Matt Mahoney

|access-date=2019-05-10

|archive-date=21 January 2022

|archive-url=https://web.archive.org/web/20220121165038/http://mattmahoney.net/dc/silesia.html

|url-status=live

}}{{cite web

|url=http://mattmahoney.net/dc/text.html

|title=Large Text Compression Benchmark, .2157 zstd

|author=Matt Mahoney

|date=2016-08-29

|access-date=2016-09-01

|archive-date=31 March 2022

|archive-url=https://web.archive.org/web/20220331031650/http://mattmahoney.net/dc/text.html

|url-status=live

}} Zstandard reaches the current Pareto frontier, as it decompresses faster than any other currently available algorithm with similar or better compression ratio.{{As of when|date=December 2024}}{{citation

|title=TurboBench: Static/Dynamic web content compression benchmark

|url=https://sites.google.com/site/powturbo/home/web-compression

|publisher=PowTurbo

|access-date=21 March 2018

|archive-date=17 March 2022

|archive-url=https://web.archive.org/web/20220317204149/https://sites.google.com/site/powturbo/home/web-compression

|url-status=live

}}{{citation

|author=Matt Mahoney

|title=Silesia Open Source Compression Benchmark

|url=http://mattmahoney.net/dc/silesia.html

|access-date=5 April 2018

|archive-date=21 January 2022

|archive-url=https://web.archive.org/web/20220121165038/http://mattmahoney.net/dc/silesia.html

|url-status=live

}}

Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.{{Cite web |date=2017-10-11 |title=Facebook developers report massive speedups and compression ratio improvements when using dictionaries |url=https://indico.fnal.gov/event/15154/contribution/5/material/slides/0.pdf |access-date=2024-03-27 |website=Fermilab |archive-date=25 January 2018 |archive-url=https://web.archive.org/web/20180125015337/https://indico.fnal.gov/event/15154/contribution/5/material/slides/0.pdf |url-status=live }}{{cite web

|url=https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/

|title=Smaller and faster data compression with Zstandard

|date=31 August 2016

|publisher=Facebook

|access-date=3 September 2016

|archive-date=8 November 2020

|archive-url=https://web.archive.org/web/20201108105437/https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard

|url-status=live

}} In particular, one dictionary can be loaded to process large sets of files with redundancy between files, but not necessarily within each file, such as for log files.

Zstandard combines a dictionary-matching stage (LZ77) with a large search window and a fast entropy-coding stage. It uses both Huffman coding (used for entries in the Literals section){{cite web|url=https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#entropy-encoding|title=facebook/zstd|website=GitHub|date=28 October 2021}} and finite-state entropy (FSE) – a fast tabled version of ANS, tANS, used for entries in the Sequences section. Because of the way that FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).

{{Infobox file format

| name = Zstandard

| extension = .zst

{{cite IETF

|title=Zstandard Compression and the application/zstd Media Type

|rfc=8878

|last1=Collet

|first1=Yann

|editor1-last=Kucherawy

|editor1-first=Murray S.

|editor1-link=Murray Kucherawy

|date=February 2021

|access-date=26 February 2023

|series=Internet Engineering Task Force Request for Comments}}

| mime = application/zstd

| genre = Data compression

| magic = 28 b5 2f fd

| standard = {{IETF RFC|8878}}

| url = {{URL|https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md}}

}}

{{Infobox file format

| name = Zstandard Dictionary

| magic = 37 a4 30 ec

| standard = {{IETF RFC|8878}}

| url = {{URL|https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format}}

}}

The Linux kernel has included Zstandard since November 2017 (version 4.14) as a compression method for the btrfs and squashfs filesystems.{{cite web |last=Corbet |first=Jonathan |date=2017-09-17 |title=The rest of the 4.14 merge window [LWN.net] |url=https://lwn.net/Articles/733846/ |access-date=2024-03-27 |website=lwn.net |archive-date=22 November 2021 |archive-url=https://web.archive.org/web/20211122141753/https://lwn.net/Articles/733846/ |url-status=live }}{{cite web |date=2017-12-30 |title=Linux_4.14 - Linux Kernel Newbies |url=https://kernelnewbies.org/Linux_4.14 |access-date=2018-08-16 |publisher=Kernelnewbies.org |archive-date=10 January 2018 |archive-url=https://web.archive.org/web/20180110114105/https://kernelnewbies.org/Linux_4.14 |url-status=live }}{{cite web |last=Larabel |first=Michael |date=2017-09-08 |title=Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix |url=https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.14-Zstd-Pull |website=www.phoronix.com |access-date=13 November 2017 |archive-date=25 July 2019 |archive-url=https://web.archive.org/web/20190725122757/https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.14-Zstd-Pull |url-status=live }}

In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel,{{cite web | url=https://github.com/freebsd/freebsd/commit/28ef16535cde21eeeaf75f6006b3a77952b3b51 | title=Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165 | website=GitHub }} and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-concept OpenZFS compression method which was integrated in 2020.{{cite web|url=https://github.com/openzfs/zfs/commit/10b3c7f5e424f54b3ba82dbf1600d866e64ec0a0|title=Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f|website=GitHub|access-date=12 October 2020|archive-date=10 September 2020|archive-url=https://web.archive.org/web/20200910151903/https://github.com/openzfs/zfs/commit/10b3c7f5e424f54b3ba82dbf1600d866e64ec0a0|url-status=live}}

The AWS Redshift and RocksDB databases include support for field compression using Zstandard.{{cite web

|url=https://docs.aws.amazon.com/redshift/latest/dg/zstd-encoding.html

|title=Zstandard Encoding - Amazon Redshift

|date=20 April 2019

|access-date=24 January 2018

|archive-date=14 August 2021

|archive-url=https://web.archive.org/web/20210814225631/https://docs.aws.amazon.com/redshift/latest/dg/zstd-encoding.html

|url-status=live

}}

In March 2018, Canonical tested{{cite web |url=https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Zstd-Deb-Packages |title=Canonical Working On Zstd-Compressed Debian Packages For Ubuntu |last=Larabel |first=Michael |date=12 March 2018 |website=phoronix.com |publisher=Phoronix Media |access-date=29 October 2019 |quote=The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10. |archive-date=16 August 2021 |archive-url=https://web.archive.org/web/20210816101454/https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Zstd-Deb-Packages |url-status=live }} the use of zstd as a deb package compression method by default for the Ubuntu Linux distribution. Compared with xz compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Support was added to Debian (and subsequently, Ubuntu) in April 2018 (in version 1.6~rc1).{{cite web

|title=New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm

|url=https://news.softpedia.com/news/new-ubuntu-installs-could-be-speed-up-by-10-with-the-zstd-compression-algorithm-520177.shtml

|website=Softpedia

|access-date=13 August 2018

|date=Mar 12, 2018

|archive-date=6 October 2021

|archive-url=https://web.archive.org/web/20211006045638/https://news.softpedia.com/news/new-ubuntu-installs-could-be-speed-up-by-10-with-the-zstd-compression-algorithm-520177.shtml

|url-status=live

}}

{{cite web

|title=Debian Changelog for apt

|url=https://metadata.ftp-master.debian.org/changelogs//main/a/apt/apt_1.8.2.3_changelog

|website=Debian

|access-date=7 November 2022

|language=en

|date=19 April 2021

}}

Fedora added ZStandard support to RPM in May 2018 (Fedora release 28) and used it for packaging the release in October 2019 (Fedora 31).{{cite web |title=Changes/Switch RPMS to ZSTD compression |url=https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression |website=Fedora Project Wiki |access-date=8 July 2020 |archive-date=2 June 2019 |archive-url=https://web.archive.org/web/20190602192906/https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression |url-status=live }} In Fedora 33, the filesystem is compressed by default with zstd.{{cite web |date=2021-04-14 |title=Fedora Workstation 34 feature focus: Btrfs transparent compression |url=https://fedoramagazine.org/fedora-workstation-34-feature-focus-btrfs-transparent-compression/ |access-date=2022-05-12 |website=Fedora Magazine |language=en-US}}{{cite web |title=Changes/BtrfsTransparentCompression |url=https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression |access-date=2022-05-12 |website=Fedora Project Wiki}}

Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager{{cite web |last=Larabel |first=Michael |date=2019-10-16 |title=Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs |url=https://www.phoronix.com/scan.php?page=news_item&px=Arch-Linux-Pacman-Zstd-Near |website=Phoronix |access-date=21 October 2019 |archive-date=18 March 2022 |archive-url=https://web.archive.org/web/20220318032712/https://www.phoronix.com/scan.php?page=news_item&px=Arch-Linux-Pacman-Zstd-Near |url-status=live }} and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -; the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, and compression memory increased but scales with the number of threads used.{{cite web |url=https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ |title=Now using Zstandard instead of xz for package compression |last=Broda |first=Mara |date=2020-01-04 |website=Arch Linux |access-date=2020-01-05 |archive-date=18 March 2022 |archive-url=https://web.archive.org/web/20220318032814/https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ |url-status=live }}{{cite mailing list |url=https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html |mailing-list=arch-dev-public |title=RFC: (devtools) Changing default compression method to zstd |first=Mara |last=Broda |date=2019-03-25 |access-date=5 January 2020 |archive-date=17 August 2021 |archive-url=https://web.archive.org/web/20210817122817/https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html |url-status=live }}

{{cite web |last1=Broda |first1=Mara |last2=Polyak |first2=Levente |date=2019-12-27 |title=makepkg.conf: change default compression method to zstd |website=GitHub |url=https://github.com/archlinux/devtools/commit/bcda211dd86b3bf54a9bc40d2e19f1aad4bbfbb8}}

Arch Linux later also switched to zstd as the default compression algorithm for mkinitcpio initial ramdisk generator.{{cite web|last=Razzolini|first=Giancarlo|date=2021-02-19|title=News: Moving to Zstandard images by default on mkinitcpio|url=https://archlinux.org/news/moving-to-zstandard-images-by-default-on-mkinitcpio/|access-date=2021-12-28|website=Arch Linux}}

A full implementation of the algorithm with an option to choose the compression level is used in the .NSZ/.XCZ{{cite web |url=https://gbatemp.net/threads/nszip-nsp-compressor-decompressor-to-reduce-storage.530313/ |title=RELEASE - nsZip - NSP compressor/decompressor to reduce storage |website=GBAtemp.net - The Independent Video Game Community |date=20 October 2019 |language=en-US |access-date=2019-11-03 |archive-date=15 August 2021 |archive-url=https://web.archive.org/web/20210815051318/https://gbatemp.net/threads/nszip-nsp-compressor-decompressor-to-reduce-storage.530313/ |url-status=live }} file formats developed by the homebrew community for the Nintendo Switch hybrid game console.{{Citation |last=Bosshard |first=Nico |title=nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip |date=2019-10-31 |url=https://github.com/nicoboss/nsZip |access-date=2019-11-03 |archive-date=27 March 2022 |archive-url=https://web.archive.org/web/20220327151207/https://github.com/nicoboss/nsZip |url-status=live }} It is also one of many supported compression algorithms in the .RVZ Wii and GameCube disc image file format.

On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.{{citation |url=http://www.pkware.com/documents/APPNOTE/APPNOTE-6.3.8.TXT |title=APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.8 |date=15 June 2020 |access-date=7 July 2020 }}{{citation |url=http://www.pkware.com/documents/APPNOTE/APPNOTE-6.3.7.TXT |title=APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.7 |date=1 June 2020 |access-date=6 June 2020 }}

In March 2024, Google Chrome version 123 (and Chromium-based browsers such as Brave or Microsoft Edge) added zstd support in the HTTP header Content-Encoding.{{Cite web |date=2024-03-19 |orig-date= |title=New in Chrome 123 {{!}} Chrome Blog |url=https://developer.chrome.com/blog/new-in-chrome-123 |access-date=2024-04-16 |website=Chrome for Developers |language=en}} In May 2024, Firefox release 126.0 added zstd support in the HTTP header Content-Encoding.{{cite web |title=Firefox 126.0, See All New Features, Updates and Fixes |url=https://www.mozilla.org/en-US/firefox/126.0/releasenotes/ |access-date=15 May 2024 |archive-date=13 May 2024 |archive-url=https://web.archive.org/web/20240513202356/https://www.mozilla.org/en-US/firefox/126.0/releasenotes/ |url-status=live }}

The reference implementation is licensed under the BSD license, published at GitHub.{{cite news|url=https://www.zdnet.com/article/facebook-open-sources-zstandard-data-compression-algorithm-aims-to-replace-technology-behind-zip/|title=Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip|date=August 31, 2016|publisher=ZDnet|access-date=2016-09-01}} Since version 1.0, published 31 August 2016,{{Cite web |title=Zstandard v1.0 |url=https://github.com/facebook/zstd/releases/tag/v1.0.0 |access-date=2025-01-23 |date=2016-08-31 |website=GitHub |archive-date=7 April 2023 |archive-url=https://web.archive.org/web/20230407231122/https://github.com/facebook/zstd/releases/tag/v1.0.0 |url-status=live }} it had an additional Grant of Patent Rights.{{Cite web |title=v1.3.0/PATENTS · facebook/zstd |url=https://github.com/facebook/zstd/blob/v1.3.0/PATENTS |access-date=2024-03-27 |date=2016-08-30 |website=GitHub |language=en |archive-date=15 May 2021 |archive-url=https://web.archive.org/web/20210515045048/https://github.com/facebook/zstd/blob/v1.3.0/PATENTS |url-status=live }}

From version 1.3.1, released 20 August 2017,{{Cite web |title=Release Zstandard v1.3.1 · facebook/zstd |url=https://github.com/facebook/zstd/releases/tag/v1.3.1 |access-date=2024-03-27 |date=2017-08-20 |website=GitHub |language=en |archive-date=12 September 2020 |archive-url=https://web.archive.org/web/20200912235153/https://github.com/facebook/zstd/releases/tag/v1.3.1 |url-status=live }} this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.{{Cite web |title=New license by Cyan4973 · Pull Request #801 · facebook/zstd |url=https://github.com/facebook/zstd/pull/801 |access-date=2024-03-27 |date=2017-08-19 |website=GitHub |language=en |archive-date=12 September 2020 |archive-url=https://web.archive.org/web/20200912201251/https://github.com/facebook/zstd/pull/801 |url-status=live }}

• LZ4 (compression algorithm) – a fast member of the LZ77 family
• LZFSE – a similar algorithm by Apple used since iOS 9 and OS X 10.11 and made open source on 1 June 2016
• Zlib
• Brotli – also integrated into browsers

{{Reflist|colwidth=30em}}

• {{Official website}}
• {{Github|facebook/zstd}}
• {{GitHub|mcmilk/7-Zip-zstd|7zip with Zstandard}}
• "[https://engineering.fb.com/2016/08/31/core-infra/smaller-and-faster-data-compression-with-zstandard/ Smaller and faster data compression with Zstandard]", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
• [https://www.theguardian.com/info/developer-blog/2016/dec/01/discover-new-compression-iinovations-brotli-and-zstandard The Guardian is using ZStandard instead of zlib]

{{Compression methods}}{{Archive formats}}

Category:2016 software

Category:C (programming language) libraries

Category:Free data compression software

Category:Lossless compression algorithms

Category:Software using the BSD license

Category:Data compression