sync (Unix)
{{Short description|Unix command to commit all data in the kernel filesystem to non-volatile storage buffers}}
{{DISPLAYTITLE:sync (Unix)}}
sync is a standard system call in the Unix operating system, which commits all data from the kernel filesystem buffers to non-volatile storage, i.e., data which has been scheduled for writing via low-level I/O system calls. Higher-level I/O layers such as stdio may maintain separate buffers of their own.
As a function in C, the sync() call is typically declared as void sync(void) in
The related system call fsync() commits just the buffered data relating to a specified file descriptor.[http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html fsync specification] fdatasync() is also available to write out just the changes made to the data in the file, and not necessarily the file's related metadata.[http://pubs.opengroup.org/onlinepubs/9699919799/functions/fdatasync.html fdatasync specification]
Some Unix systems run a kind of flush or update daemon, which calls the sync function on a regular basis. On some systems, the cron daemon does this, and on Linux it was handled by the pdflush daemon which was replaced by a new implementation and finally removed from the Linux kernel in 2012.{{Cite web|url=http://lwn.net/Articles/508212/|title = R.I.P. Pdflush [LWN.net]}} Buffers are also flushed when filesystems are unmounted or remounted read-only,{{Cite web|title=mount - Does umount calls sync to complete any pending writes|url=https://unix.stackexchange.com/questions/345917/does-umount-calls-sync-to-complete-any-pending-writes|access-date=2021-05-02|website=Unix & Linux Stack Exchange}} for example prior to system shutdown.
Some applications, such as LibreOffice, also call the sync function to save recovery information in an interval.
Database use
In order to provide proper durability, databases need to use some form of sync in order to make sure the information written has made it to non-volatile storage rather than just being stored in a memory-based write cache that would be lost if power failed. PostgreSQL for example may use a variety of different sync calls, including fsync() and fdatasync(),{{ cite web | url = https://ftp.osuosl.org/pub/fosdem/2019/K.1.105/postgresql_fsync.mp4 | title = PostgreSQL vs. fsync | access-date = 10 February 2019 | last = Vondra | first = Tomas | date = 2 February 2019 | website = Osuosl Org. | format = mp4 | archive-url = https://web.archive.org/web/20190210111845/https://ftp.osuosl.org/pub/fosdem/2019/K.1.105/postgresql_fsync.mp4 | archive-date = 10 February 2019 | df = dmy-all }} in order for commits to be durable.[http://www.postgresql.org/docs/current/static/wal.html PostgreSQL Reliability and the Write-Ahead Log] Unfortunately, for any single client writing a series of records, a rotating hard drive can only commit once per rotation, which makes for at best a few hundred such commits per second.[http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm Tuning PostgreSQL WAL Synchronization] {{Webarchive|url=https://web.archive.org/web/20091125043111/http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm |date=2009-11-25 }} Turning off the fsync requirement can therefore greatly improve commit performance, but at the expense of potentially introducing database corruption after a crash.
Databases also employ transaction log files (typically much smaller than the main data files) that have information about recent changes, such that changes can be reliably redone in case of crash; then the main data files can be synced less often.
Error reporting and checking
To avoid any data loss return values of fsync() should be checked because when performing I/O operations that are buffered by the library or the kernel, errors may not be reported at the time of using the write() system call or the fflush() call, since the data may not be written to non-volatile storage but only be written to the memory page cache. Errors from writes are instead often reported during system calls to fsync(), msync() or close().{{Cite web|url=https://lwn.net/Articles/457667/|title = Ensuring data reaches disk [LWN.net]}} Prior to 2018, Linux's fsync() behavior under certain circumstances failed to report error status,{{Cite web|url=https://lwn.net/Articles/752063/|title = PostgreSQL's fsync() surprise [LWN.net]}}{{Cite web|url=https://lwn.net/Articles/724307/|title = Improved block-layer error handling [LWN.net]}} change behavior was proposed on 23 April 2018.{{Cite web |url=https://patchwork.kernel.org/patch/10358111/ |title=Always report a writeback error once - Patchwork |access-date=2018-05-03 |archive-date=2018-05-04 |archive-url=https://web.archive.org/web/20180504092257/https://patchwork.kernel.org/patch/10358111/ |url-status=dead }}
Performance controversies
Hard disks may default to using their own volatile write cache to buffer writes, which greatly improves performance while introducing a potential for lost writes.[http://www.jasonbrome.com/blog/archives/2004/04/03/writecache_enabled.html Write-Cache Enabled?] Tools such as hdparm -F will instruct the HDD controller to flush the on-drive write cache buffer. The performance impact of turning caching off is so large that even the normally conservative FreeBSD community rejected disabling write caching by default in FreeBSD 4.3.[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-disk.html#AEN16830 FreeBSD Handbook — Tuning Disks]
In SCSI and in SATA with Native Command Queuing (but not in plain ATA, even with TCQ) the host can specify whether it wants to be notified of completion when the data hits the disk's platters or when it hits the disk's buffer (on-board cache). Assuming a correct hardware implementation, this feature allows the disk's on-board cache to be used while guaranteeing correct semantics for system calls like fsync.{{cite web|author=Marshall Kirk McKusick|author-link=Marshall Kirk McKusick|url=http://queue.acm.org/detail.cfm?id=2367378 |title=Disks from the Perspective of a File System - ACM Queue |publisher=Queue.acm.org |access-date=2014-01-11}} This hardware feature is called Force Unit Access (FUA) and it allows consistency with less overhead than flushing the entire cache as done for ATA (or SATA non-NCQ) disks.{{cite book|author=Gregory Smith|title=PostgreSQL 9.0: High Performance|year=2010|publisher=Packt Publishing Ltd|isbn=978-1-84951-031-8|page=78}} Although Linux enabled NCQ around 2007, it did not enable SATA/NCQ FUA until 2012, citing lack of support in the early drives.{{Cite web|url=http://www.spinics.net/lists/linux-scsi/msg61241.html|title = Enabling FUA for SATA drives (Was Re: [RFC][PATCH] libata: Enable SATA disk fua detection on default) (Linux SCSI)}}{{Cite web|url=http://lkml.indiana.edu/hypermail/linux/kernel/0702.2/1358.html|title=Linux-Kernel Archive: [PATCH RFC] libata: FUA updates}}
Firefox 3.0, released in 2008, introduced fsync system calls that were found to degrade its performance; the call was introduced in order to guarantee the integrity of the embedded SQLite database.{{Cite web|url=http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/|title=Shaver » fsyncers and curveballs|access-date=2009-10-15|archive-date=2012-12-09|archive-url=https://web.archive.org/web/20121209115158/http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/|url-status=dead}}
Linux Foundation chief technical officer Theodore Ts'o claims there is no need to "fear fsync", and that the real cause of Firefox 3 slowdown is the excessive use of fsync.{{Cite web|url=http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/|title = Don't fear the fsync!}} He also concedes however (quoting Mike Shaver) that
On some rather common Linux configurations, especially using the ext3 filesystem in the "data=ordered" mode, calling fsync doesn't just flush out the data for the file it's called on, but rather on all the buffered data for that filesystem.{{Cite web|url=http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/|title = Delayed allocation and the zero-length file problem}}
See also
References
{{Reflist}}
External links
- [http://linux.die.net/man/8/sync sync(8) - Linux man page]
- http://austingroupbugs.net/view.php?id=672
{{Core Utilities commands}}
Category:Standard Unix programs