Bufferbloat

{{See also|CoDel#Theoretical underpinnings}}

An established rule of thumb for the network equipment manufacturers was to provide buffers large enough to accommodate at least 250 ms of buffering for a stream of traffic passing through a device. For example, a router's Gigabit Ethernet interface would require a relatively large 32 MB buffer.{{cite web | url = http://conferences.sigcomm.org/sigcomm/2004/papers/p277-appenzeller1.pdf | title = Sizing Router Buffers | year = 2004 | accessdate = 2013-10-15 | author1 = Guido Appenzeller | author2 = Isaac Keslassy | author3 = Nick McKeown | work = ACM SIGCOMM | publisher = ACM}} Such sizing of the buffers can lead to failure of the TCP congestion control algorithm. The buffers then take some time to drain, before congestion control resets and the TCP connection ramps back up to speed and fills the buffers again.{{cite web | url = http://queue.acm.org/detail.cfm?id=2209336 | title = Controlling Queue Delay | authorlink = Kathleen Nichols | last1 = Nichols | first1 = Kathleen | authorlink2 = Van Jacobson | last2 = Jacobson | first2 = Van | work = ACM Queue | publisher = ACM Publishing | date = 2012-05-06 | accessdate = 2013-09-27}} Bufferbloat thus causes problems such as high and variable latency, and choking network bottlenecks for all other flows as the buffer becomes full of the packets of one TCP stream and other packets are then dropped.{{citation | url = http://www.computer.org/portal/web/csdl/doi/10.1109/MIC.2011.56 | title = Bufferbloat: Dark Buffers in the Internet | last = Gettys | first = Jim | publisher = IEEE | date = May–June 2011 | accessdate = 2012-02-20 | series = IEEE Internet Computing | pages = 95–96 | volume = 15 | issue = 3 | doi = 10.1109/MIC.2011.56 | archive-url = https://web.archive.org/web/20121012012532/http://www.computer.org/portal/web/csdl/doi/10.1109/MIC.2011.56 | archive-date = October 12, 2012 | url-status = dead }}

A bloated buffer has an effect only when this buffer is actually used. In other words, oversized buffers have a damaging effect only when the link they buffer becomes a bottleneck. The size of the buffer serving a bottleneck can be measured using the ping utility provided by most operating systems. First, the other host should be pinged continuously; then, a several-seconds-long download from it should be started and stopped a few times. By design, the TCP congestion avoidance algorithm will rapidly fill up the bottleneck on the route. If downloading (and uploading, respectively) correlates with a direct and important increase of the round trip time reported by ping, then it demonstrates that the buffer of the current bottleneck in the download (and upload, respectively) direction is bloated. Since the increase of the round trip time is caused by the buffer on the bottleneck, the maximum increase gives a rough estimation of its size in milliseconds.{{cn|date=March 2020}}

In the previous example, using an advanced traceroute tool instead of the simple pinging (for example, MTR) will not only demonstrate the existence of a bloated buffer on the bottleneck, but will also pinpoint its location in the network. Traceroute achieves this by displaying the route (path) and measuring transit delays of packets across the network. The history of the route is recorded as round-trip times of the packets received from each successive host (remote node) in the route (path).{{cite web | url = http://linux.die.net/man/8/traceroute | title = traceroute(8) – Linux man page | accessdate = 2013-09-27 | website = die.net}}

{{See also|TCP tuning#Window size|TCP congestion control#Slow start}}

Most TCP congestion control algorithms rely on measuring the occurrence of packet drops to determine the available bandwidth between two ends of a connection. The algorithms speed up the data transfer until packets start to drop, then slow down the transmission rate. Ideally, they keep adjusting the transmission rate until it reaches an equilibrium speed of the link. So that the algorithms can select a suitable transfer speed, the feedback about packet drops must occur in a timely manner. With a large buffer that has been filled, the packets will arrive at their destination, but with a higher latency. The packets were not dropped, so TCP does not slow down once the uplink has been saturated, further filling the buffer. Newly arriving packets are dropped only when the buffer is fully saturated. Once this happens TCP may even decide that the path of the connection has changed, and again go into the more aggressive search for a new operating point.{{cite journal | url = http://www.cord.edu/faculty/zhang/cs345/assignments/researchPapers/congavoid.pdf | title = Congestion avoidance and control | journal = ACM SIGCOMM Computer Communication Review | author1 = Jacobson, Van | author2 = Karels, MJ | year = 1988 | volume = 18 | issue = 4 | pages = 314–329 | doi = 10.1145/52325.52356 | url-status = dead | archiveurl = https://web.archive.org/web/20040622215331/http://www.cord.edu/faculty/zhang/cs345/assignments/researchPapers/congavoid.pdf | archivedate = 2004-06-22 }}

Packets are queued within a network buffer before being transmitted; in problematic situations, packets are dropped only if the buffer is full. On older routers, buffers were fairly small so they filled quickly and therefore packets began to drop shortly after the link became saturated, so the TCP protocol could adjust and the issue would not become apparent. On newer routers, buffers have become large enough to hold several seconds of buffered data. To TCP, a congested link can appear to be operating normally as the buffer fills. The TCP algorithm is unaware the link is congested and does not start to take corrective action until the buffer finally overflows and packets are dropped.

All packets passing through a simple buffer implemented as a single queue will experience similar delay, so the latency of any connection that passes through a filled buffer will be affected. Available channel bandwidth can also end up being unused, as some fast destinations may not be promptly reached due to buffers clogged with data awaiting delivery to slow destinations. These effects impair interactivity of applications using other network protocols, including UDP used in latency-sensitive applications like VoIP and online gaming.{{cite web | url = http://www.bufferbloat.net/projects/bloat/wiki/TechnicalIntro | title = Technical Introduction to Bufferbloat | accessdate = 2013-09-27 | website = Bufferbloat.net}}{{sps|date=April 2020|certain=y}}

Regardless of bandwidth requirements, any type of a service which requires consistently low latency or jitter-free transmission can be affected by bufferbloat. Such services include digital voice calls (VOIP), online gaming, video chat, and other interactive applications such as radio streaming, video on demand, and remote login.

When the bufferbloat phenomenon is present and the network is under load, even normal web page loads can take many seconds to complete, or simple DNS queries can fail due to timeouts.{{cite journal |title=Bufferbloat: Dark Buffers in the Internet |last1=Gettys |first1=Jim |last2=Nichols |first2=Kathleen |publisher=ACM |date=January 2012 |journal=Communications of the ACM |pages=57–65 |volume=55 |issue=1 |doi=10.1145/2063176.2063196 |doi-access=free }} Actually any TCP connection can timeout and disconnect, and UDP packets can get discarded. Since the continuation of a TCP download stream depends on acknowledgement (ACK) packets in the upload stream, a bufferbloat problem in the upload can cause failure of other non-related download applications, because the client ACK packets do not timely reach the internet server.

The DSLReports Speedtest{{cite web|url=http://dslreports.com/speedtest|title=Speed test - how fast is your internet?|work=dslreports.com|accessdate=26 October 2017}}{{Dead link|date=April 2025}} was an easy-to-use test that included a score for bufferbloat. The DSLReports site went fully offline on March 26, 2025. The ICSI Netalyzr{{cite web|url=http://netalyzr.icsi.berkeley.edu/|title=ICSI Netalyzr|work=berkeley.edu|accessdate=30 January 2015|archive-url=https://web.archive.org/web/20190407211303/http://netalyzr.icsi.berkeley.edu/|archive-date=April 7, 2019|url-status=dead}} was another on-line tool that could be used for checking networks for the presence of bufferbloat, together with checking for many other common configuration problems.{{cite web|url=https://www.newscientist.com/article/dn18953-understanding-your-netalyzr-results|title=Understanding your Netalyzr results|accessdate=26 October 2017}} The service was shut down in March 2019. The bufferbloat.net web site lists tools and procedures for determining whether a connection has excess buffering that will slow it down.{{cite web|url=https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/|title=Tests for Bufferbloat|work=bufferbloat.net|accessdate=26 October 2017}}{{sps|date=April 2020|certain=y}}{{cite web|url=https://www.bufferbloat.net/projects/bloat/wiki/Introduction/|title=Introduction to Bufferbloat|work=bufferbloat.net|accessdate=8 May 2023}}

Several technical solutions exist which can be broadly grouped into two categories: solutions that target the network and solutions that target the endpoints. The two types of solutions are often complementary. The problem sometimes arrives with a combination of fast and slow network paths.

Network solutions generally take the form of queue management algorithms. This type of solution has been the focus of the IETF AQM working group.{{cite web|url=https://datatracker.ietf.org/wg/aqm/about/|title=IETF AQM working group|work=ietf.org|accessdate=26 October 2017}} Notable examples include:

• Limiting the IP queue length, see TCP tuning
• AQM algorithms such as CoDel and PIE.{{Cite conference| last1 = Pan| first1 = Rong| last2 = Natarajan| first2 = Preethi| last3 = Piglione| first3 = Chiara| last4 = Prabhu| first4 = Mythili| last5 = Subramanian| first5 = Vijay| last6 = Baker| first6 = Fred| last7 = VerSteeg| first7 = Bill| title = 2013 IEEE 14th International Conference on High Performance Switching and Routing (HPSR)| chapter = PIE: A Lightweight Control Scheme To Address the Bufferbloat Problem| date = 2013|series = 2013 IEEE 14th International Conference on High Performance Switching and Routing (HPSR) | pages = 148–155| publisher = IEEE| doi = 10.1109/HPSR.2013.6602305| isbn = 978-1-4673-4620-7}}
• Hybrid AQM and packet scheduling algorithms such as FQ-CoDel.{{Cite IETF | last1 = Høiland-Jørgensen| first1 = Toke| last2 = McKenney| first2 = Paul| last3 = Taht| first3 = Dave| last4 = Gettys| first4 = Jim| last5 = Dumazet| first5 = Eric| title = The FlowQueue-CoDel Packet Scheduler and Active Queue Management Algorithm| rfc = 8290 }}
• Amendments to the DOCSIS standard{{cite web|url=http://www.cablelabs.com/specs |title=DOCSIS "Upstream Buffer Control" feature|publisher=CableLabs|pages=554–556|accessdate=2012-08-09}} to enable smarter buffer control in cable modems.
• Integration of queue management (FQ-CoDel) into the Wi-Fi subsystem of the Linux operating system as Linux is commonly used in wireless access points.{{Cite conference| publisher = USENIX - The Advanced Computing Systems Association| isbn = 978-1-931971-38-6| conference = 2017 USENIX Annual Technical Conference (USENIX ATC 17)| pages = 139–151| last1 = Høiland-Jørgensen| first1 = Toke| last2 = Kazior| first2 = Michał| last3 = Täht| first3 = Dave| last4 = Hurtig| first4 = Per| last5 = Brunstrom| first5 = Anna| title = Ending the Anomaly: Achieving Low Latency and Airtime Fairness in WiFi| accessdate = 2017-09-28| date = 2017| url = https://www.usenix.org/conference/atc17/technical-sessions/presentation/hoilan-jorgesen}} [https://www.cs.kau.se/tohojo/airtime-fairness/ source code].

Notable examples of solutions targeting the endpoints are:

• The BBR congestion control algorithm for TCP.
• The Micro Transport Protocol employed by many BitTorrent clients.
• Techniques for using fewer connections, such as HTTP pipelining or HTTP/2 instead of the plain HTTP protocol.

The problem may also be mitigated by reducing the buffer size on the OS and network hardware; however, this is often not configurable and optimal buffer size is dependent on line rate which may differ for different destinations.

Utilizing DiffServ (and employing multiple priority-based queues) helps in prioritizing transmission of low-latency traffic (such as VoIP, videoconferencing, gaming), relegating dealing with congestion and bufferbloat onto non-prioritized traffic. {{Cite web|last=Hein|first=Mathias|title=Bufferbloat » ADMIN Magazine|url=http://www.admin-magazine.com/Archive/2020/57/Bufferbloat-causes-and-management|access-date=2020-06-11|website=ADMIN Magazine|language=en-US}}

For the longest delay TCP connections to still get their fair share of the bandwidth, the buffer size should be at least the bandwidth-delay product divided by the square root of the number of simultaneous streams.{{Cite web |last=Huston |first=Geoff |date=2019-12-12 |title=Sizing the buffer |url=https://blog.apnic.net/2019/12/12/sizing-the-buffer/ |access-date=2022-10-16 |website=APNIC Blog |language=en-US}} A typical rule of thumb is 50 ms of line rate data,{{Cite web |title=Router/Switch Buffer Size Issues |url=https://fasterdata.es.net/network-tuning/router-switch-buffer-size-issues/ |access-date=2022-10-16 |website=fasterdata.es.net}} but some popular consumer grade switches only have 1 ms,{{Cite web |title=BCM53115 |url=https://www.broadcom.com/products/ethernet-connectivity/switching/roboswitch/bcm53115 |access-date=2022-10-16 |website=www.broadcom.com |language=en}} which may result in extra bandwidth loss on the longer delay connections in case of local contention with others.

• Bandwidth-delay product
• Congestion window
• Explicit Congestion Notification
• Head-of-line blocking
• L4S, an end-to-end solution to bufferbloat
• Packet loss

{{Reflist}}

• [http://queue.acm.org/detail.cfm?id=2076798 BufferBloat: What's Wrong with the Internet?] A discussion with Vint Cerf, Van Jacobson, Nick Weaver, and Jim Gettys
• {{YouTube|qbIozKVz73g|Google Tech Talk}} April, 2011, by Jim Gettys, introduction by Vint Cerf
• {{YouTube|npiG7EBzHOU|Bufferbloat: Dark Buffers in the Internet — Demonstrations Only}} April, 2011, by Jim Gettys, introduction by Vint Cerf
• {{YouTube|-D-cJNtKwuw|Bufferbloat: Dark Buffers in the Internet — Demonstrations and Discussions}} 21 minute demonstration and explanation of typical broadband bufferbloat
• {{YouTube|uQ9ziQg_1zU|LACNIC - BufferBloat}} May 2012, by Fred Baker (IETF chair) in Spanish, English slides available [https://www.dropbox.com/sh/ihvbcuxfs2t8kp8/2xoXrlSeq9/Fred_Baker_LACNIC%20Wednesday.pptx]
• [https://lwn.net/Articles/564978/ TSO sizing and the FQ scheduler] (Jonathan Corbet, LWN.net)

Category:Flow control (data)

Category:Internet architecture

Category:Network performance