The QUIC Bug That Trapped CUBIC's Congestion Window

CUBIC, the default congestion controller in Linux and the foundation for most TCP and QUIC connections on the internet, governs how connections probe bandwidth and respond to loss. At Cloudflare, our QUIC implementation (quiche) relies on CUBIC. However, a subtle bug—born from a Linux kernel optimization meant to align CUBIC with RFC 9438's app‑limited exclusion—caused the congestion window to become permanently stuck at its minimum after a severe loss event. This bug eluded detection until erratic test failures revealed its presence. Below, we explore the problem through common questions and answers.

What exactly is CUBIC and why is it so widely used?

CUBIC is a loss‑based congestion control algorithm standardized in RFC 9438. It's the default in Linux, making it the algorithm behind the majority of TCP and QUIC connections on the public internet. The central mechanism is a congestion window (cwnd): a cap on how many bytes can be in flight at once. When no packet loss occurs, CUBIC aggressively grows cwnd to probe for available bandwidth. When loss is detected, it assumes network capacity has been exceeded and sharply reduces cwnd. This balance between growth and back‑off aims to maximize throughput while avoiding congestion collapse.

The QUIC Bug That Trapped CUBIC's Congestion Window — Source: blog.cloudflare.com

What was the specific bug discovered in Cloudflare's QUIC implementation?

The bug caused CUBIC's congestion window to become permanently pinned at its minimum value after a severe loss event, preventing recovery. Normally, after a loss, the algorithm should gradually increase cwnd again. But due to a side effect of the app‑limited exclusion logic (defined in RFC 9438 §4.2‑1/2), the cwnd never got a chance to grow because the sender appeared to be application‑limited (i.e., it had no more data to send), even though data was available. This created a feedback loop: the cwnd stayed low, the sender never sent enough to trigger growth, and the connection remained stuck in a low‑throughput state.

How did the bug first appear in testing?

The issue surfaced when our integration tests for the ingress proxy started failing erratically—about 61% of the time. These tests specifically examined CUBIC's behavior under high packet loss early in a connection. Most congestion control tests focus on steady‑state performance; few probe the recovery phase after a congestion collapse. This unusual regime is precisely where the bug lived. The tests would simulate heavy loss, then expect the algorithm to recover and ramp up throughput. Instead, connections remained crippled, failing the check.

What is the app‑limited exclusion and why did it create problems in QUIC?

The app‑limited exclusion is a rule in RFC 9438 §4.2‑1/2 that prevents a congestion controller from increasing its cwnd when the sender is application‑limited (i.e., it doesn't have enough data to fill the window). In TCP, this makes perfect sense—you shouldn't grow the window if the application can't use it. However, when ported to QUIC (which runs over UDP and has a different pacing model), the same logic could misclassify a sender as app‑limited during loss recovery. Because QUIC's pacing and loss detection differ, the sender might briefly have no pending data, triggering the exclusion at the worst possible time—when the cwnd should be growing after a loss.

How does CUBIC's congestion window logic normally handle loss and recovery?

CUBIC uses a cubic function of real time to determine the target cwnd after a loss event. Upon detecting loss, it records a recovery point and reduces cwnd by a factor (typically 0.7). Then, during recovery, it holds cwnd constant. After recovery ends, if no further loss is seen, CUBIC starts growing cwnd according to the cubic curve. The key is that growth only resumes when the network appears healthy and the sender is not app‑limited. The bug exploited a situation where, due to the app‑limited exclusion, the sender appeared app‑limited even after recovery, so growth never started.

What was the fix—and why was it so elegant?

The solution was a near‑one‑line change: we adjusted the condition that triggered app‑limited mode during recovery. Specifically, we ensured that after a congestion collapse, the sender is not considered app‑limited for a brief window, allowing cwnd to grow. The fix was minimal because the core algorithm was sound; the bug was a subtle interaction between the QUIC implementation and the app‑limited exclusion logic. By tweaking the timing of when we apply the exclusion, the window was freed to climb back to normal levels. This small change turned a 61% failure rate into consistent success.

Why is it important for a congestion controller to handle recovery from severe loss?

While severe loss (congestion collapse) is uncommon, it is exactly the scenario a congestion controller exists to handle. If an algorithm cannot recover from the minimum cwnd, the connection becomes useless or experiences extreme latency. In production at Cloudflare, many QUIC connections traverse lossy or flaky networks; bugs like this could silently degrade throughput for a significant share of traffic. Ensuring robust recovery makes the controller reliable in real‑world conditions, not just in steady‑state benchmarks.

Tags: