Jerry Labs

Performance notes

BBR vs Cubic: Test the Socket You Actually Care About

A sysctl value is only a default. A useful comparison proves the active socket, controls test order, and records latency, CPU, retransmissions, and timeouts.

Reviewed 30 July 20265 min readPerformance

First, separate TCP from UDP

Linux congestion control under net.ipv4.tcp_congestion_control applies to TCP sockets. It does not change the controller inside a UDP or QUIC application. If the workload is UDP, a kernel BBR-versus-Cubic test is measuring the wrong layer.

For TCP, confirm what the kernel offers and what new sockets will inherit.

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control
sysctl net.core.default_qdisc

BBR is commonly paired with fq; Cubic is often tested with fq_codel. Treat the controller and queue discipline as one named profile, and record both.

Prove the active flow

Changing a sysctl does not rewrite existing TCP sockets. Start the client after applying the profile, then inspect the established flow.

sudo ss -tinp 'sport = :9443'
iface=$(ip route show default | awk 'NR == 1 {print $5}')
tc -j qdisc show dev "$iface"

The ss output names the congestion controller for each TCP flow. The qdisc query confirms the live root queue on the interface. A test should fail if the expected flow is missing, several unrelated flows match, or the socket reports the wrong controller.

Use an A-B-B-A sequence

Network conditions drift. Running every Cubic round in the morning and every BBR round later makes time of day a hidden variable. An A-B-B-A block alternates profiles and gives both candidates early and late samples.

Use the same endpoint, payload sizes, direction, client, and concurrency. Warm the connection before recording. Run at least three measured rounds per profile, and create a new TCP connection after each profile change.

nstat -az | grep -E 'TcpRetransSegs|TcpExtTCPTimeouts|IpInDiscards'
ss -tinm
pidstat -p "$(pidof service-name)" 1

Capture throughput, duration, idle latency, latency under load, loss under load, system CPU, service CPU, retransmissions, and timeouts. ICMP may be deprioritized, so ping loss is comparative evidence rather than a complete account of application loss.

Choose by the distribution, not the peak

A single fast round is not a decision. Compare median throughput first, then the spread between rounds. Reject a profile that buys speed with unstable latency, repeated timeouts, or excessive CPU. Keep raw rows so an outlier remains visible; averages hide too much.

After choosing, persist the profile in a dedicated sysctl drop-in. Reboot, then repeat the runtime, root-qdisc, socket, and transfer checks. That reboot is the difference between "configured" and "survives startup."

Benchmark checklist

  • Confirm the workload is TCP.
  • Test controller and qdisc as a named pair.
  • Verify the active socket with ss -tin.
  • Alternate profiles with an A-B-B-A order.
  • Run at least three rounds per profile.
  • Keep throughput, latency, loss, CPU, retransmission, and timeout data.
  • Reboot and repeat before committing the profile.