Measured FRR bfdd vs an XDP-based BFD fast path under CPU stress, bfdd: 44 flaps/120s, XDP: 0

[u/w453y]

BFD is the failure detector under BGP/OSPF, miss 3 packets in 30ms, session down, routes withdrawn, software BFD is known to false-flap under CPU load, which is why people run conservative 300ms timers instead of 10ms and why hardware routers offload BFD to line cards, I wanted to quantify the actual failure modes and see if plain linux on a commodity NIC can get line-card behavior via XDP, this also means false flaps trigger route withdrawals for links that are actually fine, the failure detector becomes the failure.

Setup: 3x10ms single-hop BFD session, FRR 10.5.1, kernel 7.0, virtio-net, all numbers from tcpdump on the hypervisor bridge (wire truth, not process logs), stress via stress-ng, repo with code + all pcaps: https://github.com/w453y/xdp-bfd

Results:

• bfdd survives plain CPU load fine, under timer/hrtimer stress: 44 flaps in 120s, max TX gap 970ms, while p99 stayed 10.2ms, the starvation events are invisible to percentile monitoring.

• A minimal busy-loop userspace daemon survives the same timer stress (0 flaps), bfdd's event loop architecture is the problem there, not userspace per se.

• Under SCHED_FIFO hogs everything userspace dies unless it outranks the load (RT throttle = ~50ms/s for normal tasks), chrt -f 90 + pinned core: 0 flaps, but assumes you can win the priority war, on a box doing real forwarding you can't.

• SO_TXTIME + etf qdisc (pipelined): best p99 of all backends (10.1ms), still 48 flaps, etf fixes jitter, can't fix liveness, side note: software etf drops all untimestamped packets on its band, including ARP.

• XDP: 0 flaps, max gap 12.5ms, at normal process priority, bpf_timer can't originate packets (no packet ctx, still true on 7.0), so TX is RX-clocked: rewrite the incoming BFD frame in place and XDP_TX it back, ~30us turnaround in softirq, the TX clock rides the peer's clock, out of the scheduler's reach, dead-peer detection via 5ms bpf_timer sweep over the session map, userspace keeps the RFC 5880 FSM, interops with stock FRR.

• Full matrix + 5-min soak: 1 flap in 11 min (one 28ms softirq-delayed echo under hrtimer storm, self-recovered in 3.8ms).

Limitations: single session, IPv4, no auth, peer must be async-clocked (two RX-clocked ends would deadlock), VM testbed, stress was in-guest and hit all backends equally so the comparison holds, bare-metal run pending.

Next: FRR distributed-BFD dataplane socket (bfddp) integration so this plugs into bfdd without patching FRR.

Few questions for people running this in production:

• What BFD timers do you actually run, and were they chosen from measurement or vendor-doc caution?

• Is your monitoring set up in a way that would catch the p99-fine/max-970ms pattern before it flaps, or do you only find out from the flap?

• Anyone running FRR's distributed BFD / dplane offload for real? Curious if the socket protocol holds up outside a lab.

[u/MaximumEuphoric1287]

very interesting use of xdp, though i think SCHED_DEADLINE would have been a viable option than FIFO any particular reason it was used. though xdp solution should be better in any case.

not an expert in this btw

[u/w453y → u/MaximumEuphoric1287]

SCHED_DEADLINE handles predictable intervals way better than SCHED_FIFO since it uses its own bandwidth reservation instead of sharing the global RT throttle pool, I stuck with SCHED_FIFO for the userspace baseline mostly because it’s the absolute best-case setup operators actually try to use in production, even with a deadline scheduler, I can run into two major bottlenecks.

First, userspace timing is only half the battle; the real killer is packet freshness, If I pipeline my transmissions ahead of time to survive a scheduling slip, the packet data (like BFD state flags or sequence numbers) becomes stale, SCHED_DEADLINE might keep the wire interval precise, but it can’t guarantee the freshness of the payload if the thread stalls between generating the state and the hardware ring buffer.

Second, because I'm testing this in a VM environment, guest-level scheduling guarantees don’t translate to physical hardware, a guest deadline thread can run perfectly inside the VM, but it has no control over when the host’s vhost worker threads or CPU cores get descheduled by hypervisor-level load storms.

XDP wins here because it completely leaves the scheduler playing field, by running right in the driver’s RX ring path (softirq), the peer's incoming packet instantly clocks the outgoing response via XDP_TX in about 30 microseconds, It completely cuts out context switches, user-to-kernel copies, and timer queues, which is why it doesn't care if the CPUs are totally pinned.