Why BGP? It's an app, not an IGP.
Same five routers. Same cables. Same addresses. One runs eBGP, one runs OSPF. We break one link in both and count who has to do work. One protocol makes every router redraw the whole map. The other passes a note about two addresses.
Run it yourself → this kit: sudo ./lab.sh up builds both twins · online version (with the video)
- 1. The objection
- 2. BGP is an app
- 3. Why AI fabrics run eBGP
- 4. Where OSPF / IS-IS still win
- 5. The twin topology
- 6. Configs side by side
- 7. Run it yourself
- 8. The measurement
- 9. Wireshark: TCP 179
- 10. Exercises
- 11. Recap & the kit
1The objection
"BGP is really only for network borders."
— isaacyonemoto, Discord, 10/12
That was the rule in the 1990s: an IGP (OSPF, IS-IS) inside your network, BGP at the edge to talk to other networks. It's a reasonable rule. It's also not what the biggest networks on earth do inside their AI data centers. Today we find out why, with real numbers from a real lab.
2BGP is an app, not an IGP
eBGP: an application
Rides a normal TCP session on port 179, like a web or database connection. Handshake, then messages: OPEN, KEEPALIVE, UPDATE.
Talks only to its direct neighbors.
Ships answers: "I can reach 172.16.1.0/24, here's the AS path." Only best paths, per prefix.
When something changes, it sends a per-prefix diff. Nobody recomputes the network.
OSPF / IS-IS: a link-state IGP
Its own IP protocol (OSPF = IP protocol 89, multicast to 224.0.0.5). No TCP; it built its own reliability.
Floods every router's link description (LSAs) to every router in the area.
Ships the whole map. Every router holds an identical copy.
When any link changes, every router re-runs SPF (Dijkstra) over the entire map.
The one-liner: OSPF makes every router redraw the whole map. BGP just passes a note about the prefixes that changed.
Neither is "better" in the abstract. Flooding the map is great when the network is small and you want every router to know everything. It gets expensive when there are thousands of routers and links flap all day.
3Why hyperscalers run eBGP inside AI fabrics
RFC 7938, "Use of BGP for Routing in Large-Scale Data Centers" (2016, written from Microsoft's experience), lays out the design we built in episode 1: a Clos (spine-leaf) fabric where every switch is its own BGP speaker and eBGP is the only routing protocol. Their reasons, in plain words:
- Failures stay local. A flap changes a few prefixes on a few sessions. In a link-state protocol, every flap is news to every router in the area, and every router recomputes.
- One protocol, simple state. No areas, no DR/BDR elections, no LSDB to keep in sync. Each switch only knows what its neighbors told it.
- Policy is built in. Drain a switch for maintenance, steer one prefix, filter a route: that's a route-map on one session. Link-state can only change link costs, which floods everywhere.
- Mature, boring code. Every vendor's BGP is battle-tested at internet scale, and the same skills cover the border and the fabric.
Why this matters more for AI: GPU training fabrics are huge (tens of thousands of links), traffic is synchronized (one slow path stalls the whole job), and optics fail often. You want each flap to touch as little as possible. That's the eBGP property we're about to measure.
4Fair answer: where OSPF and IS-IS still win
The objection isn't silly. Link-state protocols still have real strengths:
- Small and medium networks. A campus or a branch with 20 routers: OSPF is zero-touch, fast, and well understood. The flooding cost is tiny.
- Every router sees the whole map. That enables things BGP can't do on its own: precomputed backup paths (LFA / TI-LFA), and segment routing / traffic engineering that needs the full topology.
- Service-provider cores. Most ISP backbones run IS-IS underneath for loopback reachability and fast reroute, with BGP on top carrying customer and internet routes. That's the "BGP at the border" picture, and it's still right there.
- Convergence isn't automatically better with BGP. With BFD and tuned timers both converge in well under a second. The win is the blast radius of each change, not raw speed.
So the honest version: "BGP only at the border" was a design choice for networks where an IGP's full-map view was cheap and useful. In a giant Clos fabric the full map stops being cheap, and the per-neighbor, per-prefix model wins. Hyperscalers didn't break the rule; the size of the network did.
5The twin topology
Two labs, built side by side from the same FRR 10.5.5 image (GNS3 projects in the video; two containerlab topologies in the free kit). Same node names, same 9 cables, same loopbacks and host subnets as episode 1. Fair timers on both: 3 s hello/keepalive, 9 s dead/hold. The only difference is the routing protocol. Both twins are unnumbered: BGP peers over IPv6 link-local and carries IPv4 routes with an IPv6 next hop (RFC 8950); the OSPF twin is OSPFv2, the IPv4 version (OSPFv3 is the IPv6 one), with every fabric link borrowing the loopback /32. We flap link 2: leaf1 eth1 ↔ spine2 eth0.
| Node | Loopback / router-id | BGP twin ASN | Host LAN |
|---|---|---|---|
| spine1 | 10.0.0.1/32 | 65000 | — |
| spine2 | 10.0.0.2/32 | 65000 | — |
| leaf1 | 10.0.1.1/32 | 65101 | 172.16.1.0/24 (h1 .10) |
| leaf2 | 10.0.1.2/32 | 65102 | 172.16.2.0/24 (h2 .10) |
| leaf3 | 10.0.1.3/32 | 65103 | 172.16.3.0/24 (h3 .10) |
Cabling rule (same as episode 1): leaf eth0 → spine1, leaf eth1 → spine2, leaf eth2 → host; on a spine, ethN goes to leaf N+1.
6Configs side by side (leaf1)
Left: the episode 1 eBGP config, unchanged. Right: the OSPF twin. Both are the real files running in the lab, and both are unnumbered: no IPv4 subnets on fabric links. The only difference is the routing protocol.
eBGP twin — leaf1
OSPF twin — leaf1
onlink: that's correct.Spines: BGP spine = router bgp 65000 + neighbor eth0/eth1/eth2 interface peer-group FABRIC + its loopback. OSPF spine = the same eth0 block on eth0, eth1, eth2 (borrowing 10.0.0.N/32) + router ospf. Timers 3 / 9 everywhere.
7Run it yourself (~10 min)
The free kit builds both twins with containerlab on any Linux box with Docker. Everything below is the kit's real commands; the video does the same in GNS3.
- Build both twins. One command; it waits until both are healthy (about a minute on our server).
$ sudo ./lab.sh up [BGP twin] PASS Established: 12/12 spine1=3 spine2=3 leaf1=2 leaf2=2 leaf3=2 PASS ECMP leaf1: 172.16.2.0/24=2 172.16.3.0/24=2 10.0.1.2/32=2 10.0.1.3/32=2 ... [OSPF twin] PASS Full: 12/12 spine1=3 spine2=3 leaf1=2 leaf2=2 leaf3=2 ... ALL CHECKS PASSED - OSPF holds the map. OSPF twin, leaf1 (
docker exec -it clab-ospf-leaf1 vtysh):
Then on leaf3 and spine1:leaf1# show ip ospf neighbor # 2 Full leaf1# show ip route 172.16.3.0/24 # via 10.0.0.1 eth0 onlink + 10.0.0.2 eth1 onlinkshow ip ospf database. Same 5 Router LSAs on both. Every router carries the whole map. - BGP holds answers. BGP twin, leaf1 (
docker exec -it clab-bgp-leaf1 vtysh):
BGP carries answers, not the map.leaf1# show bgp summary # spine1(eth0), spine2(eth1): 5 PfxRcd, 8 PfxSnt leaf1# show ip route 172.16.3.0/24 # 2 next hops, fe80:: via eth0 + eth1 - Predict, then flap. Say out loud which routers you think will react. Then:
It waits for the counters to go quiet, snapshots both twins, starts 10-per-second pings, breaks link 2 (leaf1 eth1 ↔ spine2 eth0) in both twins at the same moment for 20 s, repairs it, waits 20 s, and prints the per-router change. Read$ sudo ./lab.sh flapspf_runsfirst. Section 8 has the numbers to compare. - Look at the wire. Capture a BGP session coming up, and OSPF's hellos (section 9).
- Tear down when you're done:
sudo ./lab.sh down.
How the kit breaks the link: 100% packet loss on both ends with containerlab tools netem, while both interfaces stay up, the same as GNS3's Suspend link in the video. Neither protocol gets a "link down" hint, so both find out by their timers (9 s). ip link set down would be a different, faster test. Break or repair it by hand with sudo ./lab.sh link suspend / resume.
Gotcha from the live build: after the OSPF twin was switched in place from /31s to unnumbered, it showed 12/12 Full but every ping failed: the spines still held the leaves' old /31 addresses as neighbor addresses (FRR doesn't refresh a Full neighbor's address). Fix: clear ip ospf neighbor on both spines. The kit builds fresh, so you won't hit it, but the lesson stands: control plane "perfect" doesn't mean the data plane works. Always ping. (sudo ./lab.sh check does.)
8The measurement: flap one link, count who reacts
On camera (16:40) we suspended link 2 in both twins at the same moment for 20 s, resumed it, let it settle 20 s, and diffed the counters. This is the real output:
$ # OSPF twin: change during one 20 s flap of link 2
router lsa_updates lsdb_lsas rib_routes spf_runs
spine1 88 (+4) 5 (+0) 8 (+0) 38 (+3)
spine2 88 (+4) 5 (+0) 8 (+0) 42 (+4)
leaf1 88 (+4) 5 (+0) 9 (+0) 44 (+4)
leaf2 88 (+4) 5 (+0) 9 (+0) 44 (+4)
leaf3 88 (+4) 5 (+0) 9 (+0) 38 (+4)
$ # BGP twin: same flap, same moment
router bgp_prefixes bgp_updates_rcvd bgp_updates_sent rib_routes
spine1 7 (+0) 46 (+2) 24 (+0) 7 (+0)
spine2 7 (+0) 79 (+6) 78 (+9) 7 (+0)
leaf1 8 (+0) 45 (+5) 74 (+8) 9 (+0)
leaf2 8 (+0) 30 (+2) 30 (+0) 9 (+0)
leaf3 8 (+0) 30 (+2) 30 (+0) 9 (+0)OSPF: 19 SPF runs across all 5 routers (+3 on spine1, +4 on the other four), and +4 LSA updates on every router. BGP: zero SPF runs (it doesn't have one); 17 UPDATEs in total, and spine1, which isn't on the broken link, sent none.
Before the recording, the agents ran the same flap five times with packet captures and a kernel route monitor on every router, to see who did the work. Per-router work for one flap (typical run; ranges where runs differed): live lab output
| Router | On link 2? | OSPF SPF runs | OSPF LSAs rcvd | OSPF routes changed | BGP SPF runs | BGP UPDATEs rcvd | BGP UPDATEs sent | BGP routes changed |
|---|---|---|---|---|---|---|---|---|
| spine1 (off the path) | no | +3 | 9 | 1 | 0 | +2 to +6 (loop-dropped) | 0 | 0 |
| spine2 | yes | +4 | 3 | 3 | 0 | +6 to +8 | +9 | 2 |
| leaf1 | yes | +4 | 2 | 5 | 0 | +5 | +8 to +10 | 5 |
| leaf2 | no | +4 | 5–7 | 2 | 0 | +2 | 0 to +2 | 2 |
| leaf3 | no | +4 | 7–8 | 2 | 0 | +2 | 0 to +2 | 2 |
| total | 19, all 5 | 28, all 5 | 13 | 0 | 17–23 | 17–23 | 11 |
Look at spine1. It isn't on the broken link. In OSPF it still received 9 LSAs and re-ran SPF 3 times. In BGP it sent nothing and changed no routes. Its receive counter ticks only because leaves re-advertise paths like 65101 65000 65102, which spine1 drops on sight: its own AS is already in the path.
Same result from the kit. On containerlab, sudo ./lab.sh flap on a freshly built lab printed the on-camera deltas: OSPF spf_runs +3/+4/+4/+4/+4 (19), lsa_updates +4 on every router; BGP updates_rcvd +2/+6/+5/+2/+2, updates_sent 0/+9/+8/0/0, spine1 sent 0. In other runs OSPF counted 20 (spine1 +4: SPF throttling sometimes folds two LSA arrivals into one run); BGP matched every time. Your run may differ slightly, and the absolute counters will differ; only the bracketed changes matter.
How to read it: lsa_updates +4 on every router = every router stored the same 4 new versions of the map. rib_routes is a route count, so it's +0 after recovery in both twins; it doesn't mean "untouched". Who actually rewrote routes ("routes changed" above) came from kernel ip monitor route on each router during the flap. Absolute counters depend on how long the lab has been up: always snap, then diff (lab.sh flap does).
And the packets? A wash, on purpose.
10 pings per second through the flap, in the five dry runs before the recording; only the flow that hashes onto link 2 loses anything (other flows: 0 lost in both twins).
| Run | OSPF lost (h1↔h2) | BGP lost (h1↔h3) |
|---|---|---|
| 1 | void (the stale-neighbor gotcha, section 7) | 61 |
| 2 | 70 | 60 |
| 3 | 77 | 67 |
| 4 | 76 | 86 |
| 5 | 60 | 66 |
The kit's lab.sh flap prints the same loss table for h1→h2 and h1→h3 in both twins. Our kit runs on containerlab: OSPF 68–72 lost, BGP 79–81 lost (the flows that hashed onto link 2). Loss depends on where in the 3-second hello/keepalive cycle the break lands, so expect anything from about 30 to 90 in either protocol.
Say it plainly: BGP did not converge faster. Both lost about 6–9 seconds of traffic (60–86 pings at 10 per second), because both wait for the same 9 s dead/hold timer: hello 3 / dead 9 in OSPF, keepalive 3 / hold 9 in BGP. On leaf1, spine2, leaf2 and leaf3 the forwarding changes were the same in both twins. BGP's win here is blast radius and CPU: no SPF anywhere, no map flooded to everyone, and a router off the path (spine1) doing nothing. Speed comes from BFD, in either protocol (episode 1).
Zoom-in (optional), leaf3 on both twins. OSPF show ip ospf database router 10.0.1.1: LS age 39, seq bumped, 4 links. leaf1's news reached a router that didn't need it. (Aside: unnumbered links show "Router Interface address: 0.0.1.19". That's an ifindex, not an IP.) BGP show bgp ipv4 unicast 172.16.1.0/24: 2 paths, 65000 65101 via spine1 and spine2, one fewer during the flap, then back.
leaf3# show ip ospf database # OSPF twin; spine1 shows the identical table
Link ID ADV Router Age Seq# CkSum Link count
10.0.0.1 10.0.0.1 404 0x8000000d 0xc4a9 4
10.0.0.2 10.0.0.2 38 0x80000017 0xc5a1 4 # link 2 end, re-originated
10.0.1.1 10.0.1.1 39 0x80000015 0xdaf5 4 # link 2 end, re-originated
10.0.1.2 10.0.1.2 405 0x8000000b 0xcbfc 4
10.0.1.3 10.0.1.3 404 0x8000000b 0xbffe 4Your seq numbers and ages will differ.
9Wireshark: BGP is just TCP port 179
In the video we captured inside leaf1 and opened the files in Wireshark. With the kit, capture from the host: containerlab registers each container's network namespace under its name, so the host's tcpdump can listen on leaf1's eth0 directly.
$ sudo ip netns exec clab-bgp-leaf1 timeout 15 tcpdump -i eth0 -U -w bgp.pcap 'tcp port 179' &
$ sleep 2; sudo docker exec clab-bgp-leaf1 vtysh -c 'clear bgp eth0' # fresh SYN, OPEN, UPDATEs
$ sudo ip netns exec clab-ospf-leaf1 timeout 15 tcpdump -i eth0 -U -w ospf.pcap 'ip proto 89'
$ wireshark bgp.pcap ospf.pcap # or tshark -r bgp.pcapbgp.pcap
Old session: NOTIFICATION + FIN. Then SYN / SYN-ACK to port 179 over IPv6 link-local, 4 OPEN (both sides dial; one connection wins), 11 UPDATE, KEEPALIVEs. Filter bgp.type == 2; an UPDATE carries 172.16.x.0/24 with AS path like 65000 65103. A socket, a handshake, messages. That's an app. live lab output
ospf.pcap
No TCP. 9 packets in 15 s, all OSPFv2 Hello from 10.0.x.x to 224.0.0.5, IP protocol 89. In steady state there's no LS Update at all; during a flap OSPF uses its own LS Update + LS Ack reliability because it isn't riding a transport. Plumbing, not an app. live lab output
clear bgp eth0 drops that one session for about a second; leaf1 still has eth1, so h1 keeps working. Don't run it during a flap count.
10Exercises
1. Where does the news go?
Each answer can be reproduced with the kit: sudo sh solutions/exerciseN/run.sh applies it to both twins, prints the counter diff and undoes it. Try it by hand first.
In the OSPF twin, shut leaf3's host port (interface eth2 / shutdown). Which routers re-run SPF? Do the same in the BGP twin: which routers' RIBs change, and did anyone recompute?
Answer
OSPF: all 5. leaf3's Router LSA changes and floods area-wide; spf_runs goes +1 everywhere.
BGP: 172.16.3.0/24 is withdrawn from every router's RIB (they all needed that prefix), but nobody ran SPF. 12 UPDATEs in total, a withdraw per session. Same reach, far less work per router. On camera:
router lsa_updates lsdb_lsas rib_routes spf_runs
spine1 98 (+1) 5 (+0) 7 (-1) 42 (+1)
spine2 98 (+1) 5 (+0) 7 (-1) 47 (+1)
leaf1 98 (+1) 5 (+0) 8 (-1) 49 (+1)
leaf2 98 (+1) 5 (+0) 8 (-1) 49 (+1)
leaf3 98 (+1) 5 (+0) 7 (-2) 43 (+1)
router bgp_prefixes bgp_updates_rcvd bgp_updates_sent rib_routes
spine1 6 (-1) 57 (+3) 36 (+3) 6 (-1)
spine2 6 (-1) 88 (+3) 81 (+3) 6 (-1)
leaf1 7 (-1) 52 (+2) 86 (+2) 8 (-1)
leaf2 7 (-1) 34 (+2) 34 (+2) 8 (-1)
leaf3 8 (+0) 34 (+2) 34 (+2) 7 (-2)leaf3 keeps its own 172.16.3.0/24 in its BGP table, marked invalid ("inaccessible, import-check enabled"), and stops advertising it. The kit gave the same deltas.
2. Scale-out cost
Add a new loopback prefix 10.99.0.3/32 on leaf3 (OSPF: add it on lo, already in area 0; BGP: also network 10.99.0.3/32). Compare the counter diffs (sudo python3 scripts/count.py snap both, change, diff both). What would adding a whole leaf4 cost each protocol?
Answer
OSPF: leaf3 re-originates its Router LSA, every copy of the map gets updated (lsa_updates +1 on all 5), and every router runs SPF (+1 on all 5) for one new /32. BGP: one UPDATE per session (12 in total), +1 prefix everywhere, no recompute. Measured on camera, and the same deltas from the kit. Both twins end up with 2 ECMP next hops to 10.99.0.3/32.
A real leaf4 in OSPF adds a new LSA and changes both spines' LSAs, on every router. In BGP it's one neighbor eth3 interface peer-group FABRIC line per spine (episode 1, exercise 1).
3. Policy is the app feature
In the BGP twin, make spine2 stop carrying traffic for leaf1 only, without touching any link. Can you do the same in OSPF without making other routers re-run SPF?
Answer
BGP: a route-map on spine2 that prepends (or denies) 172.16.1.0/24 outbound, scoped by a prefix-list (episode 1's DRAIN pattern), plus a prepend-everything route-map towards leaf1 so leaf1 stops sending via spine2 too. No SPF anywhere, and spine1 sends nothing. spine2 re-sends its table to the three leaves when the outbound policy changes (21 UPDATEs in our kit run), but that's per prefix, per neighbor, and nobody else recomputes anything.
OSPF: the only knob is link cost (ip ospf cost 1000 on spine2 eth0). That changes a Router LSA, floods everywhere, and every router re-runs SPF: with the cost raised on both ends of link 2, the kit counted +2 SPF runs on all 5 routers (10 total). There's no per-prefix, per-neighbor policy in the protocol; a cost moves everything that crosses the link. Full config in solutions/exercise3/.
11Recap & the kit
- BGP is an application: a TCP session on port 179, neighbor to neighbor, carrying best paths per prefix.
- OSPF / IS-IS flood the whole map to every router and every router re-runs SPF on every change.
- We flapped one link in identical twins: OSPF re-ran SPF on all 5 routers (19 runs, +4 LSA updates on every router); BGP ran SPF nowhere (17 small UPDATEs on camera), and spine1 sent nothing at all.
- Packet loss was the same in both (same 9 s timers). The win is blast radius and CPU, not speed.
- That blast-radius difference is why RFC 7938 fabrics, and today's AI fabrics, run eBGP inside, not just at the border.
- Link-state still wins in small networks and where the full map is the point (fast-reroute, traffic engineering, ISP cores under BGP).
Try it yourself
The free kit builds both twins with one command and reproduces the flap table and all three exercises. Needs Linux with Docker, containerlab and python3; about 350 MB of RAM for the 16 containers (measured) and about 850 MB of disk for the two images.
$ cd bgp-vs-ospf-lab # the folder this page lives in, one level up
$ sudo ./lab.sh up # build BOTH twins, wait until healthy (~1 min)
$ sudo ./lab.sh flap # break link 2 in both for 20 s, print who did work
$ sudo ./lab.sh down # destroy both twinsTested end to end on Ubuntu 24.04 with containerlab 0.73, from the shipped zip: 12/12 OSPF Full, 12/12 BGP Established, 2 ECMP next hops on every leaf, full host ping mesh in both twins, and the flap reproduced the on-camera numbers (OSPF 19–20 SPF runs on all 5 routers, BGP zero, same UPDATE counts). The README has every detail.
More: episode 1's lab (the eBGP fabric this builds on) · AI Agents Build the Lab on YouTube · how Graham runs the agent team that built this