Build an AI-cluster fabric: BGP unnumbered, ECMP, and the 9-second failure BFD fixes

1. Why this matters
2. The topology
3. IP & ASN plan
4. Config walkthrough
5. Verification
6. The demo: gray failure vs BFD
7. Hands-on exercises
8. Recap & try it yourself

1Why this matters

2The topology

Two spines, three leaves, one host per leaf. The rule to remember: leaf eth0 → spine1, leaf eth1 → spine2, leaf eth2 → host. On a spine, ethN goes to leaf N+1.

3IP & ASN plan

Fabric links (6) — no IPv4 at all

Lab images: FRRouting 10.5.5 containers for the five routers, a small Linux host image (ping, traceroute, iperf3, tcpdump) for h1–h3. The router console opens to a bash shell — type vtysh to get the FRR CLI.

4Config walkthrough

Step 0: kernel prep (every router)

Unnumbered BGP rides on IPv6 link-local addresses, so IPv6 must be on. We also tell Linux to hash ECMP on L4 ports, so parallel flows spread across spines.

sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
for i in eth0 eth1 eth2 eth3; do [ -e /proc/sys/net/ipv6/conf/$i ] && sysctl -w net.ipv6.conf.$i.disable_ipv6=0; done
sysctl -w net.ipv4.fib_multipath_hash_policy=1   # hash on src/dst IP + ports

And bgpd=yes and bfdd=yes in /etc/frr/daemons (zebra always runs). In the kit the script lives in /etc/frr/sysctl.sh on every router and runs automatically at boot (the kit's FRR image calls it in GNS3; containerlab runs it as an exec step). If you ever change it, re-run sh /etc/frr/sysctl.sh by hand.

leaf1 — /etc/frr/frr.conf

leaf2 / leaf3: same file, change hostname, ASN (65102 / 65103), loopback (10.0.1.2 / .3), eth2 (172.16.2.1 / 172.16.3.1) and the two network lines. That's the whole per-leaf diff: three variables.

spine1 — /etc/frr/frr.conf

spine2: identical, with hostname spine2 and 10.0.0.2.

What the router actually saved

After write memory, the saved file looks shorter than what we typed. FRR only writes lines that differ from the active defaults, and with frr defaults datacenter, no bgp ebgp-requires-policy, maximum-paths 64 and detect-multiplier 3 are all defaults. Same behavior, fewer lines. This is the real leaf1 file from the lab:

frr version 10.5.5_git
frr defaults datacenter
hostname leaf1
log stdout
service integrated-vtysh-config
!
interface eth2
 description to-h1
 ip address 172.16.1.1/24
exit
!
interface lo
 ip address 10.0.1.1/32
exit
!
router bgp 65101
 bgp router-id 10.0.1.1
 bgp bestpath as-path multipath-relax
 neighbor FABRIC peer-group
 neighbor FABRIC remote-as external
 neighbor eth0 interface peer-group FABRIC
 neighbor eth1 interface peer-group FABRIC
 !
 address-family ipv4 unicast
  network 10.0.1.1/32
  network 172.16.1.0/24
 exit-address-family
exit
!
bfd
 profile FAST
  transmit-interval 100
  receive-interval 100
 exit
 !
exit

That's the entire leaf: about two dozen lines, and not one of them is a fabric IP address.

5Verification

The fabric has no IP addresses

leaf1# show interface brief
Interface       Status  VRF             Addresses
---------       ------  ---             ---------
eth0            up      default         fe80::42:51ff:fef9:900/64
eth1            up      default         fe80::42:51ff:fef9:901/64
eth2            up      default         172.16.1.1/24
                                        fe80::42:51ff:fef9:902/64
lo              up      default         10.0.1.1/32
pimreg          up      default

eth0 and eth1 carry only an automatic IPv6 link-local address. That's all BGP needs. (pimreg is a PIM placeholder interface in this image — ignore it.)

BGP sessions, by interface name

leaf1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 10.0.1.1, local AS number 65101 VRF default vrf-id 0
BGP table version 12
RIB entries 15, using 1920 bytes of memory
Peers 2, using 33 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V    AS  MsgRcvd  MsgSent  TblVer  InQ OutQ  Up/Down State/PfxRcd  PfxSnt
spine1(eth0)    4 65000       34       35      12    0    0 00:01:21            5       8
spine2(eth1)    4 65000       35       37      12    0    0 00:01:24            5       8

Total number of neighbors 2

(The Desc column, which shows FRRouting/10.5.5_git, is trimmed here to fit the screen.) Each spine is a 3-neighbor version of the same thing: leaf1(eth0), leaf2(eth1), leaf3(eth2), 2 prefixes each.

leaf1# show bgp neighbors eth0
  BGP state = Established, up for 00:01:22
  Hold time is 9 seconds, keepalive interval is 3 seconds
Local host: fe80::42:51ff:fef9:900, Local port: 42968
Foreign host: fe80::42:e8ff:fee3:cb00, Foreign port: 179     # filtered to the key lines

The session runs between two fe80:: addresses, and the hold timer is 9 seconds from frr defaults datacenter. Remember that number.

Spine hostnames, interface names, a prefix count — Established. Leaves show 2 neighbors, spines show 3. If you see Active or Connect, IPv6 is off on that link (check ip -6 addr show dev eth0 for an fe80:: address).

ECMP: two next hops for one prefix

leaf1# show bgp ipv4 unicast 172.16.3.0/24
BGP routing table entry for 172.16.3.0/24, version 10
Paths: (2 available, best #1, table default)
  Advertised to peers:
  spine1(eth0) spine2(eth1)
  65000 65103
    fe80::42:75ff:fe5f:c800(spine2) from spine2(eth1) (10.0.0.2)
    (fe80::42:75ff:fe5f:c800) (used)
      Origin IGP, valid, external, multipath, bestpath-from-AS 65000, best (Older Path)
  65000 65103
    fe80::42:e8ff:fee3:cb00(spine1) from spine1(eth0) (10.0.0.1)
    (fe80::42:e8ff:fee3:cb00) (used)
      Origin IGP, valid, external, multipath

Two paths, one per spine, same AS path 65000 65103 (up to a spine, down to leaf3), and both marked multipath.

leaf1# show ip route
...
B>* 10.0.0.1/32 [20/0] via fe80::42:e8ff:fee3:cb00, eth0, weight 1, 00:01:21
B>* 10.0.0.2/32 [20/0] via fe80::42:75ff:fe5f:c800, eth1, weight 1, 00:01:24
C>* 10.0.1.1/32 is directly connected, lo, weight 1, 00:01:26
B>* 10.0.1.2/32 [20/0] via fe80::42:75ff:fe5f:c800, eth1, weight 1, 00:01:17
  *                    via fe80::42:e8ff:fee3:cb00, eth0, weight 1, 00:01:17
B>* 10.0.1.3/32 [20/0] via fe80::42:75ff:fe5f:c800, eth1, weight 1, 00:01:21
  *                    via fe80::42:e8ff:fee3:cb00, eth0, weight 1, 00:01:21
C>* 172.16.1.0/24 is directly connected, eth2, weight 1, 00:01:26
B>* 172.16.2.0/24 [20/0] via fe80::42:75ff:fe5f:c800, eth1, weight 1, 00:01:17
  *                      via fe80::42:e8ff:fee3:cb00, eth0, weight 1, 00:01:17
B>* 172.16.3.0/24 [20/0] via fe80::42:75ff:fe5f:c800, eth1, weight 1, 00:01:21
  *                      via fe80::42:e8ff:fee3:cb00, eth0, weight 1, 00:01:21

IPv4 routes with IPv6 next hops. That's RFC 8950 on screen. Every remote leaf prefix has two lines: two stars = both next hops are in the forwarding table. The spine loopbacks have one path each, because each spine is only one hop away on one link.

leaf1:~# ip route show 172.16.3.0/24
172.16.3.0/24 nhid 17 proto bgp metric 20
        nexthop via inet6 fe80::42:75ff:fe5f:c800 dev eth1 weight 1
        nexthop via inet6 fe80::42:e8ff:fee3:cb00 dev eth0 weight 1

The Linux kernel agrees: nexthop via inet6, twice.

Valley-free, proven

spine2# show bgp ipv4 unicast 172.16.3.0/24
BGP routing table entry for 172.16.3.0/24, version 3
Paths: (1 available, best #1, table default)
  Advertised to peers:
  leaf1(eth0) leaf2(eth1) leaf3(eth2)
  65103
    fe80::42:9fff:fed6:a601(leaf3) from leaf3(eth2) (10.0.1.3)
    (fe80::42:9fff:fed6:a601) (used)
      Origin IGP, metric 0, valid, external, bestpath-from-AS 65103, best (First path received)

Exactly one path. Leaf1 and leaf2 do offer spine2 a copy they learned from spine1, but its AS path contains 65000, so spine2 drops it as a loop.

End to end

h1:~# traceroute -n -q1 -w1 -U --sport=40001 172.16.3.10
 1  172.16.1.1  0.213 ms     leaf1
 2  10.0.0.2  0.238 ms       spine2 (answers from its loopback)
 3  10.0.1.3  0.305 ms       leaf3 (loopback, too)
 4  172.16.3.10  0.382 ms    h3

h1:~# traceroute -n -q1 -w1 -U --sport=40003 172.16.3.10
 1  172.16.1.1  0.215 ms
 2  10.0.0.1  0.229 ms       spine1 this time
 3  10.0.1.3  0.311 ms
 4  172.16.3.10  0.381 ms

Leaf, spine, leaf, host: the Clos in four hops. The routers answer from their loopbacks, because the fabric links have no IPv4 address to answer from. And the only thing we changed between the two runs is the UDP source port: hop 2 flips between spine2 and spine1. In the lab, source ports 40001–40006 went 2, 2, 1, 1, 1, 2. That's ECMP hashing on L4 ports, live.

Prove the traffic really spreads

h3:~# iperf3 -s
spine1/2:~# watch -n1 "ip -s link show eth2 | sed -n '4p;6p'"   # eth2 = link to leaf3
h1:~# iperf3 -c 172.16.3.10 -P 8 -t 30

Byte counters climb on both spines. Eight flows, hashed by port number, shared across two spines. That's the point for AI training: every GPU flow gets every spine's bandwidth.

6The demo: a silent failure, without and with BFD

GNS3's Suspend link simulates exactly that: carrier stays up, every packet is dropped. (Don't use Delete link or ip link set down — those are instant failures and spoil the contrast.)

Round 1 — without BFD

1. Start a fast ping on h1 (10 per second) and leave it running:

   h1:~# ping -i 0.1 172.16.3.10

2. Find which uplink carries it. ICMP has no ports, so it sticks to one path. In GNS3, right-click link 1 (leaf1 eth0 – spine1 eth0) and link 2 (leaf1 eth1 – spine2 eth0) → Start capture. Whichever shows ICMP is our target. Stop both captures.
   containerlab: sudo scripts/which-uplink.sh compares the packet counters on leaf1 eth0 and eth1 and tells you.
3. Right-click that link → Suspend. Both interfaces stay up. The ping stops getting replies.
   containerlab: sudo scripts/link.sh suspend eth1 (or eth0) puts 100% packet loss on both ends of the link with containerlab tools netem. Same effect: interfaces up, packets gone.
4. Watch BGP not notice:

   leaf1# show bgp summary

   Run it a few times. The session to that spine stays Established… for about 9 seconds. Then the hold timer expires, the session drops, and the ping recovers over the other spine.
5. Ctrl-C the ping and read the loss line.
6. Resume the link (containerlab: sudo scripts/link.sh resume eth1) and wait for both sessions to come back in show bgp summary.

--- 172.16.3.10 ping statistics ---
300 packets transmitted, 225 received, 25% packet loss, time 31094ms
# 75 lost = 7.5 seconds of black hole (GNS3 runs: 79 and 71)

BGP held the dead session as Established for 7.6 seconds before giving up. The hold timer expires 9 s after the last keepalive that got through, so where you cut inside the 3 s keepalive cycle sets the exact number: expect 60 to 90.

Round 2 — turn on BFD, live

1. On leaf1, attach the profile to the whole peer group:

   leaf1:~# vtysh
   leaf1# configure terminal
   leaf1(config)# router bgp 65101
   leaf1(config-router)# neighbor FABRIC bfd profile FAST
   leaf1(config-router)# end

2. On spine1 and spine2 (eth0 is the link to leaf1 on both):

   spine1# configure terminal
   spine1(config)# router bgp 65000
   spine1(config-router)# neighbor eth0 bfd profile FAST
   spine1(config-router)# end

   BFD needs both ends. We only turn it on for leaf1's two links so the change stays small on camera; in production you'd set neighbor FABRIC bfd profile FAST everywhere.
3. Shortcut: the same change as one line each, pasted into the router's bash console (tested on both GNS3 and containerlab):

   # leaf1
   vtysh -c 'conf t' -c 'router bgp 65101' -c 'neighbor FABRIC bfd profile FAST' -c 'end'
   # spine1 and spine2
   vtysh -c 'conf t' -c 'router bgp 65000' -c 'neighbor eth0 bfd profile FAST' -c 'end'

4. Check the BFD sessions (before this step the same command says Session count: 0: BFD really was off):
   live lab output

   leaf1# show bfd peers brief
   Session count: 2
   SessionId  LocalAddress                 PeerAddress                 Status
   =========  ============                 ===========                 ======
   1882543610 fe80::42:51ff:fef9:900       fe80::42:e8ff:fee3:cb00     up
   3591402277 fe80::42:51ff:fef9:901       fe80::42:75ff:fe5f:c800     up

5. Repeat round 1 exactly: ping -i 0.1, suspend the same link, wait ~10 s, Ctrl-C.
6. Look at the damage. show bfd peers brief shows one peer down; show bgp summary shows that session already gone; show ip route 172.16.3.0/24 has one next hop.
7. Resume the link: BFD up, BGP back, two next hops again.

--- 172.16.3.10 ping statistics ---
200 packets transmitted, 197 received, 1.5% packet loss, time 20696ms
# 3 lost = about 300 ms (GNS3 run: 4)

7Hands-on exercises

Exercise 1 — Scale out with zero IP planning

Add leaf4 (AS 65104, lo 10.0.1.4/32) with h4 on 172.16.4.0/24. Cable leaf4 eth0 → spine1 eth3 and leaf4 eth1 → spine2 eth3. (In the GNS3 project the spines already have a spare eth3. In containerlab, add leaf4, h4 and the three links to the topology file and redeploy.) Goal: h1 pings h4, and leaf1 shows two ECMP next hops for 172.16.4.0/24. How many lines did you change on each spine?

router bgp 65000
 neighbor eth3 interface peer-group FABRIC

Exercise 2 — Drain a spine with zero packet loss

Before maintenance on spine2, make the fabric stop using it gracefully. On spine2, build a route-map that prepends the AS path and apply it outbound. Keep the 10 pps ping and the iperf from section 5 running while you do it. Goal: leaf routes collapse to spine1, spine2's counters go flat, and the ping shows 0 loss. Then undo it and watch ECMP return. Why is this lossless when the suspended link wasn't?

spine2# configure terminal
spine2(config)# route-map DRAIN permit 10
spine2(config-route-map)#  set as-path prepend 65000 65000
spine2(config-route-map)# exit
spine2(config)# router bgp 65000
spine2(config-router)#  address-family ipv4 unicast
spine2(config-router-af)#   neighbor FABRIC route-map DRAIN out
spine2(config-router-af)# end
spine2# clear bgp ipv4 unicast * soft out

Exercise 3 — Break the ASN design on purpose

Change spine2 to AS 65001. (a) On spine2, run show bgp ipv4 unicast 172.16.3.0/24: how many paths now, and what are the extra ones? (b) Why didn't the leaves need any config change? (c) Remove bgp bestpath as-path multipath-relax on leaf1, then clear bgp * soft. What happens to ECMP, and why?

8Recap

• Clos + eBGP everywhere is how AI and hyperscale fabrics are built: simple, scalable, ECMP across every spine.
• BGP unnumbered (neighbor eth0 interface) peers over IPv6 link-local and carries IPv4 with IPv6 next hops (RFC 8950). No per-link IPs, ever.
• Same-ASN spines make the fabric valley-free (RFC 7938).
• ECMP shows up as multiple next hops in FRR and the Linux kernel; L4 hashing spreads flows across spines.
• Down is easy, silent is deadly. BGP alone waits out its 9 s hold timer. BFD at 100 ms × 3 detects it in ~300 ms: ~75 lost packets became ~4 (tested: 79 and 71 without BFD, 4 with).

Try it yourself

Try it yourself

Everything in this lesson is in this kit. Pick one:

Option A — containerlab (one command, no GUI)

Any Linux box (or WSL2) with Docker and containerlab:

$ cd ai-fabric-bgp-unnumbered-lab
$ sudo containerlab deploy -t ai-fabric.clab.yml
$ docker exec -it clab-ai-fabric-leaf1 vtysh        # router CLI
$ docker exec -it clab-ai-fabric-h1 bash             # host shell
$ sudo containerlab destroy -t ai-fabric.clab.yml --cleanup   # when you're done

It uses the stock quay.io/frrouting/frr:10.5.5 and nicolaka/netshoot images, and the interface names match the GNS3 version exactly. GNS3's Suspend link becomes sudo scripts/link.sh suspend eth1.

Option B — GNS3 (the version in the video)

$ docker build -t ai-fabric-lab/frr:10.5.5 docker/frr     # on your GNS3 server / VM

Then in GNS3: File → Import portable project → gns3/ai-fabric-baseline.gns3project, and start all nodes. The routers boot with their configs and kernel settings already in place; BFD is off, ready for the demo. Hosts use gns3/ipterm, which GNS3 pulls by itself.

Needs: about 2 GB of free RAM and 2 CPU cores for the 8 containers (they idle near zero), plus about 1 GB of disk for the images (containerlab) or 300 MB (GNS3). Tested on Ubuntu 24.04 with containerlab 0.73 and GNS3 2.2.56. The README in the kit has the full walkthrough, the demo script, and all exercise answers.