AI Agents Build the Lab · Episode 3 · Free lab kit · containerlab · SRv6 uSID

OpenAI turned BGP off

The switches don't route.
The host does.

In Episode 2, eBGP beat OSPF inside an AI fabric because each flap touched less of the network. OpenAI's MRC goes all the way: no routing protocol in the switches at all. The host writes the whole path into one IPv6 address. When a path dies, the host stops using it and no switch even notices.

Today we build that with the same 5 switches and the same 9 cables as Episodes 1 and 2, and then we break it.

Download the free lab kit (.zip)

The whole fabric and the host path agent in one containerlab command, plus the failure test, the packet x-ray and exercise answers. MIT licensed. How to run it ↓ · Episode 2's lab · The series on YouTube

On this page

  1. The idea in plain English
  2. The ticket punch: one address, three stops
  3. The topology and SID plan
  4. The entire switch config
  5. What the host does
  6. Run it yourself
  7. The numbers
  8. Exercises (with answers)
  9. What we can't reproduce
  10. Sources

1The idea in plain English

MRC (Multipath Reliable Connection)

A transport from OpenAI with AMD, Broadcom, Intel, Microsoft and NVIDIA, published through OCP in May 2026. The GPU's network card sprays one transfer across hundreds of paths, watches each path, and stops using a bad one within tens of microseconds. It's used in production on 50K to 75K GPU clusters.

SRv6 uSID

Segment Routing over IPv6, in its compressed "micro-SID" form. Every switch owns a short 16-bit ID. A path is just those IDs written in order inside the destination address. Each switch looks at the front, removes its own ID, and forwards on the next one. All it needs is a static table.

Put together: the switches hold a fixed table that never changes. The host chooses the path, so it's the host that has to notice failures. With no protocol there are no neighbors, no hellos, no hold timers, and nothing to converge.

Episode 2's argument was "touch less on every failure." This is the end of that road: a failure touches nothing in the fabric. Only the hosts that were using the dead path change anything.

2The ticket punch: one address, three stops

Think of the address as a train ticket with a row of stops printed on it. At each stop the conductor punches off their own stop and sends you on to the next one printed. Nobody on the train has to know the whole route. They only read the front of the ticket.

Our addressing: the block fcbb:bb00 (32 bits) followed by 16-bit IDs. Spine1 = 100, spine2 = 200, leaf3 = 3, and fe00 = "unwrap and hand it to the host." This is the real address h1 uses to reach h3 through spine1:

h1 writes
fcbb:bb00:100:3:fe00:: "Go to spine1, then leaf3, then unwrap." h1 wraps its IPv4 packet (172.16.1.10 → 172.16.3.10) in IPv6 with this destination.
leaf1
fcbb:bb00:100:3:fe00:: Not leaf1's ticket. It's a plain lookup: static route fcbb:bb00:100::/48 → the spine1 link. No change.
spine1 punches
fcbb:bb00:100:3:fe00:: Matches its own uN SID fcbb:bb00:100::/48. It shifts its 16 bits out and the rest slides left.
on the wire
fcbb:bb00:3:fe00:: The new destination. Static route fcbb:bb00:3::/48 → the leaf3 link.
leaf3 punches
fcbb:bb00:3:fe00:: Matches fcbb:bb00:3:fe00::/64 = End.DX4. It strips the IPv6 wrapper and hands the original IPv4 packet straight out eth2 to h3.

Change one number, change the path. fcbb:bb00:200:3:fe00:: is the same trip through spine2. The host picks which one, packet by packet if it likes. The switches never have to agree on anything.

One address holds up to 6 of these 16-bit stops (F3216 format), so a two-tier fabric path fits easily with no extra header bytes.

Animation: "The Ticket Punch." Made by the agent team during the episode: one address, each switch punching off its own stop. Open it on its own page

3The topology and SID plan

It's the same Clos as Episodes 1 and 2: 2 spines, 3 leaves, 3 hosts, 9 cables. The difference is what's running: zebra + staticd only. bgpd, ospfd, isisd and bfdd are all switched off.

link 2 (we fail this) spine1uN fcbb:bb00:100::/48 spine2uN fcbb:bb00:200::/48 leaf1fcbb:bb00:1::/48 leaf2fcbb:bb00:2::/48 leaf3fcbb:bb00:3::/48 h1172.16.1.10 h2172.16.2.10 h3172.16.3.10 leaf eth0 → spine1 · leaf eth1 → spine2 · leaf eth2 → host (decap fe00 = End.DX4)
WhatValue
Spine uN SIDsfcbb:bb00:100::/48 (spine1), fcbb:bb00:200::/48 (spine2)
Leaf uN SIDsfcbb:bb00:N::/48 for leafN
Decap (uDX4)fcbb:bb00:N:fe00::/64 → End.DX4 to 172.16.N.10 on eth2
Fabric linksfd00:N:M00::/64, leaf ::1, spine ::2 (leaf1–spine2 = fd00:1:200::/64)
HostsIPv4 172.16.N.10/24 (gw .1) as in ep1/ep2, plus IPv6 fd00:a:N::10
A pathfcbb:bb00:<spine>:<leaf>:fe00::

4The entire switch config

This is all of leaf1 (configs/leaf1.conf plus configs/srv6.sh in the kit). Every switch has the same shape. The spines just carry three ipv6 route lines, one to each leaf /48.

interface eth0 ipv6 address fd00:1:100::1/64
Uplink to spine1. Just an address.
interface eth1 ipv6 address fd00:1:200::1/64
Uplink to spine2, which is link 2.
interface eth2 ip address 172.16.1.1/24 ipv6 address fd00:a:1::1/64
Host port. h1 keeps its old IPv4 address.
ipv6 route fcbb:bb00:100::/48 fd00:1:100::2 ipv6 route fcbb:bb00:200::/48 fd00:1:200::2
Two static routes. "Anything for spine1's ID goes up eth0, spine2's up eth1." These never change.
# kernel, via srv6.sh ip -6 route replace fcbb:bb00:1::/48 \ encap seg6local action End \ flavors next-csid lblen 32 nflen 16 dev sr0
My own ticket punch (uN). Block length 32, node ID 16: shift my 16 bits out, then forward on what's left.
ip -6 route replace fcbb:bb00:1:fe00::/64 \ encap seg6local action End.DX4 \ nh4 172.16.1.10 dev eth2
Last stop. Strip the IPv6 wrapper and hand the IPv4 packet to h1.

That's the whole control plane. No router bgp, no router ospf, no neighbors, no hellos, nothing listening on TCP 179.

5What the host does

Each host's main route to a remote leaf lists both paths, and Linux sprays new flows across them:

h1, set up by configs/path-agent.shh1# ip route replace 172.16.3.0/24 \
  nexthop encap seg6 mode encap segs fcbb:bb00:100:3:fe00:: dev eth0 \
  nexthop encap seg6 mode encap segs fcbb:bb00:200:3:fe00:: dev eth0

Then a small bash path agent (configs/path-agent.sh in the kit, about 45 lines) does MRC's job, but slowly:

6Run it yourself: the episode in five commands

Download the lab kit (.zip)

Needs Linux with Docker, containerlab and python3, and a kernel with SRv6 uSID support (6.3 or newer; stock Ubuntu 24.04 works, WSL2 may not). All 8 containers use under 300 MB of RAM. Images: quay.io/frrouting/frr:10.5.5 and nicolaka/netshoot.

$ unzip srv6-no-routing-lab.zip && cd srv6-no-routing-lab
$ sudo ./lab.sh up      # 5 switches + 3 hosts, wait until they forward, run every check (~15 s)
$ sudo ./lab.sh check   # no routing daemons / nothing on TCP 179, SID tables, both planes, spray
$ sudo ./lab.sh fail    # break link 2 for 10 s: host agent OFF, then ON; prints lost pings + ms
$ sudo ./lab.sh xray    # capture h1 -> h3 on both planes: the path is written in the packet
$ sudo ./lab.sh down    # destroy the lab

Tested end to end on Ubuntu 24.04 with containerlab 0.73, from the shipped zip: up in 10 s with every check passing, agent ON 5 lost / 69 ms, agent OFF 997 lost (the whole 10 s break), x-ray showing spine1 punch its 100. The README has every detail.

  1. No routing protocol, and traffic still flows. sudo ./lab.sh up builds the lab and runs every check. From the final test of the shipped kit: 
    [1] No routing protocol in the fabric
  PASS spine1  running: mgmtd staticd watchfrr zebra | routing daemons: none | TCP 179 listeners: 0 | router/neighbor config lines: 0
  ... (same for spine2, leaf1, leaf2, leaf3)
[3] h1 -> h3 on each plane (pinned with ping -m 1 / -m 2), counted on the spines' links to leaf3
  PASS plane 1: 10/10 replies (avg 0.088 ms), path fcbb:bb00:100:3:fe00::  spine1 eth2 +10, spine2 eth2 +0
  PASS plane 2: 10/10 replies (avg 0.086 ms), path fcbb:bb00:200:3:fe00::  spine1 eth2 +0, spine2 eth2 +10
  PASS full mesh: 12/12 (6 host pairs x 2 planes)
    Look for yourself: docker exec clab-srv6-spine1 ss -tln (nothing on 179), docker exec clab-srv6-spine1 vtysh -c 'show running-config' (interfaces and ipv6 route, that's all), docker exec clab-srv6-h1 ip route show 172.16.3.0/24 (two nexthops; the host wrote both).
  2. Spray with zero protocol. Check [4] sends 1000 new UDP flows from h1 to h3 and reads leaf1's two uplink counters: 
    [4] Spray: 1000 new UDP flows h1 -> h3, unpinned (main table = ECMP over both planes, per-flow hash)
  PASS leaf1 sent 502 via spine1 (eth0) / 498 via spine2 (eth1); h3 received 1000 of 1000
    (510 / 490 in QUINN's live test on camera night.)
  3. Break link 2, agent OFF, then agent ON. sudo ./lab.sh fail runs a 100 pps ping pinned to spine2, breaks leaf1 eth1 ↔ spine2 eth0 for 10 s (100% loss, carrier stays up, so no switch notices), and repeats with the host agents running: 
    === link 2 failure, host agent OFF ===
  RESULT agent off: 1600 sent, 603 replies, 997 lost; biggest reply gap 11001 ms
=== link 2 failure, host agent ON ===
  agent logs:
    h1 13:50:36.789 leaf3 plane2 DOWN after 3 misses -> moved to other plane
    h2 13:50:36.792 leaf1 plane2 DOWN after 3 misses -> moved to other plane
    h3 13:50:36.795 leaf1 plane2 DOWN after 3 misses -> moved to other plane
    h1 13:50:48.207 leaf3 plane2 UP -> spraying again
  RESULT agent on: 1600 sent, 1595 replies, 5 lost; biggest reply gap 69 ms (icmp_seq 271 -> 277)
Compare: episode 2's eBGP twin lost 6-9 s on this same link (hold timer 9 s).
    Agent OFF, the pinned flow is lost for the whole break: nothing in a static fabric heals. Agent ON, every host whose path crossed the dead link moves within a few milliseconds of the others, and no switch changes anything.
  4. X-ray: the path is in the packet. sudo ./lab.sh xray captures the same ping leaving leaf1 and leaving the spine: 
    --- h1 -> spine1 (leaving leaf1)
    dst fcbb:bb00:100:3:fe00::   | SRH: segleft=0 segment[0]=fcbb:bb00:100:3:fe00:: | inside: 172.16.1.10 > 172.16.3.10: ICMP echo request
--- spine1 -> leaf3 (after spine1 punched 100)
    dst fcbb:bb00:3:fe00::       | SRH: segleft=0 segment[0]=fcbb:bb00:100:3:fe00:: | inside: 172.16.1.10 > 172.16.3.10: ICMP echo request
    Segment[0] still shows what h1 wrote; the destination shows what's left after spine1 shifted its own 16 bits off. Want Wireshark? docker exec clab-srv6-leaf1 tcpdump -i eth0 -w /tmp/x.pcap, docker cp it out, and filter ipv6.routing.type == 4.
    Animation: "The Path Is in the Packet." Built from the real packets captured on camera. Those pings hashed via spine2, so you see 200 punched off. Open it on its own page
  5. Clean up. sudo ./lab.sh down destroys the lab. Also: sudo ./lab.sh agent start|stop|status and sudo ./lab.sh link suspend|resume to drive it by hand.

7The numbers: same link, same failure

Link 2 (leaf1 eth1 ↔ spine2 eth0) fails the same way in every episode: on camera with a GNS3 link Suspend, in the kit with 100% netem loss. Either way the carrier stays up, so no switch notices. Ping runs at 100 pps, so each lost packet is about 10 ms.

until fixed
Ep3, host agent OFF
the flow pinned to the dead plane stays lost until the link comes back
6–9 s
Ep2 eBGP, same link
(9 s hold timer)
~50 ms
Ep3, host agent ON, on camera
5 lost at 100 pps (icmp_seq 4437 → 4443)
RunLostOutage
On camera, agent ON5icmp_seq 4437 → 4443 at 100 pps = ~50 ms. Log: "plane 2 down after 3 misses."
QUINN's measured runs, agent ON (GNS3, same night)5–666–77 ms biggest reply gap; h1 and h3 logged DOWN within ~4 ms of each other
This kit, final test from the zip (containerlab)569 ms biggest reply gap (icmp_seq 271 → 277)
On camera, agent OFF4,722 of 14,6919,969 received, 32% loss over 157 s: the pinned flow was lost until the link was unsuspended
QUINN, agent OFF, 10 s suspend918 in a row10.11 s = the whole outage; back only on unsuspend
This kit, agent OFF, 10 s break99711.0 s reply gap = the whole break
Ep2, eBGP, same link60–86 at 10 pps6–9 s; every router got the withdraw
eBGP + BFD (on paper)—~0.9 s (300 ms × 3)
Real MRC (in the NIC)—microseconds

Why two ways to count: "lost × 10 ms" gives ~50 ms; the biggest gap between reply timestamps also includes the normal 10 ms spacing on each side, so it reads 66–77 ms. Both say the same thing: about 100× faster than episode 2, and no switch did anything.

"9 seconds in episode 2. About 50 milliseconds now, and no switch did anything."

The tuning knob for the curious: INT=0.005 MISS=3 in the agent halves detection time at the cost of twice the probe traffic.

8Exercises (with answers)

1. Add a third stop

Make h1 → h3 traffic go through leaf1's own uN on purpose, then spine1, then leaf3. What's the address, and what does leaf1 do with it?

Answer

fcbb:bb00:1:100:3:fe00::. leaf1 matches its own uN fcbb:bb00:1::/48 and punches 1 off, leaving fcbb:bb00:100:3:fe00::. It then forwards that to spine1 on the static route. That's three stops in one address and still one segment. F3216 fits 6 stops per address. (Tested: 0% loss.)

2. Who notices a dead spine1–leaf3 link?

With all agents running, suspend spine1 eth2. Which hosts move traffic, and which do nothing?

Answer

h1 and h2 move, because their leaf3-via-plane-1 probes fail. h3 moves too, because its replies to h1 and h2 on plane 1 go back through spine1 → leaf3, so its plane-1 probes fail. h1 ↔ h2 traffic via spine1 is untouched, and nothing in the fabric changes. In ep2, a BGP withdraw would have reached every router.

3. Drop the header

Change one of h1's table-1 routes from mode encap to mode encap.red and capture again. What disappears, and why does it still work?

Answer

The Segment Routing Header disappears, leaving just IPv6 with IPv4 inside. It still works because with uSID the whole path already lives in the destination address. You only need the SRH when the path is too long for one 128-bit address. That's why MRC uses uSID: zero extra header bytes. (Tested.)

Each answer has a script in the kit that reproduces it and then undoes it: sudo ./solutions/exercise1/run.sh, exercise2/run.sh, exercise3/run.sh (each folder also has answer.txt).

9What we can't reproduce

Same idea, 1000× slower clock

Real MRCOur lab
RDMA NICs (RoCEv2 extended), path logic in the NIC hardwareNo RDMA NICs. Linux kernel seg6 encap plus a bash loop on each host.
Per-packet spray across hundreds of pathsPer-flow ECMP hashing across 2 paths. One flow stays on one spine.
Packet trimming on congestion (the switch cuts the payload and forwards the header) instead of PFCNo trimming, no congestion signal at all.
Per-path congestion controlNone. It only reacts to "path dead."
Hardware in the NIC reacts in microsecondsA bash agent detects in ~30 ms (100 Hz pings, 3 misses); the outage end to end is ~50–70 ms
50K–75K GPUs, multi-plane fabrics3 hosts, 2 planes, in containers (GNS3 on camera, containerlab in the kit)

What is real: the switches really run no routing protocol, the path really is written into the address by the host, the uSID shift really happens in the Linux kernel, and the outage numbers are measured, not simulated.

Static routes ARE routing. Graham's point on camera: "no routing protocol" doesn't mean no routing. Every switch still has a forwarding table (a few static /48s and its own SIDs); it just never changes. The decisions (which path, when to give up on one) moved out of the switches and into the host. The path agent is the routing now; it just runs at the edge.

The trade-off MRC accepts: a static fabric can't heal itself (agent OFF = loss until someone repairs the link). Every host has to carry that job.

10Sources

  1. OpenAI, "Supercomputer networking to accelerate large scale AI training" (MRC, May 2026): https://openai.com/index/mrc-supercomputer-networking/. Don't miss Mark Handley's animation on that page: packets sprayed across the planes and steered around a failure.
  2. "Resilient AI Supercomputer Networking using MRC and SRv6," arXiv 2605.04333: https://arxiv.org/html/2605.04333v1
  3. MRC transport paper, arXiv 2606.18170 (June 2026): https://arxiv.org/pdf/2606.18170
  4. Cisco, "MRC and SRv6: how foundational networking innovations are enabling the next generation of AI supercomputers" (May 14, 2026): blogs.cisco.com
  5. IETF draft-filsfils-srv6ops-srv6-ai-backend-04 (June 2026): https://datatracker.ietf.org/doc/html/draft-filsfils-srv6ops-srv6-ai-backend-04
  6. Converge Digest, "MRC redesigns Ethernet for 100,000-GPU AI clusters" (May 6, 2026): convergedigest.com
  7. Meta, OCP Summit 2026: Flattened Planar Fabric, hosts steering around dead planes (Oct 12, 2026): engineering.fb.com
  8. Linux seg6local (End, next-csid flavor, End.DX4): https://man7.org/linux/man-pages/man8/ip-route.8.html
  9. Episode 2 lab (eBGP vs OSPF, the 6–9 s baseline on the same link): https://www.grahampaasch.com/labs/bgp-vs-ospf
  10. Episode 1 lab (the eBGP AI-cluster fabric this all started from): https://www.grahampaasch.com/labs/ai-fabric

More: episode 2's lab · episode 1's lab · AI Agents Build the Lab on YouTube · how Graham runs the agent team that built this