AI Supply Chain Research · Sector 06

Networking, Interconnect & Optics

Bandwidth, not FLOPs, is the wall — copper reaches its 448G ceiling, and light takes over scale-up.  ·  ← back to the series  ·  Background: Primer §00, §03, §04, §07, §10

At a glance
Winners
TSMCBroadcom (AVGO)Nvidia (NVDA)Marvell (MRVL) + Celestial AIAmphenol / connector & copper vendorsCCL / high-speed PCB materials (Panasonic, Doosan)Transceiver assemblers (InnoLight, Eoptolink, Fabrinet)
Bottlenecks
External Laser Sources (ELS) — the top CPO failure point; production ramp of multi-wavelength lasers (Sivers, Lumentum, Lightmatter GUIDE) gates scale-up CPO.Fiber attach / FAU alignment — still largely manual, yield drops with each fiber; automation (Ficontec) is low-throughput. Nubis' 36-fiber 2D array is the densest shipping.MRM thermal control — a 2°C drift shifts resonance past collapse; managing it under a hot XPU (Lightmatter/Celestial placing modulators beneath the die) is unproven at scale.CPO supply-chain immaturity — only ~10-15k scale-out CPO switches ship in 2026 as a pipe-cleaner; Rubin Ultra CPO timeline (late 2027) is 'too ambitious', pushing real scale-up to Feynman.Copper's 448G wall — true 448G uni-directional SerDes needs PAM6/8 with uncertain timing; co-packaged copper (twinax on substrate) is the interim bridge.
Risks
CPO's scale-out TCO is underwhelming (2-7% cluster cost) — adoption could stall if reliability/serviceability fears outweigh a modest saving; Google refuses CPO on reliability grounds.Vendor lock-in loss of bargaining power — CPO collapses a multi-transceiver-vendor market into a few switch vendors, a key reason hyperscalers hesitate.Startup crowding vs incumbents — Ayar/Celestial/Lightmatter/Nubis must out-execute Nvidia/Broadcom/Marvell who already have proprietary solutions; consolidation is underway (Nubis→Ciena, Celestial→Marvell).Marvell DSP moat erosion — Broadcom, Nvidia and in-house hyperscaler efforts are designing Marvell out of DSPs; its thesis now rides on custom XPU + Celestial CPO execution.Ethernet share shift away from Nvidia InfiniBand — hyperscalers adopting Tomahawk 5 + UEC erodes Nvidia's networking attach, though Nvidia counters with Spectrum-X.
Catalysts
Nvidia scale-out CPO ramp (Quantum-X 2H25 IB, Spectrum-X 2H26 Ethernet) as the supply-chain pipe-cleaner; watch 10-15k unit 2026 volume.AWS Trainium 4 with Celestial/Marvell scale-up CPO ramping late 2027 — the first high-volume scale-up CPO deployment; Amazon warrants vest on Photonic Fabric purchases.Larger-scale Meta/Broadcom CPO field reliability data — the industry needs more than 15 lab switches before committing billions to scale-up CPO.UEC RC1 productization + interoperability plugfests, and UALink/SUE IP appearing inside hyperscaler host chips.Rubin Ultra 'Kyber' (144 GPU packages / 576 dies, 4x NVL72 density) and the eventual copper-to-optics scale-up transition at Feynman.

Overview

In the AI datacenter, networking is the second-largest capex and power bucket behind the accelerators themselves — roughly 15% of a GB300 NVL72 cluster's cost on a 3-layer InfiniBand network, rising to 18% on 4 layers, with the back-end (scale-out) fabric alone accounting for ~85% of networking cost and ~86% of networking power. SemiAnalysis frames the sector around two distinct fabrics with opposite economics. Scale-up (NVLink, UALink, Broadcom SUE) is the ultra-high-bandwidth, low-latency, all-to-all mesh that fuses dozens of GPUs into one 'world size'; NVLink 5 already moves 7.2 Tbit/s (900 GB/s) per GPU — 9x the 800 Gbit/s of scale-out — and doubles again in Rubin. Scale-out (InfiniBand vs. Ethernet) links thousands of racks. The central tension: copper is cheap, cool and reliable but capped at ~2 m reach and is running out of SerDes runway (Nvidia is delivering '448G' in Rubin only via a bi-directional trick, not true 448G), while pluggable optics carry ~60% of scale-out networking cost and burn a DSP that eats ~50% of an 800G module's power. Co-packaged optics (CPO) is SemiAnalysis's marquee thesis: it kills the DSP and the long-reach SerDes, cuts optical-link energy ~65-73%, and — critically — removes the reach limit so scale-up world sizes can span multiple racks. But CPO's TCO win in scale-out is modest (only ~3-7% cluster cost, 2-4% cluster power), so the real prize is scale-up, arriving with Rubin Ultra / Feynman and hyperscaler ASICs (AWS Trainium 4) around late 2027. Around this sit the merchant-silicon wars — Broadcom vs. Marvell vs. Nvidia in switch ASICs, DSPs and custom XPUs — the Ethernet-vs-InfiniBand standards fight (UEC, UALink, SUE), Google's optical circuit switching, and a China variant (Huawei CloudMatrix) that throws power and 100%-optics at the problem because it lacks silicon, not electricity.

Positioning: Who Wins and Why

TSMC — COUPE bumpless SoIC is winning next-gen CPO integration from GF/Tower — even Broadcom, Ayar and Lightmatter are pivoting to it, and using COUPE locks customers into TSMC-made PICs.

Broadcom (AVGO) — The silent #2 AI chip company: merchant switch leader (Tomahawk 5/6), first to ship production CPO (Bailly/Humboldt/Davisson), and custom-XPU designer for Google TPU, Meta MTIA and now OpenAI.

Nvidia (NVDA) — Owns the scale-up moat (NVLink), bundles Spectrum-X/Quantum with NCCL, and is pipe-cleaning the CPO supply chain first — Spectrum-X already out-ships Quantum in Blackwell.

Marvell (MRVL) + Celestial AI — Leads 112G/224G PAM4 DSPs and coherent optics; the multi-billion-dollar Celestial AI buy (EAM optical interposers, with a $2.25B revenue-contingent earn-out) targets scale-up CPO in AWS Trainium 4 with $1B run-rate guided by CY28-end.

Amphenol / connector & copper vendors — NVLink's 5,184-cable copper backplane (Paladin) plus co-packaged copper (Luxshare) keep growing as copper persists in-rack even into Rubin; connectors are the value, not wire.

CCL / high-speed PCB materials (Panasonic, Doosan) — Rubin's cableless design forces M8/M9 CCL, HVLP4 foil and (debated) quartz cloth, growing high-end PCB area ~2.3x vs GB300 — a quiet content-growth story.

Transceiver assemblers (InnoLight, Eoptolink, Fabrinet) — Optics scale with GPU count (the 'boogeyman' fear is wrong); CX-8 drives 800G DR4/1.6T DR8; supply broadens as Nvidia adds Eoptolink for LPO/1.6T DSP.

Key Data

MetricValueNote
Networking share of AI cluster cost15% (3-layer IB) → 18% (4-layer)Second-largest capex after the server itself, for a GB300 NVL72 cluster.
Back-end fabric share of networking cost/power~85% cost / ~86% powerScale-out back-end dominates; transceivers alone ~60% of 3-layer networking cost, ~45% of power.
CPO optical-link power saving vs DSP transceiver65-73% (up to 80% aspired)800G DSP transceiver ~16-17W vs CPO OE+ELS ~4-5.4W per 800G.
DSP share of transceiver power / BoM~50% power / 20-30% BoMFor an 800G SR8; ~480 kW of DSP power in an 18k-GPU GB300 back-end.
NVLink 5 per-GPU scale-up bandwidth7.2 Tbit/s (900 GB/s), 9x scale-outRises to 14.4 Tbit/s in Rubin; scale-out per GPU is 800 Gbit/s on CX-8.
Copper scale-up reach limit~2 m at 200G/laneCaps NVLink world size to 1-2 racks; optics removes this to scale across racks.
GB200 NVL72 copper NVLink cabling5,184 differential pairs / cablesOptics for the same spine would need 648 1.6T transceivers, ~20 kW, ~$550k/rack BoM.
NVL576 optical scale-up BoM penalty>$5.6M/rack (~$9.7k/GPU)~$38.8k/GPU to customer at 75% margin — why optical scale-up (and DGX H100 NVL256) never shipped.
CPO scale-out cluster power / cost saving2-4% power / 3-7% costNetworking is only ~9% of cluster power, so CPO's big networking savings dilute at cluster level.
Meta/Broadcom CPO reliability (Bailly 51.2T)2.6M-hr MTBF vs 0.5-1M for 2xFR4Zero UCWs to 4M port-device-hours, within a ~15M port-device-hour campaign across 15 switches over ~11 months.
TSMC COUPE bandwidth-density gain>23x (SoIC bumpless vs bumped, iso-power)PIC on N65, EIC on N6; interposer-integrated OEs could reach ~4 Tbit/s/mm.
Modulator sizes (MZM vs MRM vs EAM)~12,000 vs 25-225 vs ~250 µm2MRM ~50-500x smaller than MZM but 10-100x more temperature-sensitive.
Nvidia CPO MRM line rate milestone200G/lane PAM4 MRM in productionMatches fastest MZMs; disproves the notion MRMs are NRZ-limited.
Ethernet vs InfiniBand switch radix128x400G (TH5/SN5600) vs 64x400G (Q-2)Higher radix lets a 100k cluster run 3 tiers not 4 (1.33x fewer transceivers).
Google Apollo OCS power~108 W vs ~3,000 W (136-port EPS)Bandwidth/wavelength-agnostic; used in all TPU v4/v5 pods; saves >$3B.
Huawei CloudMatrix 384 optics6,912 400G LPO transceivers, ~18/chip100% optics / 0% copper; ~300 PFLOPs BF16 but 4.1x GB200 NVL72 power.
Time to first job failure (100k H100)~26.3 min (single 400G) → ~42 min (Cedar-7)Optics failures dominate; drives memory-reconstruction fault recovery over checkpointing.
Celestial AI link efficiency~2.5 pJ/bit E-O-E (+0.7 laser) vs ~10 pJ/bit copperMarvell buy: $500M run-rate exiting FY28, guided to $1B by CY28-end; Trainium 4 target.
Lightmatter Passage M1000 photonic interposer~2,100 mm² photonic interposer, 4 tiles, 32 optical waveguides/tile @127µm pitch, >95% electrical-assembly yield, ~680W cooled from the concentrated test areaCo-packaged optical interposer packaging details (ECTC 2026, SemiAnalysis).
Marvell optical I/O (OMIB / Photonic Fabric)OMIB reaches 1.8 Tbps/mm² bandwidth density; Photonic Fabric uses electro-absorption modulators (EAMs), 224 Gb/s bidirectional per test chipEAM path vs micro-ring modulators (ECTC 2026, SemiAnalysis).

Key Theses

1. CPO's killer app is scale-up, not scale-out — because in scale-out the TCO win is diluted to almost nothing.
CPO cuts scale-out transceiver power ~84% and cost ~86%, but networking is only ~9% of cluster power to begin with, so a 3-layer CPO network saves just ~2% total cluster power and ~3% cluster cost (2-layer: ~4% power, ~7% cost). Against that sit lost transceiver-vendor bargaining power, blast-radius fear and field-service pain. Scale-up is where the reach limit and SerDes wall actually bite, so that is the real prize.
“CPO for scale-up is now not a matter of if and why, but when and how.”— Co-Packaged Optics (CPO) Book – Scaling with Light (2026-01-01), SemiAnalysis
2. Copper's SerDes runway is nearly gone; Nvidia's Rubin '448G' is a bi-directional trick, and that erosion of the NVLink moat lets AMD and hyperscalers catch up.
NVLink bandwidth grew 11x from v1 to v5 almost entirely via SerDes speed (20G→200G), not lane count. NVLink 6 stays on 200G and doubles bandwidth only by simultaneous bi-directional signaling (224G TX + 224G RX on one differential pair with echo cancellation), avoiding a doubling of the ~5,000 backplane cables to ~10,000. True 448G uni-directional needs PAM6/8 and is uncertain. Once SerDes stalls, Nvidia's scale-up moat narrows.
“For Nvidia, whose NVLink scale up fabric is an important moat, this roadblock could make it easier for competitors such as AMD, and the hyperscalers to catch up.”— Co-Packaged Optics (CPO) Book – Scaling with Light (2026-01-01), SemiAnalysis
3. The DSP is 'public enemy number one' of optical cost and power — which is why the industry is trying to delete it.
The DSP is ~50% of an 800G SR8 module's power and 20-30% of its BoM. An 18k-GPU GB300 cluster's back-end alone would burn ~480 kW of DSP power (~1.8 kW per rack). LPO (linear pluggable) attacks it first by driving optics straight from switch SerDes, but LPO hasn't taken off as Marvell's Loi Nguyen predicted; CPO finishes the job by co-packaging the engine so no DSP is needed at all.
“So – accounting for 50% of the power draw and 20-30% of the BoM of a typical transceiver – some regard DSPs as public enemy number one of cost and power efficiency.”— Co-Packaged Optics (CPO) Book – Scaling with Light (2026-01-01), SemiAnalysis
4. TSMC's COUPE has quietly won CPO integration — even taking Broadcom, the CPO pioneer, off its own packaging.
COUPE's bumpless SoIC hybrid bonding gives >23x the bandwidth density of bumped integration at iso-power. Broadcom's own FOWLP (via SPIL) can't scale past 100G/lane due to through-mold-via parasitics, forcing Broadcom to move future CPO to COUPE — and to switch from edge-coupling+MZM to grating-coupling+MRM, essentially starting fresh. Adopting COUPE locks the customer into TSMC-made PICs, since TSMC won't package others' SiPho wafers.
“This highlights TSMC's technological edge, enabling them to secure wins even in optics, a domain where they have historically been considered weaker.”— Co-Packaged Optics (CPO) Book – Scaling with Light (2026-01-01), SemiAnalysis
5. The 'optical boogeyman' that actually cuts transceiver demand is copper NVLink inside the rack — not the NVL72 itself.
When Jensen said NVL72's copper NVLink saved 20 kW of transceivers, optical investors panicked that scale-out optics demand would fall. SemiAnalysis: false — the NVL72 still has 72 back-end OSFP ports (1 per GPU), the same transceiver-to-GPU ratio as H100, so scale-out optics scale with GPU count. Using optics for the 900 GB/s NVLink spine would have needed 648 1.6T transceivers at ~$550k/rack of BoM (~$2.2M at 75% margin) — which is exactly why Nvidia used 5,184 copper cables.
“For the back end scale out network – the NVL72 rack showcased at GTC still has 72 OSFP ports at 400G / 800G – one for each GPU... As GPU network sizes scale, the number of optical transceivers required also scales.”— Nvidia's Optical Boogeyman (2024-03-25), SemiAnalysis
6. Reliability, not TCO, is CPO's real gate — 80% of returned pluggable modules are 'no trouble found', and Meta's 15 CPO switches over 11 months isn't enough proof.
A ~1M-link pluggable cluster sees dozens of link flaps/day, and 80% of returned modules are 'no trouble found' — CPO's determinism is the draw. Meta/Broadcom (Bailly 51.2T) showed 2.6M-hour MTBF for CPO vs 0.5-1M for 2xFR4 pluggables, zero UCWs to 4M port-device-hours. But 15M '400G port-device-hours' is only ~15 switches × 11 months in a lab; a 0.06% unserviceable failure has a 64-port blast radius. SemiAnalysis calls for far larger field tests before billions commit.
“80% of optical modules that are returned due to some link failure are 'no trouble found'.”— Co-Packaged Optics (CPO) Book – Scaling with Light (2026-01-01), SemiAnalysis
7. Ethernet is clawing back share from InfiniBand; Nvidia's own Spectrum-X now out-ships Quantum InfiniBand in Blackwell.
Ethernet lost early ground to InfiniBand but is returning on cost and flexibility. Spectrum-X SN5600 and Broadcom Tomahawk 5 both offer 128x400G vs Quantum-2's 64x400G, letting a 100k cluster be 3 tiers not 4 (1.33x fewer transceivers). Hyperscalers pick Tomahawk 5 to dodge the 'Nvidia tax' and mix any transceiver — the cost of admission is patching NCCL yourself. UEC standardizes these gains across the industry.
“Even Nvidia recognizes the dominance of Ethernet and with the Blackwell generation, Spectrum-X Ethernet is out shipping their Quantum InfiniBand by a large amount.”— The New AI Networks | Ultra Ethernet | UALink vs Broadcom SUE (2025-06-11), SemiAnalysis
8. Optical circuit switching gives Google a >$3B structural cost and power edge — and threatens Broadcom's spine-switch business.
Google's in-house Apollo/Palomar 136x136 OCS (MEMS mirrors, O-band 1310nm) draws ~108 W vs ~3,000 W for an equivalent electrical switch, and because it's bandwidth/wavelength-agnostic it survives multiple optics upgrades — so Google eliminates Broadcom switches in the spine and reuses OCS across generations. Google claims +30% throughput, -40% power, -30% capex, 50x less downtime, and used it to train PaLM; OCS underpins every TPU v4/v5 pod.
“This custom networking stack enables them to save at least $3 billion versus the industry standard implemented by competitors such as Amazon and Microsoft.”— Google OCS Apollo: The >$3 Billion Game-Changer (2023-03-17), SemiAnalysis
9. Broadcom is the silent #2 AI chip company — its networking + custom-silicon franchise (Google TPU, Meta MTIA, now OpenAI) is the anti-Nvidia bet.
Networking is the second-largest AI infra spend bucket after accelerators, and Broadcom leads merchant switching (Tomahawk/Jericho) plus optical components. Its custom-silicon arm designs Google's TPU and Meta's MTIA and grew from <20% of LSI revenue to $2-3B+. In CPO, Broadcom shipped the first production systems (Bailly, Humboldt) and its CPO experience made it OpenAI's ASIC design partner — precisely because CPO is on OpenAI's roadmap.
“Broadcom is the second largest AI chip company in the world in terms of revenue behind NVIDIA, with multiple billions of dollars of accelerator sales.”— Broadcom's Google TPU Revenue Explosion (2023-08-30), SemiAnalysis
10. Marvell's DSP moat is under 'death by a thousand cuts' — and the CPO era favors rivals who are ahead in productization.
Marvell (via Inphi) leads 112G PAM4 DSPs/TIAs and coherent optics, and had 100% of Nvidia's H100-gen DSP. But Broadcom, Nvidia, Arista, Microsoft, Meta and Macom are all designing Marvell out — Broadcom broke into Nvidia's GB200 DSP, and in CPO, Broadcom/Intel/Nvidia/Ayar are ahead in productization timelines. Marvell's counter is custom XPUs and its multi-billion-dollar Celestial AI buy — with a $2.25B revenue-contingent earn-out — to own scale-up CPO for AWS Trainium 4.
“All the other titans in networking including Broadcom, Nvidia, Arista Networks, Microsoft, Meta, Macom, and many others are rallying toward designing Marvell out.”— Marvell's DSP Dilemma? Networking's Tectonic Shift (2023-03-08), SemiAnalysis
11. Huawei's CloudMatrix 384 shows a system-level win — beating GB200 NVL72 on aggregate compute by throwing 100% optics and ~4x the power at the problem.
CM384 wires 384 Ascend 910C chips all-to-all across 16 racks (100% optics, 0% copper, ~6,912 400G LPO transceivers, ~18 per chip). It delivers ~300 PFLOPs dense BF16 (~1.7x GB200 NVL72), 3.6x memory capacity and 2.1x bandwidth — but at 4.1x the power and 2.5x worse power/FLOP. Because China is silicon-constrained not power-constrained, forgoing copper density for optical scale-up is the rational trade. It resembles Nvidia's abandoned DGX H100 NVL256 'Ranger'.
“Huawei is a generation behind in chips, but its scale-up solution is arguably a generation ahead of Nvidia and AMD's current products on the market.”— Huawei AI CloudMatrix 384 (2025-04-16), SemiAnalysis
12. Rubin's networking innovation is evolutionary, but content growth is real — 2x NICs per GPU, ~2.3x high-end PCB area, and cableless design.
VR NVL72 doubles scale-out to 1.6T per GPU by putting two 800G ConnectX-9 packages per GPU (not one faster NIC), enabling multi-plane deployments; CX-9 adds 800G Ethernet on 4x200G SerDes. NVLink Switch count doubles to 36/rack. Cableless design (no flyover cables except switch-to-SMM) forces PCB upgrades to M8/M9 CCL, HVLP4 foil and (debated) quartz cloth, growing high-end PCB area ~2.3x vs GB300. VR NVL72 costs ~45% more per GPU than GB300.
“NVLink 6 used in Vera Rubin doubles NVLink bandwidth by implementing bi-directional signaling over the same number of copper cables - effectively delivering 4 Lanes of 200G per NVLink.”— Vera Rubin – Extreme Co-Design (2026-02-25), SemiAnalysis
13. High-radix 'switch-in-a-box' lets Nvidia flatten networks and hide the shuffle — a real selling point that also raises switching cost.
Nvidia's CPO switches present as very high port count — Quantum 3450 = 144x800G, Spectrum 6800 = 512x800G — by internally shuffling across 4 ASICs (a multi-plane 'topology in a box'). Spectrum 6800 can connect 131,072 GPUs on 2 layers vs 8,192 for the 6810, letting customers drop from 3 layers to 2 and skip patch panels/octopus cables. Because a 2-tier fabric scales with the square of switch radix, doubling logical ports ~4x's the cluster size — the magic of high radix.
“the Spectrum 6800 at 512 ports of 800G can connect 131,072 GPUs.”— Co-Packaged Optics (CPO) Book – Scaling with Light (2026-01-01), SemiAnalysis

Article Deep-Dives

Co-Packaged Optics (CPO) Book – Scaling with Light for the Next Wave of Interconnect (2026-01-01)
The sector's defining 24k-word treatise. Five parts: (1) TCO — CPO's scale-out savings dilute to 2-7% at cluster level, so scale-up is the killer app; (2) the tech — DSP/LPO/CPO evolution, SerDes hitting the 224G→448G wall, Wide I/O (UCIe ~10 Tbit/s/mm), co-packaged copper as a 448G bridge; (3) bringing it to market — TSMC COUPE bumpless SoIC (>23x density), edge vs grating coupling, and the MZM/MRM/EAM modulator trade-offs (Nvidia ships 200G MRMs); (4) products — Nvidia Quantum-X/Spectrum-X, Broadcom Bailly/Davisson, Intel/MediaTek roadmaps, and deep profiles of Ayar Labs, Nubis (→Ciena), Celestial AI (→Marvell, EAM/Trainium 4), Lightmatter (M1000 optical interposer), Xscape, Ranovus, Scintil; (5) Nvidia's CPO supply chain (paywalled). Verdict: scale-out CPO ships only 10-15k units in 2026 as a pipe-cleaner; the real deployment is scale-up at Rubin Ultra/Feynman.
Vera Rubin – Extreme Co-Design: An Evolution from Grace Blackwell Oberon (2026-02-25)
Full teardown of Nvidia's Rubin platform's six co-designed silicon products and the VR NVL72 rack. Networking core: NVLink 6 doubles bandwidth via bi-directional SerDes (224G TX+224G RX per differential pair with echo cancellation over ~1m), keeping backplane cable count flat but doubling NVLink Switch chips to 36/rack. ConnectX-9 stays at 800G but doubles NICs per GPU to give 1.6T scale-out; Spectrum-6 CPO (102.4T/409.6T) enables larger scale-out. The rack goes cableless — only one flyover cable (switch-to-SMM) remains — forcing PCB upgrades to M8/M9 CCL, HVLP4 copper, 32-layer NVLink6 boards, and (debated) quartz cloth, growing high-end PCB area ~2.3x. VR NVL72 is ~45% more expensive per GPU than GB300, and is the first Nvidia generation to see CPO in the scale-out back-end.
GB200 Hardware Architecture - Component Supply Chain & BOM (2024-07-17)
The definitive GB200 rack BoM across 50+ subcomponents and four form factors (NVL72, NVL36x2, Ariel, x86 Miranda). Networking essence: four fabrics (front-end Ethernet, back-end IB/Ethernet, NVLink scale-up, OOB); NVLink uses 5,184 copper differential pairs on Amphenol Paladin backplane — most cost in connectors, not wire — and copper beats optics decisively (optical spine = ~$550k/rack BoM, 20 kW). Debunks the '$3k/GPU NVLink' investor myth. CX-7→CX-8 drives 400G SR4 → 800G DR4 optics and 1.6T DR8; Nvidia expands transceiver supply to Eoptolink (800G LPO/1.6T DSP) beyond Fabrinet/InnoLight, and Broadcom breaks Marvell's DSP monopoly.
100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability (2024-06-17)
Anatomy of frontier training clusters. A 100k H100 cluster needs ~150 MW IT load ≈ 1.31 TWh/yr (≈1.59 TWh at ~181.5 MW facility power, PUE≈1.21). Network design follows parallelism (tensor within node/NVLink, pipeline within island, data across islands). Rail-optimized vs middle-of-rack is a copper-vs-optics choice: a non-rail design converts 98,304 transceivers to cheap DAC copper (25-33% of the GPU fabric). Reliability dominates: optics link failures give ~26 min to first job failure, forcing memory-reconstruction recovery. Spectrum-X vs Tomahawk 5: TH5 dodges the 'Nvidia tax' and needs its own NCCL work; Spectrum-X's first gen needs power-hungry BlueField-3 (+5 MW/100k).
The New AI Networks | Ultra Ethernet UEC | UALink vs Broadcom Scale Up Ethernet SUE (2025-06-11)
In-depth read of the open-standard stack challenging Nvidia. UEC RC1 (565 pages) hardens Ethernet for scale-out: hardware-accelerated LibFabric, the 'Job' abstraction, packet spraying with entropy-based load balancing, time-based congestion control (sub-500ns) that deprecates RoCE/DCQCN/PFC, and post-quantum security. UALink and Broadcom SUE (a lean 20-page spec) target scale-up only — single switch layer, ≤1024 ports — and will ship as IP blocks inside host chips (like NVLink). SUE delegates spraying/load-balancing to software (a win for software-strong firms); UALink is heavier with credit-based flow control. UEC lags on referencing 200G links.
Google OCS Apollo: The >$3 Billion Game-Changer in Datacenter Networking (2023-03-17)
How Google's in-house optical circuit switching rewires datacenter economics. Apollo replaces electrical packet switches in the spine with MEMS-mirror OCS that redirect light without decoding packets — bandwidth/wavelength-agnostic and 'set-and-forget'. The Palomar 136x136 OCS (moved to O-band 1310nm to cut crosstalk) draws ~108 W vs ~3,000 W for an equivalent EPS, is reusable across optics-speed upgrades (~70% of EPS capex), and eliminates Broadcom spine switches. Google claims +30% throughput, -40% power, -30% capex, 50x less downtime; OCS underpins all TPU v4/v5 systems and helped train PaLM.

Reference: Value Chain

Switch ASICs (scale-out & scale-up)Broadcom (Tomahawk 5/6, Jericho), Nvidia (Quantum InfiniBand, Spectrum-X Ethernet, NVLink Switch), Marvell (Teralynx/Innovium), Cisco (Silicon One) — The silicon that routes packets. Broadcom's Tomahawk 5/6 (128x400G / 102.4T) is the merchant standard hyperscalers use to escape the 'Nvidia tax'; Nvidia bundles Spectrum-X/Quantum with NCCL for first-class support; NVLink Switch (28.8T) is the scale-up crossbar with SHARP in-network compute.
▼
Optical DSP / retimers / AEC chipsMarvell (Inphi), Broadcom, Nvidia (in-house 1.6T DSP), Astera Labs, Credo, MaxLinear, Semtech — The DSP retimes and FEC-codes the electrical signal in every transceiver. Marvell (via Inphi) led 100% of Nvidia's H100-gen DSP; Broadcom broke in at GB200. Nvidia taped out an internal 1.6T DSP but it guzzles too much power to ramp. Astera Labs supplies PCIe retimers/AECs.
▼
Pluggable transceivers & optical modulesInnoLight, Eoptolink, Fabrinet, Coherent, Lumentum, Nvidia LinkX, Cloud Light, Accelink — Assemble laser + modulator + DSP into 400G/800G/1.6T modules (DR/FR/SR). Fabrinet + InnoLight were the bulk of Nvidia supply; Eoptolink added for 800G LPO / 1.6T DSP. Nvidia's own LinkX transceivers carry a large premium over generic parts. Modules are ~60% of scale-out networking cost.
▼
CPO optical engines & startupsAyar Labs (TeraPHY), Celestial AI→Marvell (Photonic Fabric/EAM), Lightmatter (Passage M1000), Nubis→Ciena, Xscape, Ranovus, Scintil — OE-chiplet and optical-interposer vendors racing to be designed into hyperscaler XPUs. Ayar uses UCIe optical retimers + MRM; Celestial (bought by Marvell in a multi-billion-dollar deal) uses EAM on optical interposers, targeting AWS Trainium 4; Lightmatter builds a 4,000 mm2 optical interposer with 114 Tbit/s and built-in OCS.
▼
Silicon-photonics foundries & laser sourcesTSMC (COUPE), GlobalFoundries (Fotonix 45CLO), Tower, AMF, Intel; lasers: Lumentum, Sivers, Macom, Coherent, Scintil — Fab the PIC/EIC and integrate them. TSMC's COUPE (bumpless SoIC) is winning the next-gen even from GF/Tower, who led legacy SiPho. External Laser Sources (ELS) are the top failure point, so most CPO uses pluggable/redundant lasers (Ayar's SuperNova by Sivers; Lightmatter's GUIDE integrates hundreds of InP lasers on one chip).
▼
Copper backplane, connectors & cablesAmphenol (Paladin/DensiLink), Luxshare (co-packaged copper), TE, Samtec (Si-Fly), Bizlink — NVLink's copper spine — GB200 NVL72 uses 5,184 differential pairs / cables. Most cost is in connectors (crosstalk-suppressing), not wire. Co-packaged copper (CPC) puts twinax straight on the substrate, a simpler path to 448G. Amphenol is Nvidia's primary NVLink backplane source.
▼
High-speed PCB / CCL / substratesPanasonic (Megtron M7/M8/M9 CCL), Doosan, EMC, glass cloth (Q-glass/quartz), HVLP copper foil; ABF substrates — As SerDes hit 224G+, insertion loss forces CCL upgrades (M7→M8/M9), HVLP4 ultra-low-profile copper foil, and debated quartz (Q) glass cloth. Rubin's cableless NVL72 grows high-end PCB area ~2.3x vs GB300, with the NVLink6 board at 32 layers M8+ — a quiet content-growth story for materials suppliers.

Reference: Core Concepts

Scale-up vs Scale-out. Scale-up = the ultra-fast, low-latency, all-to-all fabric fusing GPUs into one shared-memory 'world' (NVLink, UALink, SUE); ~9x the per-GPU bandwidth of scale-out. Scale-out = the back-end fabric linking racks/pods (InfiniBand or Ethernet). Scale-up demands are so much higher that scale-up interconnect TAM already dwarfs scale-out.

Co-Packaged Optics (CPO) / NPO / OBO. Placing the optical engine on the same package as the switch/XPU ASIC, eliminating the DSP and the 15-30 cm electrical run to a front-panel transceiver. NPO = optical engine socketed on a nearby substrate (detachable); OBO = engine on the system PCB ('worst of both worlds'). CPO cuts link energy 65-73% and removes copper's reach limit.

DSP / LPO / DR/FR optics. The DSP retimes and error-corrects the electrical signal inside a pluggable transceiver; it is the single most power-hungry and costly module component (~50% of an 800G SR8's power, 20-30% of BoM). LPO (linear pluggable) tries to delete the DSP by driving optics straight from switch SerDes; DR (parallel single-mode, ~500m), FR (WDM, ~2km) and LR (~6-10km) are the datacenter reach classes.

SerDes & the 224G→448G wall. SerDes serialize parallel data onto a differential pair; Nvidia/Broadcom lead at 224G. Higher speeds raise insertion loss and only survive over very short reach. A 448 Gb/s raw lane can be run as ~224-GBd PAM4 (2 bits/symbol); PAM6/8 are alternative baud-rate/channel trade-offs, not requirements — the hard part is the baud rate, channel loss and power. Nvidia's Rubin '448G per lane' is actually bi-directional 224G+224G sharing one wire — a clever dodge, not a real doubling.

Modulators: MZM vs MRM vs EAM. MZM (Mach-Zehnder): easiest, thermally stable, but huge (~12,000 µm2). MRM (micro-ring): tiny (25-225 µm2), built-in WDM, but 10-100x more temperature-sensitive — a 2°C drift shifts resonance past collapse. EAM (electro-absorption): compact, thermally tolerant to ~80°C, Celestial AI's differentiator but stuck in C-band. Nvidia shocked the industry by shipping 200G-per-lane MRMs.

TSMC COUPE. TSMC's 'Compact Universal Photonic Engine' — the emerging standard for heterogeneously integrating a PIC (photonic IC, N65) with an EIC (electronic IC, N6) via bumpless SoIC hybrid bonding, giving >23x the bandwidth density of bumped integration at iso-power. Ships first in Nvidia's CPO; Broadcom, Ayar, Lightmatter all pivoting to it. Locks customers to TSMC-made PICs.

World size & rail-optimized topology. World size = the number of GPUs in a single all-to-all scale-up domain (GB200 jumped from 8 to 72). Rail-optimized = each server's 8 GPUs connect to 8 different leaf switches so any GPU reaches distant GPUs in one hop — great for MoE all-to-all, but forces optics (vs. cheap copper in a middle-of-rack design). A nonblocking k-ary fat tree supports ~k^3/4 hosts; real host count depends on how ports are split among endpoints/uplinks/downlinks and on oversubscription.

UEC / UALink / SUE. The open-standard answers to Nvidia's proprietary stack. UEC (Ultra Ethernet, 565-page spec) hardens Ethernet for scale-out with LibFabric, packet spraying and proper congestion control (deprecating RoCE/DCQCN). UALink and Broadcom's Scale-Up Ethernet (SUE, a lean 20-page spec) target scale-up only — single switch layer, up to 1024 ports — and will ship as IP blocks inside host chips.

Optical Circuit Switch (OCS). A MEMS-mirror switch that redirects light without decoding packets — bandwidth- and wavelength-agnostic, 'set-and-forget' (reconfig takes seconds). Google's in-house Apollo/Palomar 136x136 OCS uses ~108 W vs. ~3,000 W for an equivalent electrical switch, saves >$3B, and lets Google reuse the switch across optics-speed upgrades. Used in all TPU v4/v5 pods.

Open Questions

Sources (SemiAnalysis)

Independence & sourcing. This is independent analysis by Yicheng Yang, distilled from publicly accessible SemiAnalysis articles (free posts and free previews; no paywall circumvention) and verified against the underlying text. It is not affiliated with, endorsed by, or a substitute for SemiAnalysis — subscribe there for the full research. All referenced claims are sourced and linked per SemiAnalysis's attribution terms. No SemiAnalysis images are reproduced. Nothing here is investment advice.