Solutions & Applications
Telecom Network Sync · 5G Infrastructure · Data Center Connectivity · Optical Networks · Critical Networks
1. Network Topology
| Parameter | Value |
|---|---|
| Architecture | Two-tier leaf-spine InfiniBand fabric |
| Oversubscription Ratio | 1:1 (non-blocking) |
| Total Compute Nodes | 8 × GB200 NVL72 systems |
| Total GPU Count | 576 GPUs (72 per system) |
| Fabric Bandwidth | 1.6T per link, 12.8T bisection bandwidth |
| Leaf Layer | 3 × NVIDIA Q3400-RA switches |
| Spine Layer | 1 × NVIDIA Q3400-RA switch |
2. Complete Bill of Materials
Optical Transceiver Requirements
| Component | Specification | Qty | Location |
|---|---|---|---|
| 1.6T OSFP InfiniBand | HDR, 0–100m SMF | 216 | Switch-to-switch links |
| — Leaf Uplink | To Spine switches | 72 | 3 Leaf × 24 uplinks |
| — Leaf Downlink | To Spine switches | 72 | 3 Leaf × 24 downlinks |
| — Spine Layer | To Leaf switches | 72 | 1 Spine × 72 ports |
| 400G OSFP InfiniBand | NDR, 0–100m SMF | 576 | Server-to-leaf connections |
Switch & NIC Hardware
| Component | Model | Qty | Configuration |
|---|---|---|---|
| InfiniBand Leaf Switch | NVIDIA Q3400-RA | 3 | 24 downlinks + 12 uplinks per switch |
| InfiniBand Spine Switch | NVIDIA Q3400-RA | 1 | 72 ports (3×24 from leaves) |
| 400G InfiniBand NIC | NVIDIA ConnectX-7 VPI | 576 | 72 per GB200 system × 8 systems |
| InfiniBand Cables | MTP/MPO-24 to 6×LC duplex | 216 | For 1.6T OSFP connections |
| 400G DAC/AOC Cables | OSFP to OSFP, 3–5m | 576 | Alternative for short reaches |
3. Detailed Configuration Breakdown
Leaf Layer (3 Switches)
Switch: NVIDIA Q3400-RA, 36 × 1.6T OSFP ports
Server Connections: 24 × 400G downlinks per switch
Spine Connections: 12 × 1.6T uplinks per switch
Total Port Utilization: 24 + 12 = 36 ports (fully utilized)
Connectivity per Leaf: Each leaf connects to 8 GB200 nodes
Spine Layer (1 Switch)
Switch: NVIDIA Q3400-RA, 36 × 1.6T OSFP ports
Leaf Connections: 72 × 1.6T downlinks (24 from each of 3 leaves)
Redundancy: Optional second spine for N+1 (adds 72 more transceivers)
Server Configuration (8 × GB200 NVL72)
NICs per Server: 72 × ConnectX-7 400G VPI adapters
Transceivers per Server: 72 × 400G OSFP
Bandwidth per Server: 400G × 72 = 28.8 Tbps theoretical
Port Mapping: Each server connects to all 3 leaf switches (24 ports per leaf)
4. Performance Specifications
| Metric | Specification |
|---|---|
| Bisection Bandwidth | 12.8 Tbps full non-blocking |
| Switch-to-Switch Latency | < 600 ns |
| End-to-End Latency | < 1 µs |
| Message Rate | 200 million messages/sec per port |
| Fabric Bandwidth | 1.6T per link, 345.6T aggregate |
| Switch ASIC Bandwidth | 25.6 Tbps (Q3400-RA) |
| Power per 1.6T OSFP | 18–22W typical |
| Power per Switch | 3–5 kW (liquid cooling recommended) |
5. Optical Transceiver Specifications
| Spec | 1.6T OSFP (HDR InfiniBand) | 400G OSFP (NDR InfiniBand) |
|---|---|---|
| Form Factor | OSFP | OSFP |
| Data Rate | 1.6 Tbps (8 × 200G lanes) | 400 Gbps (8 × 50G lanes) |
| Protocol | HDR InfiniBand | NDR InfiniBand |
| Reach (MMF) | 0–100m OM4 | 0–100m OM4 |
| Reach (SMF) | 0–2km | 0–2km |
| Wavelength | 850nm VCSEL / 1310nm | 850nm VCSEL / 1310nm |
| Power | 18–22W | 10–12W |
| Temperature | 0°C to 70°C | 0°C to 70°C |
| Compatibility | Q3400-RA, Quantum-2 ASIC | ConnectX-7, Quantum-2 ASIC |
6. Configuration Alternatives
| Configuration | Transceivers | Cost Impact | Performance | Recommended For |
|---|---|---|---|---|
| Minimal (2 Leaf) | 144 × 1.6T + 576 × 400G | −25% switch cost | 2:1 oversubscription | Budget-constrained AI training |
| Recommended (3 Leaf) | 216 × 1.6T + 576 × 400G | Baseline | 1:1 non-blocking | Production AI/HPC clusters |
| Redundant (3+2) | 288 × 1.6T + 576 × 400G | +33% switch cost | N+1 fault tolerance | Mission-critical workloads |
| Dual-rail (2×NIC) | 216 × 1.6T + 1,152 × 400G | +100% NIC cost | 2× bandwidth per GPU | Extreme performance |
7. Key Technical Considerations
Thermal Management: 1.6T OSFP transceivers generate 18–22W each; ensure adequate cooling
Power Requirements: Each Q3400-RA switch consumes 3–5kW; plan power distribution accordingly
Cable Management: 576 fiber connections require structured cabling with proper bend radius
Compatibility Testing: All optical transceivers must be validated with NVIDIA switches and NICs
Firmware Management: Consistent firmware levels across all switches and NICs for optimal performance
Monitoring: Implement NVIDIA UFM or similar for fabric management and monitoring
Building an AI supercluster?
Our InfiniBand fabric engineers will model your topology, recommend a configuration, and deliver a complete BOM with lead times — typically within 48 hours.
Email: Info@apexallinone.com | WhatsApp/Phone: +852 9821 3834
