NVLink
Nvidia's high-bandwidth GPU-to-GPU interconnect forming the scale-up network within a rack.
As defined in Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
Where it comes up · 8
- The AI Reset Is Here: Search, Jobs, and Everything Else w/ Anish Acharya, Dave Blundin, Salim Ismail2 claims
- Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner2 claims
- How AI Datacenters Eat the World2 claims
- Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat2 claims
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]2 claims
- The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis2 claims
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served1 claim
- Satya Nadella — How Microsoft is preparing for AGI1 claim
Most important claims · 10 of 14Search all claims for “NVLink” →
- Nvidia achieved a marginal cost reduction in computing of 100,000x over the course of 10 years, with Moore's Law alone accounting for about 100x, through accelerated computing, tensor cores, HBM, NVLink, new numerical precisions, and new architectures.Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner
- Huawei's Ascend 910B and C chips, packaged into the Cloud Matrix 384 system with 384 chips connected across 12 racks using optics and power distribution, represent an architecture that Nvidia attempted with the DGX H100 Ranger (256 GPUs) but failed to bring to production due to cost, power consumption, and reliability issues.The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis
- In Nvidia's rack topology, GPUs talk within a rack via fast NVLink (scale-up network) reaching any GPU in two hops through central NV switches, while leaving the rack uses a scale-out network up to a data-center switch that is typically about eight times slower in bandwidth.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Grace Blackwell with NVLink was architected specifically to optimize for inference with low time-to-first-token, requiring infinite bandwidth and infinite compute simultaneously to achieve millisecond response times with rich context.Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner
- Sharding a large matrix multiply across up to eight GPUs with NVLink reduces latency but scales sublinearly, so eight times the hardware yields perhaps 4 to 5 times the speed due to communication overhead, making tensor-parallel sharding a poor choice for throughput-optimized serving.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Nvidia's GB200 NVL72 rack contains over 5,000 wires and two miles of copper, using copper-based NVLink interconnects throughout the rack because optical interconnects would consume 20,000 watts more power, illustrating the priority of power efficiency over distance in AI datacenter design.How AI Datacenters Eat the World
- The interconnect between GPU clusters (NVLink spine) is becoming a dominant technical constraint and innovation focus, with Nvidia's 130 terabytes/second spine transferring more data than peak internet traffic, which drives both performance and cooling infrastructure requirements.The AI Reset Is Here: Search, Jobs, and Everything Else w/ Anish Acharya, Dave Blundin, Salim Ismail
- NVLink is mandatory for low-latency inference and tensor parallelism is basically Nvidia-only, so Sail does not care much about low latency and instead optimizes flops per dollar using other vendors' chips with alternative parallelism such as expert or pipeline parallelism.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Huawei's Ascend 910B and C chips, integrated into the Cloud Matrix 384 system architecture with 384 chips connected across 12 racks using optics and power infrastructure, represents an architecture that Nvidia attempted and failed to deploy called DGX H100 Ranger, which had 256 Nvidia GPUs connected in NVLink but proved too expensive, power-hungry, and unreliable to bring to production.The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis
- Because Moore's Law now yields only ~25% per year, the only way to achieve 10x-100x leaps is to change the algorithm and how it's computed yearly; Nvidia's general programmability via CUDA enables new architectures (hybrid SSMs, fused diffusion/autoregressive models, MoEs), which is why Blackwell achieved roughly 50x Hopper's efficiency despite only ~75% transistor improvement.Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat