scripod.com
Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Overview

Shownote

Highlights

Transcript

Chapters

Pins

Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Training Data

Jun 30
Training Data

Training Data

Jun 30
OverviewShownoteHighlightsTranscriptChaptersPins

Shownote

Dylan Patel, founder of SemiAnalysis, argues the biggest gains in AI don't come from faster chips, they come from software-hardware co-design. Optimizing the model, the kernels, and the silicon together turns a 2x here and a 2x there into 100x. He explains why DeepSeek's experts were shaped for Nvidia's Hopper (and why TPUs struggle to run it), why OpenAI's sparser models and Anthropic's denser ones pull them toward different hardware, and why the so-called CUDA moat was never really about CUDA. Dylan breaks down InferenceX, his living benchmark that runs the latest models on over $50M of donated hardware daily, tracking a roughly 60x annual drop in cost per unit of quality. He makes the case that inference will be a bigger market than oil, that the compute crunch persists because models expand the value of useful work faster than compute grows, and why Jensen Huang is bankrolling neoclouds to engineer a multipolar world. Hosted by Shaun Maguire and Sonya Huang, Sequoia Capital

Highlights

In this podcast, Dylan Patel, founder of SemiAnalysis, delves into the transformative potential of AI inference, arguing it will surpass oil as a market. He explains how hardware-software co-design, not just faster chips, drives massive efficiency gains, and discusses the creation of InferenceX, a living benchmark tracking cost drops. Patel also explores the strategic dynamics between neoclouds and hyperscalers, and why Nvidia's Jensen Huang is funding a multipolar compute world.
00:00
Engineers and former hedge fund analysts debate technology and cost.
11:19
Deep technical understanding requires personal contacts.
13:47
Inference will be a massive market, bigger than oil.
28:13
Co-optimizing across layers yields 100x improvements.
51:43
Model improvement is accelerating faster than compute growth.
1:04:17
Neoclouds' equity incentives drive faster delivery.

Chapters

From family motel to SemiAnalysis: Dylan Patel's journey and the culture of deep tech analysis
00:00
Launching SemiAnalysis: Overcoming personal struggles and building a global supply chain network
07:07
Inference as the next oil: Why AI inference will be a massive market and how InferenceX tracks it
13:47
The 100x secret: How hardware-software co-design and memory bandwidth drive AI efficiency
28:13
The economic flywheel: Why AI model improvements outpace compute growth and create profit loops
51:43
Neoclouds vs. hyperscalers: Why Jensen Huang is funding a multipolar world to prevent compute monopolies
1:04:17

Transcript

Dylan Patel: I think it's really fun inside of SemiAnalysis because we have the 90 people. And like, a big chunk of them are technologists, engineers across the whole supply chain. And then a big chunk is people who are formerly at hedge funds. And you see...