scripod.com

Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Training Data
In this podcast, Dylan Patel, founder of SemiAnalysis, delves into the transformative potential of AI inference, arguing it will surpass oil as a market. He explains how hardware-software co-design, not just faster chips, drives massive efficiency gains, and discusses the creation of InferenceX, a living benchmark tracking cost drops. Patel also explores the strategic dynamics between neoclouds and hyperscalers, and why Nvidia's Jensen Huang is funding a multipolar compute world.
Dylan Patel argues that AI inference will become a market larger than oil, with costs dropping 60x annually per unit of quality, as tracked by his InferenceX benchmark. He emphasizes that hardware-software co-design, optimizing models, kernels, and silicon together, yields 100x improvements, with memory bandwidth as a key bottleneck. Patel explains how model architecture choices make Google's TPUs and Nvidia's GPUs better suited for different AI companies, and that the CUDA moat is evolving as model companies prioritize custom kernels for any chip. He notes that AI model improvements expand economic value faster than compute grows, creating a pseudo-recursive self-improvement loop. Patel also discusses why neoclouds like CoreWeave outperform hyperscalers due to CPU-optimized features hurting GPU performance, and how Jensen Huang's strategy of funding neoclouds creates a multipolar world to prevent hyperscaler dominance.
00:00
00:00
Engineers and former hedge fund analysts debate technology and cost.
11:19
11:19
Deep technical understanding requires personal contacts.
13:47
13:47
Inference will be a massive market, bigger than oil.
28:13
28:13
Co-optimizing across layers yields 100x improvements.
51:43
51:43
Model improvement is accelerating faster than compute growth.
1:04:17
1:04:17
Neoclouds' equity incentives drive faster delivery.