scripod.com
Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

Overview

Shownote

Highlights

Transcript

Chapters

Pins

Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

Latent Space: The AI Engineer Podcast

3 DAYS AGO
Latent Space: The AI Engineer Podcast

Latent Space: The AI Engineer Podcast

3 DAYS AGO
OverviewShownoteHighlightsTranscriptChaptersPins

Shownote

We’ve been running a bit of an Agent Cloud series surveying all the top inference/compute/cloud providers, from Databricks to Daytona to Railway and, even further back, E2B, but we’re excited to conclude this series returning to Modal, which has just raise...

Highlights

Modal CTO Akshat Bubna joins the podcast to discuss the company's evolution from a better runtime for bursty workloads to a full-fledged AI cloud platform, following their massive $355M Series C. The conversation explores why traditional cloud infrastructure like Kubernetes is ill-suited for AI and agent workloads, and how Modal is shifting its focus from developer experience to agent experience.
00:03
Party featured art installations alongside products
00:39
Python decorators instead of YAML for better developer experience
04:32
Agent-native primitives simplify deployment
06:21
Modal focuses on AI workloads, not general web servers.
09:14
Elastic scaling from zero to many GPUs.
12:12
Modal's primary use case is elastic inference for custom models.
15:24
DeFlash predicts blocks of tokens for multiplicative speedup.
20:01
Modal's value lies in expertise, open-source contributions, elasticity, and production-grade inference management.
22:01
Shift to an 'inference inflection' balancing GPU and CPU.
24:18
Modal runs on 17 neoclouds without its own data centers.
29:35
RDMA bypasses TCP for faster node-to-node transfers
35:46
LLMs are now good at generating Modal code
37:37
Proactive capacity management is like hedging fuel for airlines.
41:01
New permission models like Claude Code's LLM-mediated permissions are needed
43:06
Hard guardrails in sandboxes are essential
46:06
Focus on primitives enables diverse use cases beyond LLMs
48:31
Code-level infrastructure, not simple APIs.
51:53
Modal's focus on runtime sandboxes for agents was a better market fit
57:28
Still a long way to go

Chapters

Introduction
00:00
Modal’s origin and why Kubernetes wasn’t enough
00:39
Developer Experience → Agent Experience
04:32
Modal’s AI cloud primitives
06:21
Sandboxes, agent loops, and proto-Cognition
09:14
Elastic inference, GPU snapshotting, and 100,000 sandboxes
12:12
DeFlash, speculative decoding, and Auto Endpoints
15:24
Production-grade inference beyond raw GPUs
19:59
Background agents, Ramp Inspect, and the agent lifecycle
22:00
Modal’s 17-cloud supercloud strategy
24:08
Networked sandboxes, private IPv6, and RDMA
26:40
Multi-node training, post-training, and auto research
32:48
Compute strategy, capacity planning, and batch tiers
37:36
Open models, real-time AI, and production agent infra
40:55
Hard guardrails, managed agents, and specialized sandboxes
43:06
Why AI made infrastructure exciting again
46:06
Model APIs, differentiated products, and agentic video
48:30
CI, coding-agent infra, SDKs, and Modal Bench
51:50
Closing Thoughts
57:28

Transcript

swyx: We're here with Akshat of Modal, CTO of Modal, together with Vibhu. Congrats on your 3C, Thank you. Your party yesterday was amazing. Yeah. From all the photos and all the swag. Akshat Bubna: We had a bunch of art installations, which is kind of fun...