Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO
Latent Space: The AI Engineer Podcast
3 DAYS AGO
Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO
Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO
Latent Space: The AI Engineer Podcast
3 DAYS AGO
Modal CTO Akshat Bubna joins the podcast to discuss the company's evolution from a better runtime for bursty workloads to a full-fledged AI cloud platform, following their massive $355M Series C. The conversation explores why traditional cloud infrastructure like Kubernetes is ill-suited for AI and agent workloads, and how Modal is shifting its focus from developer experience to agent experience.
Modal was founded to address the limitations of Kubernetes for bursty, compute-heavy AI workloads, using Python decorators instead of YAML for a better developer experience. The company has since pivoted to focus on agent experience, building primitives like sandboxes, elastic inference, and GPU snapshotting. Key technical innovations include DeFlash for speculative decoding, Auto Endpoints for optimized inference, and a 17-cloud supercloud strategy for capacity. Modal's platform now supports elastic inference for custom models, RL rollouts requiring 100,000 sandboxes, and multi-node training for post-training workloads. The company emphasizes hard guardrails over LLM-mediated permissions for production agents and is developing Modal Bench to identify where agents struggle with observability and debugging. Modal sees infrastructure becoming exciting again due to AI's scale demands, and is expanding beyond Python to Go and TypeScript SDKs while partnering with foundation labs like Anthropic.
00:03
00:03
Party featured art installations alongside products
00:39
00:39
Python decorators instead of YAML for better developer experience
04:32
04:32
Agent-native primitives simplify deployment
06:21
06:21
Modal focuses on AI workloads, not general web servers.
09:14
09:14
Elastic scaling from zero to many GPUs.
12:12
12:12
Modal's primary use case is elastic inference for custom models.
15:24
15:24
DeFlash predicts blocks of tokens for multiplicative speedup.
20:01
20:01
Modal's value lies in expertise, open-source contributions, elasticity, and production-grade inference management.
22:01
22:01
Shift to an 'inference inflection' balancing GPU and CPU.
24:18
24:18
Modal runs on 17 neoclouds without its own data centers.
29:35
29:35
RDMA bypasses TCP for faster node-to-node transfers
35:46
35:46
LLMs are now good at generating Modal code
37:37
37:37
Proactive capacity management is like hedging fuel for airlines.
41:01
41:01
New permission models like Claude Code's LLM-mediated permissions are needed
43:06
43:06
Hard guardrails in sandboxes are essential
46:06
46:06
Focus on primitives enables diverse use cases beyond LLMs
48:31
48:31
Code-level infrastructure, not simple APIs.
51:53
51:53
Modal's focus on runtime sandboxes for agents was a better market fit
57:28
57:28
Still a long way to go
