scripod.com
Notion’s Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs of Notion

Overview

Shownote

Highlights

Transcript

Chapters

Pins

Notion’s Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs of Notion

Latent Space: The AI Engineer Podcast

Apr 15
Latent Space: The AI Engineer Podcast

Latent Space: The AI Engineer Podcast

Apr 15
OverviewShownoteHighlightsTranscriptChaptersPins

Shownote

For all those who missed out on London, see you in Miami next week! Notion, the knowledge work decacorn, has been building AI tooling since before ChatGPT, with many hits from Q&A in 2023 and unified AI in 2024 and Meeting Notes in 2025. At the end of the...

Highlights

This episode features Notion's Sarah Sachs and Simon Last diving deep into the multi-year journey behind Custom Agents—a foundational shift in how productivity software interfaces with AI. They unpack the technical, organizational, and philosophical decisions that shaped one of the most ambitious agent-native systems in enterprise software today.
00:00
MCP is good for narrow, lightweight agents with tight permissions
00:39
Simon and Sarah from Notion join the Latent Space podcast
00:52
Early 2022 attempts failed because models were too dumb and had short context length
04:32
Notion's two crucial skills for frontier capabilities: avoiding swimming upstream and anticipating product development
11:28
Notion has rebuilt its Custom Agents 3–4 times
14:48
Jimmy’s image generation project on the database collections team became a full-fledged feature thanks to low-ego leadership and rapid iteration
15:43
The majority of traffic will come from agents in the future
19:13
Every team owns their own evals, many integrated into CI or run nightly
23:49
Notion's Last Exam only passes 30% of the time and has full-time staff dedicated to it
24:22
The Model Behavior Engineer role combines data science, product management, and prompt engineering to understand model capabilities and headroom
25:58
Supervision for coding agents can come from non-engineers like UXRE personnel who triage failures and guide investment
26:57
Software engineers at Notion are going through an identity crisis, realizing that delegation and context-switching are more important than code-writing
30:56
Custom agents route bugs to appropriate teams and post in Slack, replacing manual processes rather than people
32:22
There's a limit on the number of recursions to avoid infinite loops when composing agents
39:02
Using language models for deterministic tasks and interfacing with third-party providers is wasteful
42:52
Notion built its own mail and calendar in-house, spending time fine-tuning tools, building triggers, and using the right tools at the right time
47:46
Adding new tools hit a bottleneck due to token usage, efficiency, and quality trade-offs
51:01
Making it too easy to use can diminish the agent's capabilities
52:10
Custom agents can set and debug themselves, and users can ask about failures and update instructions
1:07:39
Most problems in the system are due to tool bugs rather than model issues

Chapters

C vs MCP Basics
00:00
Show Intro Notion Guests
00:38
Custom Agents Launch
00:52
AGI Pilled Bets
04:31
User Journeys Focus
08:58
Hackathons Culture
12:34
Team Size Org Design
15:43
Prototyping Demos
16:43
Model Degradation Talk
21:14
Model Behavior Engineers
24:22
Evals As Agent Harness
25:57
Future Of Engineers
26:56
Software Factory Design
28:45
Composing Agents Together
32:21
CLI vs MCP Tradeoffs
36:46
Choosing APIs MCP or Sandbox
40:41
Internal Tool Abstractions
43:09
Scaling Tool Ownership
48:42
Agents That Set Themselves Up
52:09
Credits and Usage Pricing
56:14

Transcript

Simon Last: Broadly speaking, I'm really bullish on CLIs. I'm still bullish on MCPs in a certain environment. I think it'd be really great for when you want a narrow, lightweight agent. I think there's definitely a lot of use cases where you don't want a f...