scripod.com

AI Agents Hit The Verification Wall

The Daily AI Show
This episode explores the strategic use of high-cost AI models like Fable for planning and auditing, rather than routine execution. The hosts discuss compound engineering, verification loops, and the shift toward embedded AI engineering in enterprises, alongside a debate on Meta's aggressive safety testing practices.
The hosts argue that Fable is best used as an expert model for high-value tasks, such as auditing work plans and catching errors, while cheaper models handle routine execution. They introduce compound engineering as a method to close verification loops, noting that AI agents have made building cheap but shifted costs to verification. The discussion also covers Microsoft Frontier Co and the trend of embedding AI engineers into businesses for hands-on consulting. A new plugin called Caveman is highlighted for reducing token usage by 65% through terse prompting. The episode concludes with a debate on Meta's Project Cannes, where contractors posed as underage girls to stress-test competitors' AI models, raising questions about the ethics and oversight of such safety testing.
00:00
00:00
Fable analyzes complex legal and financial situations.
01:41
01:41
Confusion over Fable's usage credits and discount deadlines
08:12
08:12
Distilling high-end model reasoning into work plans
09:23
09:23
Building is cheap, verification is expensive.
15:40
15:40
Fable consumed many credits with poor results
25:09
25:09
Changing the worldview is the hardest part.
28:43
28:43
Expertise reduces the threat from AI
34:08
34:08
Caveman cuts token spend by 65%
38:14
38:14
Navigating unknown territory in AI
39:53
39:53
Ultra will be available in Codex
41:42
41:42
Helpful for delegating orthogonal tasks
44:18
44:18
Extreme stress testing of frontier models is necessary for safety.
58:08
58:08
Fable usage may end by July 7th