scripod.com

He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI]

Shownote

The Transformer architecture (which powers ChatGPT and nearly all modern AI) might be trapping the industry in a localized rut, preventing us from finding true intelligent reasoning, according to the person who co-invented it. Llion Jones and Luke Darlow, ...

Highlights

The current trajectory of AI development, dominated by Transformer-based models, may be limiting the field's potential for achieving genuine reasoning capabilities. In this discussion, pioneers Llion Jones and Luke Darlow challenge the status quo, arguing that the industry’s reliance on a single architecture risks overlooking more biologically grounded and adaptive forms of intelligence.
04:13
Research freedom is essential for breakthrough innovation in AI.
17:30
Current AI shows jagged intelligence, revealing fundamental flaws in architecture
22:24
Scaling masks AI's inability to truly understand patterns like spirals
35:36
No trained model can predict 100–200 steps down the maze path in one shot.
36:01
CTM naturally allocates more time to difficult problems without forced computation limits.
57:45
Model shows leapfrogging behavior under constrained thinking time
1:04:22
Progress on Sudoku Bench would signify meaningful AI advancement.

Chapters

Stepping Back from Transformers
00:00
Introduction to Continuous Thought Machines (CTM)
00:43
The Changing Atmosphere of AI Research
01:09
Sakana’s Philosophy: Research Freedom
04:13
The Local Minimum of Large Language Models
07:45
Representation Problems: The Spiral Example
18:30
Technical Deep Dive: CTM Architecture
29:12
Adaptive Computation & Maze Solving
36:00
Model Calibration & Uncertainty
47:15
Sudoku Bench: Measuring True Reasoning
1:00:43

Transcript

Luke Darlow: This episode is brought to you by Indeed. You're ready to move your business forward, but first, you need to find the right team. Start your search with Indeed Sponsored Jobs. It can help you reach qualified candidates fast, ensuring your list...