Will scaling work? [Narration]
Dwarkesh Podcast
2024/01/19
Will scaling work? [Narration]
Will scaling work? [Narration]

Dwarkesh Podcast
2024/01/19
This podcast explores the central debate in artificial intelligence: whether simply scaling up current AI models with more data and compute will be enough to achieve human-level intelligence, or if fundamental breakthroughs are still needed.
The discussion pits skeptics against believers on the path to AGI. Skeptics argue we will soon exhaust high-quality training data and that current benchmarks test memorization, not true reasoning. They also question whether the compression achieved during training equates to intelligence. Believers counter that synthetic data and self-play can overcome data bottlenecks, pointing to consistent performance improvements across many orders of magnitude in compute. They compare the scaling trajectory to Moore's Law, arguing it has historically worked despite a lack of full theoretical understanding. The narrator ultimately assigns a 70% probability of achieving transformative AI by 2040, acknowledging key challenges like the effectiveness of self-play, murky theoretical foundations, and the need for models to develop insight-based learning similar to humans.
00:00
00:00
We will soon run out of high-quality data
03:28
03:28
Scaling a simple transformer could produce AGI.
10:21
10:21
Benchmarks test memorization, not reasoning
17:14
17:14
Scaling has historically worked despite lacking full theoretical understanding
23:33
23:33
Insight-based learning is incompatible with gradient descent.