Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
Training Data
1 DAYS AGO
Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

Training Data
1 DAYS AGO
This podcast explores a contrarian approach to AI, focusing on memory and continual learning as the key to making models truly useful for specific teams and companies. Instead of relying on ever-larger context windows or retrieval systems, the discussion centers on baking knowledge directly into a model's weights, allowing it to learn and improve over time like a seasoned employee.
The conversation introduces Engram, a company building a 'neolab' around the idea that memory and continual learning are two sides of the same coin. Their core premise is that instead of stuffing prompts with context or using RAG, a team's knowledge should be internalized into the model's weights through fine-tuning. This approach can match or beat frontier models while using up to 100x fewer tokens. The founders, Dan Biderman and Jessy Lin, argue that the real bottleneck in AI is not raw intelligence but the ability to learn continuously. They contrast their vision of personalized, privately trained models with the frontier labs' race toward a single, massive AGI. The discussion delves into the challenge of separating factual knowledge from algorithmic processing, the need for dedicated breakthroughs in memory, and the ultimate goal of creating a model that feels like an intern that genuinely gets smarter overnight.
00:00
00:00
Memory and continual learning are two sides of the same coin
08:17
08:17
Internalizing facts enables deeper reasoning
10:55
10:55
Deep learning merged knowledge and processing, but they are separating again.
18:49
18:49
Human memory's fuzzy representations inspire new AI approaches
21:28
21:28
Scaling compute on novel contexts over emergent capabilities
27:06
27:06
The key challenge is not storage but knowing how to query the right information