Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
Training Data
1 DAYS AGO
Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
Training Data
1 DAYS AGO
Shownote
Shownote
Dan Biderman and Jessy Lin, co-founders of Engram, are building a neolab around memory and continual learning, which they call two sides of the same coin. Their contrarian premise: instead of stuffing ever-larger prompts into the context window or bolting ...
Highlights
Highlights
This podcast explores a contrarian approach to AI, focusing on memory and continual learning as the key to making models truly useful for specific teams and companies. Instead of relying on ever-larger context windows or retrieval systems, the discussion centers on baking knowledge directly into a model's weights, allowing it to learn and improve over time like a seasoned employee.
Chapters
Chapters
Memory and Learning: Two Sides of the Same Coin
00:00The 100x Token Advantage: Fine-Tuning for Efficiency
05:45The Great Divide: Factual Knowledge vs. Algorithmic Processing
10:55Why Frontier Labs Aren't Solving the Memory Problem
16:04Is Memory an Emergent Property or a Distinct Component?
21:28The Unsolved Problem: What to Internalize vs. What to Retrieve
27:06Transcript
Transcript
Jessy Lin: What about pre-training, or even post-training? Makes it possible for the models to generalize in these magical, emergent ways and controlling that process so that a company has a set of private data? How do we make the models learn that just as...
