factual
Missing middle ground between training and in-context learning
Today's systems lack a middle ground between large-scale training (producing a single static snapshot model) and in-context learning (sample-efficient but destroyed each instance); a useful capability would combine online learning with cognitive skills like introspecting on one's own knowledge and actively seeking out knowledge to fill holes—partly unexplored because ever-longer context windows removed the incentive.
factualpending
Evidence Quote
“People haven't really pushed too hard on this middle ground between large-scale training... and on the other hand in-context learning.”
Source
John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:30 AM
Your notes
Loading notes...