factual

Missing middle ground between training and in-context learning

Today's systems lack a middle ground between large-scale training (producing a single static snapshot model) and in-context learning (sample-efficient but destroyed each instance); a useful capability would combine online learning with cognitive skills like introspecting on one's own knowledge and actively seeking out knowledge to fill holes—partly unexplored because ever-longer context windows removed the incentive.

factualpending

Speaker

John Schulman (OpenAI Cofounder)

Evidence Quote

“People haven't really pushed too hard on this middle ground between large-scale training... and on the other hand in-context learning.”

Source

John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:30 AM

Your notes

Loading notes...