Sheela
Sheela
Home
Add
Questions
Thinkers
Claims
People
Concepts
Jargon
Sources
Health
Predictions
Collections
Domain Landscape
Contribute
Review
Dedup
Topic requests
Settings
Help

← Jargon

Jargonmachine learning· 2 claims· 1 episode

test time fine-tuning

Fine-tuning a model on the specific test problem at inference time so it adapts to that task, rather than keeping the model frozen.

As defined in Francois Chollet — Why the biggest AI models can't solve simple puzzles


Where it comes up · 1
  • Francois Chollet — Why the biggest AI models can't solve simple puzzles2 claims

Most important claims · 2 of 2Search all claims for “test time fine-tuning” →
  • Jack Cole's state-of-the-art LLM approach to ARC (35% with a 240M-parameter model) works only because of test-time fine-tuning—on-the-fly fine-tuning a version of the LLM for each test problem—which adds the active inference LLMs normally lack; without it, performance collapses to a negligible 1-2%.Francois Chollet — Why the biggest AI models can't solve simple puzzles
  • Two ARC approaches succeed at opposite ends of one spectrum: the LLM approach (Jack Cole) uses an enormous DSL of millions of vector programs mined by pre-training with very shallow recombination, while pure discrete program search uses a tiny DSL of 100-200 primitives with very deep, sophisticated search; the best value for compute, and the most progress toward near-term AGI, lies in merging them—building a richer learned bank of primitives plus some depth of search.Francois Chollet — Why the biggest AI models can't solve simple puzzles