test time fine-tuning
Fine-tuning a model on the specific test problem at inference time so it adapts to that task, rather than keeping the model frozen.
As defined in Francois Chollet — Why the biggest AI models can't solve simple puzzles
Where it comes up · 1
Most important claims · 2 of 2Search all claims for “test time fine-tuning” →
- Jack Cole's state-of-the-art LLM approach to ARC (35% with a 240M-parameter model) works only because of test-time fine-tuning—on-the-fly fine-tuning a version of the LLM for each test problem—which adds the active inference LLMs normally lack; without it, performance collapses to a negligible 1-2%.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- Two ARC approaches succeed at opposite ends of one spectrum: the LLM approach (Jack Cole) uses an enormous DSL of millions of vector programs mined by pre-training with very shallow recombination, while pure discrete program search uses a tiny DSL of 100-200 primitives with very deep, sophisticated search; the best value for compute, and the most progress toward near-term AGI, lies in merging them—building a richer learned bank of primitives plus some depth of search.Francois Chollet — Why the biggest AI models can't solve simple puzzles