pre-training objective
The training goal (typically next-token prediction on internet text) that gives a model broad general knowledge but not necessarily research skill.
As defined in AMA: career advice given AGI, how I research ft. Sholto & Trenton
Where it comes up · 1
Most important claims · 2 of 2Search all claims for “pre-training objective” →
- Despite having all of human knowledge memorized, current LLMs do not make novel connections between different fields the way some humans have (e.g., linking magnesium deficiency brain structure to migraines), and there is no known example of an LLM ever making such a discovery.AMA: career advice given AGI, how I research ft. Sholto & Trenton
- The pre-training objective gives models flexible general knowledge but does not imbue the skill of making novel connections or doing research; significant RL on research-like tasks is likely needed before models can approach making novel scientific discoveries.AMA: career advice given AGI, how I research ft. Sholto & Trenton