Sheela
Sheela
Home
Add
Questions
Thinkers
Claims
People
Concepts
Jargon
Sources
Health
Predictions
Collections
Domain Landscape
Contribute
Review
Dedup
Topic requests
Settings
Help

← Jargon

Jargonmachine learning· 2 claims· 1 episode

pre-training objective

The training goal (typically next-token prediction on internet text) that gives a model broad general knowledge but not necessarily research skill.

As defined in AMA: career advice given AGI, how I research ft. Sholto & Trenton


Where it comes up · 1
  • AMA: career advice given AGI, how I research ft. Sholto & Trenton2 claims

Most important claims · 2 of 2Search all claims for “pre-training objective” →
  • Despite having all of human knowledge memorized, current LLMs do not make novel connections between different fields the way some humans have (e.g., linking magnesium deficiency brain structure to migraines), and there is no known example of an LLM ever making such a discovery.AMA: career advice given AGI, how I research ft. Sholto & Trenton
  • The pre-training objective gives models flexible general knowledge but does not imbue the skill of making novel connections or doing research; significant RL on research-like tasks is likely needed before models can approach making novel scientific discoveries.AMA: career advice given AGI, how I research ft. Sholto & Trenton