language modeling
Training a neural network to predict the next token (word or character) in a sequence; the core objective for pre-training large language models like GPT.
As defined in OpenAI Co-Founder: “I just had to Quit”
Where it comes up · 2
- Authors of "THE AI CON" discussing the problematic hype about AI & what those working tech should do3 claims
- OpenAI Co-Founder: “I just had to Quit”1 claim
Most important claims · 4 of 4Search all claims for “language modeling” →
- The unsupervised sentiment neuron paper (2017) was a watershed moment because it demonstrated, for the first time, that semantics could emerge from training on a character-level language modeling objective—proving that neural networks could learn meaning, not just syntax.OpenAI Co-Founder: “I just had to Quit”
- The useful use cases for LLMs are where the goal is modeling the distribution of word forms in text, such as automatic transcription and machine translation, where language modeling serves as a component that identifies likely word sequences from ambiguous inputs—these are relatively narrow use cases.Authors of "THE AI CON" discussing the problematic hype about AI & what those working tech should do
- There is no path from language modeling to reasoning—reasoning requires building a model of the world and rules for how to get from facts to conclusions, which is fundamentally different from predicting the next word in a sequence.Authors of "THE AI CON" discussing the problematic hype about AI & what those working tech should do
- OpenAI's Whisper automatic transcription system, built on large language modeling technology, will sometimes output random strings that weren't in the audio input at all because the architecture loosely couples input to output, letting the language model do text fabrication, and all biases get amplified as a result.Authors of "THE AI CON" discussing the problematic hype about AI & what those working tech should do