GPT-2
A 1.5 billion parameter transformer-based language model trained on diverse text from the web, capable of generating remarkably coherent multi-sentence passages, used by OpenAI to study capabilities and risks of large language models.
As defined in Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94
Where it comes up · 3
- Ilya Sutskever: Deep Learning | Lex Fridman Podcast #943 claims
- How to Live with Computers2 claims
- Amjad Masad - The Future of Software Creation - [Invest Like the Best, EP.310]1 claim
Most important claims · 6 of 6Search all claims for “GPT-2” →
- GPT-2, a roughly 1-billion-parameter model, could complete text coherently and be trained for tasks like translation and small code generation without task-specific labeled datasets, a fundamental jump in generality over earlier machine learning, which required millions of labeled samples.Amjad Masad - The Future of Software Creation - [Invest Like the Best, EP.310]
- GPT-2 is a 1.5 billion parameter transformer trained on approximately 40 billion tokens of text from high-quality web pages (those linked from Reddit posts with more than three upvotes), and it achieved remarkable text generation quality that was surprising even to researchers familiar with prior progress in GANs and language modeling.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94
- OpenAI withheld the code for GPT-2 because of misuse fears, which Christian sees as a significant change in AI publication norms and a sign of ethical questions arising as the field approaches AGI.How to Live with Computers
- Staged release of powerful language models like GPT-2 (releasing smaller versions first to observe applications before releasing larger ones) is a rational approach to model release, allowing time for the community to discover beneficial and harmful uses before broader deployment.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94
- A single generic GPT-2 model reached state-of-the-art results on 18 of 20 leading computational linguistics benchmarks, which Christian treats as evidence of progress toward AGI.How to Live with Computers
- People quickly adapt to AI breakthroughs and move the bar for what impresses them, such as initially being amazed by GPT-2's text generation but then dismissing it when cognitive scientists question whether it 'truly understands,' a pattern that will likely continue with AI systems.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94