GPT-2 is a 1.5 billion parameter transformer trained on approximately 40 billion tokens of text from high-quality web pages (those linked from Reddit posts with more than three upvotes), and it achieved remarkable text generation quality that was surprising even to researchers familiar with prior progress in GANs and language modeling.

factualpending

Speaker

Ilya Sutskever

Evidence Quote

“gpt-2 is a transformer with one and a half billion parameters that was trained on upon about 40 billion tokens of text which were obtained from web pages that were linked to from reddit articles with more than three upvotes”

Source

Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94— Lex Fridman
Created: 8/10/2026, 10:49:52 PM

Your notes

Loading notes...