definition

Transformer Architecture Explained

A transformer is the architecture behind ChatGPT consisting of many layers of large continuous vectors representing words, concatenated to the model's context length; word vectors are randomly initialized then learned by optimizing the next-word prediction objective, and each word's representation is updated via an attention mechanism that takes a weighted average over neighboring words' representations, all words updating simultaneously, which is why it scales so efficiently.

definitionpending

Speaker

Yejin Choi

Evidence Quote

“each word is enhanced with technical term called attention mechanism... update your own representation as a weighted... average over all the words in your neighborhood”

Source

248 | Yejin Choi on AI and Common Sense— Sean Carroll's Mindscape
Created: 6/13/2026, 12:23:11 AM

Your notes

Loading notes...