word embedding
Dense, low-dimensional vector representation of a word that captures semantic and syntactic structure, such that similar words have embeddings close to each other in vector space.
As defined in 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
Where it comes up · 3
- Illustrated Guide to Transformers Neural Network: A step by step explanation2 claims
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT2 claims
- Jeff Dean (Google): Exciting Trends in Machine Learning1 claim
Most important claims · 5 of 5Search all claims for “word embedding” →
- Transformers must add positional encoding information to input embeddings because unlike recurrent neural networks, transformers have no recurrence and therefore need explicit position information to understand word order.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Word embeddings—representing words as high-dimensional vectors learned through context prediction over trillions of tokens—capture semantic structure where similar words cluster together and directions in embedding space encode semantic relationships like the direction from king to queen being similar to the direction from man to woman, and going from present to past tense.Jeff Dean (Google): Exciting Trends in Machine Learning
- Word embedding layers function as lookup tables that map each word to a vector with continuous values representing that word, since neural networks learn through numbers and cannot directly process words.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Word embeddings are dense low-dimensional vectors that represent each word in a model's vocabulary and capture semantic and syntactical structure such that similar words have their embeddings close to each other in vector space.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- To retain sequential information in Transformers, special positional encodings are trained that represent specific indices in the sequence and are added to the corresponding word embeddings to form the final input token embedding.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT