LSTM (Long Short-Term Memory)
A gated RNN variant designed to address long-term dependencies by using memory cells and gates that control information flow, enabling better learning on longer sequences.
As defined in 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
Where it comes up · 4
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT4 claims
- Illustrated Guide to Transformers Neural Network: A step by step explanation1 claim
- OpenAI's Greg Brockman: Why Human Attention Is the New Bottleneck1 claim
- How ChatGPT changed everything: Thoughts from the frontline of the AI/ML revolution1 claim
Most important claims · 7 of 7Search all claims for “LSTM (Long Short-Term Memory)” →
- RNNs suffer from short-term memory because they have a shorter window to reference from compared to GRUs and LSTMs, which have greater capacity to achieve longer-term memory with a longer reference window.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Each Transformer layer in both encoder and decoder can process all input tokens in parallel, speeding up training by a significant margin compared to LSTM training which had to step through each input sequentially.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- LSTMs and GRUs had two main problems: (1) long-term dependency issues where networks forget information from the distant past because they cannot encode everything into a hidden state vector, and (2) slow sequential training because each token must pass through the RNN cell one-by-one sequentially, with no parallelization possible since each token's output depends on everything that came before it.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Gated RNNs (GRUs and LSTMs) add special gates to the basic RNN structure to selectively update and forget new information into and from the hidden state, showing massive improvement in the neural network's ability to learn with longer sequences compared to standard RNNs.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- The encoder-decoder architecture (introduced in 2014) solves sequence-to-sequence tasks by passing the entire input sequence token-by-token through an encoder RNN (LSTM or GRU), using the final hidden state as an embedding of the full input sequence, then passing this encoding as the initial hidden state to a decoder RNN that generates the target sequence.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- The Transformer architecture, introduced by Google in the paper 'Attention Is All You Need,' represents a breakthrough in sequence modeling by learning which parts of a sentence to attend to, enabling faster training, more efficient scaling, and superior performance compared to LSTMs and previous sequence modelsHow ChatGPT changed everything: Thoughts from the frontline of the AI/ML revolution
- OpenAI continuously innovates beyond basic scaling, including data formatting optimizations, architectural shifts like LSTM to transformer, and continued evolution beyond the 2017 transformer paper—these innovations amplify the base scaling law effect.OpenAI's Greg Brockman: Why Human Attention Is the New Bottleneck