LSTMs and GRUs had two main problems: (1) long-term dependency issues where networks forget information from the distant past because they cannot encode everything into a hidden state vector, and (2) slow sequential training because each token must pass through the RNN cell one-by-one sequentially, with no parallelization possible since each token's output depends on everything that came before it.
causalpending
Speaker
Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]Evidence Quote
“they still had long-term dependency issues where the network will tend to forget things that were said long time in the past because it simply can't encode everything into its hidden State vector and number two training was quite slow because we had to pass each token one by one sequentially through the RNN cell”
Source
10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVBCreated: 8/11/2026, 1:00:15 AM
Your notes
Loading notes...