GRU (Gated Recurrent Unit)
A variant of RNN that uses gating mechanisms to selectively update and forget information in the hidden state, improving performance on longer sequences compared to vanilla RNNs.
As defined in 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
Where it comes up · 2
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT3 claims
- Illustrated Guide to Transformers Neural Network: A step by step explanation1 claim
Most important claims · 4 of 4Search all claims for “GRU (Gated Recurrent Unit)” →
- RNNs suffer from short-term memory because they have a shorter window to reference from compared to GRUs and LSTMs, which have greater capacity to achieve longer-term memory with a longer reference window.Illustrated Guide to Transformers Neural Network: A step by step explanation
- LSTMs and GRUs had two main problems: (1) long-term dependency issues where networks forget information from the distant past because they cannot encode everything into a hidden state vector, and (2) slow sequential training because each token must pass through the RNN cell one-by-one sequentially, with no parallelization possible since each token's output depends on everything that came before it.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Gated RNNs (GRUs and LSTMs) add special gates to the basic RNN structure to selectively update and forget new information into and from the hidden state, showing massive improvement in the neural network's ability to learn with longer sequences compared to standard RNNs.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- The encoder-decoder architecture (introduced in 2014) solves sequence-to-sequence tasks by passing the entire input sequence token-by-token through an encoder RNN (LSTM or GRU), using the final hidden state as an embedding of the full input sequence, then passing this encoding as the initial hidden state to a decoder RNN that generates the target sequence.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT