RNN (Recurrent Neural Network)
Neural network architecture that processes sequential data by maintaining a hidden state that is updated at each time step, allowing it to capture information from previous tokens.
As defined in 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
Where it comes up · 2
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT7 claims
- Perplexity CEO: Disrupting Google Search with AI1 claim
Most important claims · 8 of 8Search all claims for “RNN (Recurrent Neural Network)” →
- The seminal 2017 paper 'Attention is All You Need' introduced Transformers, which replaced RNNs entirely with attention mechanisms as encoder-decoder architectures, with the ripple effects of this innovation still felt today.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- LSTMs and GRUs had two main problems: (1) long-term dependency issues where networks forget information from the distant past because they cannot encode everything into a hidden state vector, and (2) slow sequential training because each token must pass through the RNN cell one-by-one sequentially, with no parallelization possible since each token's output depends on everything that came before it.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Self-attention alleviates the long-term dependency issues of ordinary RNNs because models get a holistic view of the entire sequence and can attend to any token regardless of its position in the sequence.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Gated RNNs (GRUs and LSTMs) add special gates to the basic RNN structure to selectively update and forget new information into and from the hidden state, showing massive improvement in the neural network's ability to learn with longer sequences compared to standard RNNs.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- The encoder-decoder architecture (introduced in 2014) solves sequence-to-sequence tasks by passing the entire input sequence token-by-token through an encoder RNN (LSTM or GRU), using the final hidden state as an embedding of the full input sequence, then passing this encoding as the initial hidden state to a decoder RNN that generates the target sequence.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- The main difference between DeepMind and OpenAI is iteration speed: DeepMind thinks deeply to arrive at elegant solutions (like Transformer), while OpenAI tries v0 solutions quickly and iterates, and both approaches are necessary because pure thinking won't produce GPT-3.5 and pure iteration won't produce Transformer.Perplexity CEO: Disrupting Google Search with AI
- Bidirectional RNNs make two passes through data—one left-to-right and one right-to-left—producing two separate embeddings that are then added or concatenated to form a bidirectional embedding that captures context from both directions.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Recurrent Neural Networks (RNNs) preserve the order of tokens in a sequence by using an RNN cell initialized with a hidden state that updates at each time step by taking a token embedding as input and outputting a new embedding, with the hidden state encoding information about all previous tokens.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT