retraining
The process of taking a pre-trained model and training it again on new data and knowledge, making permanent changes to the model's weights that persist across all future uses
As defined in Feed Your OWN Documents to a Local Large Language Model!
Where it comes up · 1
Most important claims · 10 of 10Search all claims for “retraining” →
- Uploading documents to a model's context window is like providing a student with a temporary cheat sheet during an exam; the model can reference and use the documents to answer questions within that session, but once the session ends, the information is not retained unless the documents are uploaded again in the next session.Feed Your OWN Documents to a Local Large Language Model!
- The three primary methods for adding knowledge to existing large language models are: retraining the model with new data, using retrieval augmented generation (RAG), and uploading documents directly into the context window.Feed Your OWN Documents to a Local Large Language Model!
- Formal retraining of models like ChatGPT is impossible for most users because OpenAI provides no access to the ChatGPT model itself, so there is no way to directly modify or retrain it.Feed Your OWN Documents to a Local Large Language Model!
- Retraining a model is analogous to sending a student back to school: the model retains most prior knowledge but fine-tunes and adds new knowledge through repeated training cycles on updated material, making the new knowledge permanent and embedded throughout the model.Feed Your OWN Documents to a Local Large Language Model!
- RAG (Retrieval Augmented Generation) works like a student who doesn't have all information at their fingertips but knows where to look: when asked a question, the student quickly consults relevant sources from a library and combines that information with prior knowledge to provide a comprehensive answer.Feed Your OWN Documents to a Local Large Language Model!
- Retraining a large language model requires nearly as much skill and resources as training the model from scratch, including access to high-end data center GPUs like the Nvidia A100, significant amounts of RAM, and proficiency in programming frameworks like PyTorch or TensorFlow.Feed Your OWN Documents to a Local Large Language Model!
- There are three primary ways to add knowledge to large language models: retraining the model, using retrieval augmented generation (RAG), and uploading documents directly into the context window.Feed Your OWN Documents to a Local Large Language Model!
- Retraining a model requires substantial computational resources: large amounts of data, significant computing power, and considerable time investment, making it impractical for most users.Feed Your OWN Documents to a Local Large Language Model!
- Uploading documents to the context window is like giving a student a cheat sheet during an exam: the model can reference the notes to answer questions immediately, but won't internalize them permanently and will forget the information once the session ends.Feed Your OWN Documents to a Local Large Language Model!
- Retraining a model is analogous to a student returning to school to learn new material and correct understanding; it involves bringing in new training data and putting the model through additional study sessions, after which the learned knowledge becomes permanently embedded in the model's weights for all subsequent uses.Feed Your OWN Documents to a Local Large Language Model!