
Feed Your OWN Documents to a Local Large Language Model!
What this covers
This is a single-presenter walkthrough by Dave of his channel, Dave's Garage, built around a student analogy. Retraining is sending the student back to school, Retrieval Augmented Generation (RAG)) is a student who knows which library shelf to consult, and uploading to the context window is a temporary cheat sheet for one exam. He sets out the trade-offs of each, then moves to hands-on demonstrations using hosted and local tools.
The retraining section is mostly about why most people can't do it. ChatGPT gives no access to its weights, so it can't be modified directly. Open models such as Llama 3.2 expose their weights, but licensing may limit commercial or large-scale use. Dave also argues that serious retraining takes nearly as much skill and hardware as training from scratch, meaning data-center Nvidia A100-class GPUs, plenty of RAM, and fluency in PyTorch or TensorFlow.
The practical half covers the other two methods. With ChatGPT, he uploads reference material about the PDP-11 and points out the chore of re-uploading documents each session, which a custom GPT with documents built into its configuration avoids. Locally, he runs Ollama with Open WebUI to build a RAG setup that cites its source documents and links back to the passage used. He compares the "pick all documents" setting, which is slower but broader, with selecting specific documents. He notes that a model asked about something absent from its documents will answer from the nearest material it can find. One contested claim is that, with a small base model like Llama 3.2, selecting fewer documents makes the answers lean more on the supplied data and less on the model's own knowledge. A side remark observes that a Threadripper with dual RTX 6000 GPUs barely touches the CPU during generation.
Dave demonstrates three methods for adding knowledge to large language models—retraining, retrieval augmented generation (RAG), and context window uploads—and shows practical implementations of the latter two approaches using ChatGPT and local Ollama with Open WebUI.
- Retraining is impractical for most users due to hardware, software, and resource constraints
- RAG dynamically retrieves relevant documents without retraining, making it scalable for large datasets
- Context window uploads provide immediate but temporary knowledge for single sessions
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Uploading documents to a model's context window is like providing a student with a temporary cheat sheet during an exam; the model can reference and use the documents to answer questions within that session, but once the session ends, the information is not retained unless the documents are uploaded again in the next session.
“imagine you're in a one-on-one conversation with the model and you hand it some notes the model can reference these notes while talking to you but it won't internalize them in the way it would with actual retraining it's like a student who gets to peek at a cheat sheet during an exam they can look at your document and use it to answer your questions but once the exam is over or your session ends they'll forget the information”
The key strength of RAG is its ability to efficiently handle large amounts of data by pulling only what's necessary at the moment of query, making it a more scalable approach especially when dealing with complex and evolving datasets.
“the key strength of rag is its ability to efficiently handle large amounts of data pulling only what's necessary at the moment of the query this allows for a more scalable approach especially when dealing with complex evolving data sets”
The three primary methods for adding knowledge to existing large language models are: retraining the model with new data, using retrieval augmented generation (RAG), and uploading documents directly into the context window.
“there are three primary ways which you can do it retraining a model using retrieval augmented generation or rag and uploading documents directly into the context window”
There are three primary ways to add knowledge to large language models: retraining the model, using retrieval augmented generation (RAG), and uploading documents directly into the context window.
“there are three primary ways which you can do it retraining a model using retrieval augmented generation or rag and uploading documents directly into the context window”
Retraining a model is analogous to sending a student back to school: the model retains most prior knowledge but fine-tunes and adds new knowledge through repeated training cycles on updated material, making the new knowledge permanent and embedded throughout the model.
“think of the model as a bit like being a student who has already learned a lot but now you want to teach them something new or correct their understanding you go back to the basics with them bringing in new books updated lessons and putting them through another round of study sessions they don't forget that much of what they already know but they use your training to fine-tune and add to their knowledge... once they've learned it's permanent every time you use the model from then on it will have that updated knowledge embedded deep within it”
Retraining a model requires substantial computational resources: large amounts of data, significant computing power, and considerable time investment, making it impractical for most users.
“retraining a model requires a lot of resources more data lots of computing power and time it's like sending your student back to school for a while to learn and improve on what they already know”
RAG (Retrieval Augmented Generation) works like a student who doesn't have all information at their fingertips but knows where to look: when asked a question, the student quickly consults relevant sources from a library and combines that information with prior knowledge to provide a comprehensive answer.
“here instead of retraining the model it's as if your student doesn't have all the information they need at their fingertips but knows exactly where to look and when asked a question they quickly consult a library of books pulling out the most relevant sources and then give you a response that combines what they knew before and what they've just looked up”
Uploading documents to the context window is like giving a student a cheat sheet during an exam: the model can reference the notes to answer questions immediately, but won't internalize them permanently and will forget the information once the session ends.
“imagine you're in a one-on-one conversation with the model and you hand it some notes the model can reference these notes while talking to you but it won't internalize them in the way it would with actual retraining it's like a student who gets to peek at a cheat sheet during an exam they can look at your document and use it to answer your questions but once the exam is over or your session ends they'll forget the information”
Formal retraining of models like ChatGPT is impossible for most users because OpenAI provides no access to the ChatGPT model itself, so there is no way to directly modify or retrain it.
“the first reason is openness meaning that you have no access to the chat GPT model itself so there's no way you can directly your modify or retrain it”
While open-source models like Llama 3.2 have accessible model weights that can theoretically be modified, licensing and availability conditions may restrict whether users are allowed to use those weights freely for commercial purposes or large-scale projects.
“for something that's more open like llama 3.2 the models are generally designed to be open in the sense that you can access the node weights and modify them but certain conditions still apply you need to access the model weights the core data that defines the model's knowledge depending on licensing and availability you may or may not be allowed to use those weights freely for commercial purposes or large scale projects”
Retraining a large language model requires nearly as much skill and resources as training the model from scratch, including access to high-end data center GPUs like the Nvidia A100, significant amounts of RAM, and proficiency in programming frameworks like PyTorch or TensorFlow.
“retraining a model takes almost as much in terms of skill and resources as training that model did in the first place... fine tuning an LLM like llama 3.1 even on a smaller data set still requires some serious Hardware... you're looking at leasing or buying data center GPUs like the Nvidia A100... requires significant programming even if it is mostly just python but you wind up needing to write code using pytorch or tensor flow”
ChatGPT itself cannot be directly retrained or modified because users have no access to the model weights or architecture; in contrast, more open models like Llama 3.2 provide access to model weights but come with licensing conditions that may restrict commercial use or large-scale projects.
“you have no access to the chat GPT model itself so there's no way you can directly modify or retrain it with something that's more open like llama 3.2 the models are generally designed to be open in the sense that you can access the node weights and modify them but certain conditions still apply you need to access the model weights the core data that defines the model's knowledge depending on licensing and availability you may or may not be allowed to use those weights freely for commercial purposes or large scale projects”
Context window document uploads are a static approach where all provided documents are directly inserted into the model's input, forcing the model to work with everything regardless of relevance to the specific question.
“uploading documents into the context window is a more static approach when documents are uploaded their content is directly inserted to the model's input window giving the model access to that information for generating responses it's essentially like you had just typed the documents into the query window while it's a straightforward way of providing additional information to the model it can be inefficient for large or numerous documents the model has to work with everything that's provided in the input regardless of whether all of it's relevant to the specific question being asked or not”
Retrieval augmented generation (RAG) is a system that dynamically retrieves relevant information from a large repository of documents or data in response to user queries, combining retrieved information with the model's internal knowledge.
“our step then is with retrieval augmented generation or rag which is a system designed to dynamically retrieve information from an external knowledge base or database so when the user asks a question the system searches through a large repository of documents or data and pulls out the most relevant pieces this information is then combined with the model's internal knowledge to produce a more accurate and contextually informed response”
With Ollama and Open WebUI running locally, users can upload document files that the model uses as context to answer questions, and the system references the source document, allowing users to click on a link to see which part of the document provided the answer.
“I've clicked upload file by clicking on plus and I've selected the PDP 1134 user manual... it cranks out an answer really quickly and it is in fact the correct answer and at the bottom of our answer you will see that it is referenced the context document even better we can click on that link and bring it up and actually see the part of the document that is making reference to”
OpenWebUI's 'pick all documents' setting for RAG models is slower than selecting specific documents but provides more comprehensive access to the knowledge base.
“the easiest way is to pick all documents and it will be able to reference everything in your data/ doents folder however that will be slower so in this case I'm just going to give it at one or two documents that applied directly to my actual machine”
RAG's key strength is its ability to efficiently handle large amounts of data by pulling only what's necessary at the moment of query, making it particularly useful when dealing with complex, evolving datasets where accuracy and specificity are crucial.
“the key strength of rag is its ability to efficiently handle large amounts of data pulling only what's necessary at the moment of the query this allows for a more scalable approach especially when dealing with complex evolving data sets the retrieval process ensures that only the most pertinent information is used which makes rag particularly useful for situations where accuracy and specificity are crucial”
When a custom RAG model doesn't have specific information about a requested topic in its documents, it will provide answers based on the closest available information it can find, which may not be the most accurate for highly specific queries.
“because I probably didn't give it the information that it needed on the memory board the closest it could find was the RQ dx3 controller so it's going to answer its questions with pretty much that as its main knowledge about the pdp1”
While the PDP-11 computer documentation hasn't changed much over time, maintaining accuracy on more current information requires regularly uploading newer revised documents when using context windows, which adds a layer of manual upkeep that RAG eliminates.
“now the pdp1 isn't changing much these days but to maintain accuracy on more current things the user would need to regularly upload newer revised documents which adds a layer of manual upkeep”
While uploading documents to ChatGPT works well for providing context, it becomes cumbersome when you need to upload the same documents repeatedly across multiple sessions or when you have multiple documents to upload each time.
“you could do this each and every time that you wanted chat GPT to have access to this additional knowledge but it gets a little cumbersome which is especially true if you have multiple documents to do each time”
Custom GPTs created within ChatGPT can have documents baked into their configuration, making those documents permanently available to the custom GPT without requiring re-upload for each session.
“but it gets a little cumbersome which is especially true if you have multiple documents to do each time let's look at a better approach creating our own custom GP PT with our context documents fully baked in”
If you're using RAG with documents pulled directly from a frequently updated database, you're always getting the most recent snapshots of that database; conversely, if documents are updated less frequently, you at least get more current documents than with static context window uploads.
“especially if you're pulling the data from a database directly then you're getting the very latest snapshots of the database if you're not doing that even with documents if they're updated frequently at least you're getting the most frequent documents”
A Threadripper processor paired with dual RTX 6000 GPUs uses almost no CPU utilization during large language model generation, suggesting that GPU acceleration is the dominant factor in LLM inference performance and CPU is not a bottleneck.
“I would mention that it's a thread Ripper but it's using almost no CPU during that generation”
When using a smaller base model like Llama 3.2, selecting fewer documents instead of all available documents makes the RAG model more dependent on the provided data and less on its own internal knowledge.
“I'm actually going to go with the smaller but efficient L 3.2 because I want it to be more dependent on my data and less on its own data that it brings to the table”
Fine-tuning an LLM like Llama 3.1 requires serious hardware—specifically data center GPUs like Nvidia A100s—and depending on model size may require an enormous amount of RAM to execute the fine-tuning tasks.
“fine tuning and llm like llama 3.1 even on a smaller data set still requires some serious Hardware that means you're looking at leasing or buying data center gpus like the Nvidia a100 and depending on the model size it could require an enormous amount of ram to work on it”
The custom RAG model in Ollama continuously hits the RAG data as visible in the left-hand window during queries and successfully synthesizes and recites facts learned from the RAG data to provide approximately correct answers.
“so as we can see it is in fact hitting all the rag data which you can also see scrolling by in the leftand window as we do a query and again it's able to synthesize and recite the facts that it learned from looking at the rag data and give me little look at be approximately correct answers”
Custom GPTs created by uploading documents can answer very specific technical questions that require detailed knowledge from those documents, such as identifying the correct wire wrap part number for PDP-11 boards.
“8067 that's the correct part number and that is how you wire wrap the boards that would have been really handy when I did this about two weeks ago I mean why read the manual when you can just ask chat gbt”
The custom PDP-11 expert GPT correctly identified the part number 8067 for wire-wrapping boards, information that would have been valuable to have before manually completing the task two weeks prior.
“8067 that's the correct part number and that is how you wire wrap the boards that would have been really handy when I did this about two weeks ago I mean why read the manual when you can just ask chat gbt”
A dual Nvidia RTX 6000 workstation can generate 324 tokens per second with a 1 billion parameter model, sustaining over 300 tokens per second consistently.
“345 tokens per second... when it is done how many tokens per second do we get 324 so it's able to easily sustain over 300 tokens per second generating on the Dual RTX 6000 machine”
The context window itself can be quite constraining; for example, Llama 3 has a context window of approximately 4,000 tokens, which is insufficient for incorporating a set of half a dozen PDP-11 reference documents simultaneously.
“the size of the context window itself can also be quite constraining think of numbers like 4,000 tokens for a model like llama 3 if the documents you were trying to provide exceed the size of that context window clearly it's not going to work as well for example I have a set of a half dozen pdp1 reference documents that I'd like to incorporate into my searches and the context window is generally not large enough for that”
A dual Nvidia RTX 6000 system can generate over 300 tokens per second when running a 1 billion parameter model like Llama 3.2, sustaining approximately 324 tokens per second on longer text generation tasks.
“345 tokens per second... 324 so it's able to easily sustain over 300 tokens per second generating on the Dual RTX 6000 machine”
To set up RAG in Ollama with Open WebUI, documents must be copied into the documents folder, then accessed through admin settings → documents → scan to parse and register them with the system before they can be used by custom models.
“to make our documents available to models within the olama open webui system we need to go to the admin settings so we'll go to the admin panel we go to settings and we'll go to documents... in order to put the documents there but it won't automatically find them until you come in here and scan”
Local Ollama/OpenWebUI systems provide source document citations showing exactly which part of a referenced document was used to generate an answer, enabling users to verify information sources.
“and at the bottom of our answer you will see that it is referenced the context document even better we can click on that link and bring it up and actually see the part of the document that is making reference to I find that incredibly handy when you're looking up a citation of where it got the information that you're asking it about”
When you run Open WebUI from the backend script (rather than in a Docker container), the documents folder is more easily accessible in the local file system; however, if running in Docker, you must use the docker copy command to copy files into the container's version of the documents folder.
“I'm going to run the open web UI locally this time rather than in the docker container simply by enlisting in it from GitHub and then launching the start script from the folder called backend I'm going to do it that way because it makes the documents folder easily available in my local file system if you're running it in a Docker container it'll still work but you'll need to use the docker copy command to copy your files into the containers version of the data document folder”
When scanning documents in Ollama, the system parses PDP-11 manuals and technical documentation, extracting information about components like cylinders and hard drives to prepare them for RAG retrieval.
“now you can see it's obviously pulling the deck manuals and so on I can see stuff about cylinders and hard drives and so on so it's now parsing this data processing it and it is now ready”
ChatGPT without the additional uploaded PDP-11 manual has general knowledge about PDP-11 boot procedures but lacks the specific technical details from the owner's manual about the proper sequence for using the boot switch and ROM bootstrap procedure.
“that's sort of a general approach for starting a pdp11 um it does not take into account the owner's manual of the 1134 which tells you how to actually use the boot switch to boot to a ROM location which will bootstrap the machine so it doesn't seem to have that context”
When running OpenWebUI from a local GitHub repository rather than in a Docker container, documents folder access is simpler and more direct via the local file system.
“one note about the configuration I'm going to run the open web UI locally this time rather than in the docker container simply by enlisting in it from GitHub and then launching the start script from the folder called backend I'm going to do it that way because it makes the documents folder easily available in my local file system if you're running it in a Docker container it'll still work but you'll need to use the docker copy command to copy your files into the containers version of the data document folder”
Once documents are embedded in a custom GPT and the model is queried, it will use those documents to find correct answers to specific technical questions that require consulting the source material.
“I'll ask it a very specific question that will require it to do some research... it's searching its knowledge base... 8067 that's the correct part number and that is how you wire wrap the boards”
Context windows have size limitations (e.g., Llama 3 context windows are typically around 4,000 tokens), which means that if documents you want to provide exceed the context window size, the model won't be able to access all of them simultaneously for a single query.
“the size of the context window itself can also be quite constraining think of numbers like 4,000 tokens for a model like llama 3 if the documents you were trying to provide exceed the size of that context window clearly it's not going to work as well”
Custom GPTs created through ChatGPT's 'Explore GPTs' interface can accept OCR'd PDF documents as part of their knowledge base, and these custom GPTs can then be shared via a link with access controls set to 'anyone with the link'.
“I've got a folder called PDP where I have a bunch of documents that are already OCR and their PDFs and it can handle that format so we'll just upload them directly... I'll say anybody with the link can access it”
Retraining a model is analogous to a student returning to school to learn new material and correct understanding; it involves bringing in new training data and putting the model through additional study sessions, after which the learned knowledge becomes permanently embedded in the model's weights for all subsequent uses.
“think of the model as a bit like being a student who has already learned a lot but now you want to teach them something new or correct their understanding you go back to the basics with them bringing in new books updated lessons and putting them through another round of study sessions they don't forget that much of what they already know but they use your training to fine-tune and add to their knowledge”
For static knowledge bases like PDP-11 documentation (which doesn't change much these days), maintaining accuracy in context window uploads requires regularly uploading newer or revised documents, which adds a layer of manual upkeep.
“now the pdp1 isn't changing much these days but to maintain accuracy on more current things the user would need to regularly upload newer revised documents which adds a layer of manual upkeep”
Users can optimize RAG performance by selecting only the specific documents relevant to a model's purpose rather than including all available documents, because including everything would be slower and less focused.
“the easiest way is to pick all documents and it will be able to reference everything in your data/ doents folder however that will be slower so in this case I'm just going to give it at one or two documents that applied directly to my actual machine”
When selecting a base model for Open WebUI's custom model, Dave chose the smaller but efficient Llama 3.2 model because he wanted it to be more dependent on his data and less on the model's own internal knowledge.
“I'm actually going to go with the smaller but efficient L 3.2 because I want it to be more dependent on my data and less on its own data that it brings to the table”
Creating a custom GPT with uploaded documents embedded avoids the need to re-upload the same documents each time you want ChatGPT to have access to that knowledge, making the workflow less cumbersome for regular use with the same documents.
“you could do this each and every time that you wanted chat GPT to have access to this additional knowledge but it gets a little cumbersome which is especially true if you have multiple documents to do each time”
Chat GPT did not accurately answer a specific technical question about PDP-11 boot procedures without the manual provided, because it lacked the precise context from the owner's manual despite having general knowledge about PDP-11 systems.
“that's sort of a general approach for starting a pdp11 um it does not take into account the owner's manual of the 1134 which tells you how to actually use the boot switch to boot to a ROM location which will bootstrap the machine so it doesn't seem to have that context”
During inference on the dual RTX 6000 machine, the CPU usage is nearly zero, indicating that GPU memory bandwidth, not CPU, is the primary bottleneck.
“I would mention that it's a thread Ripper but it's using almost no CPU during that generation”
Custom GPTs can be shared with others by setting permissions to 'anybody with the link can access it', allowing users to distribute specialized knowledge-augmented models.
“I'll say anybody with the link can access it pdp1 expert is a name and we'll save it”
After OpenWebUI documents are scanned, parsing processes the document data (visible through the admin interface logs), and documents are then ready for use in model queries.
“we monitor the other window to see its progress now you can see it's obviously pulling the deck manuals and so on I can see stuff about cylinders and hard drives and so on so it's now parsing this data processing it and it is now ready”
Dave previously showed how to run Ollama and Open WebUI locally on a machine, and he demonstrates uploading the PDP-11/34 user manual to a local Ollama instance running under Open WebUI.
“in a previous episode I showed you how to run olama and open webui locally on your machine so let's take a look at the steps needed to provide context documents to AMA so here we are in the local machine and running o Lama and open web UI I've clicked upload file by clicking on plus and I've selected the PDP 1134 user manual”