
When Vectors Break Down: Graph-Based RAG for Dense Enterprise Knowledge - Sam Julien, Writer
What this covers
Sam Julien of Writer walks through how the company built a graph-based retrieval system after customers kept running into the limits of standard RAG. The talk is a short case study that follows the order of Writer's own work. It begins with why enterprise data is hard, moves to the research Writer drew on, and ends with what the team would tell others to do now. Its framing is pragmatic: start from the customer problem, then build components the team has the expertise to run.
The diagnosis comes first. Enterprise corpora are described as dense, specialized, very large, and highly clustered, with little vocabulary diversity between documents. Julien argues that vector search struggles here because similarity search cannot separate documents that share the same narrow terminology. Mobile phone specs are one example, since they all mention megapixels, cameras, and battery life. Naive chunking is the other, and he illustrates it with an Apple timeline in which the Lisa and Macintosh dates get confused. Some of these failure claims are flagged as contested.
The remedy draws on the original RAG paper's two-component design of retriever and generator. Julien notes that the paper never proposed the prompt-context approach that later became standard. He presents fusion-decoder as the overlooked alternative. It encodes passages independently, so cost scales linearly, then aggregates evidence jointly in the decoder. Writer combined this with context-aware splitting during text-to-graph conversion, so that graph-based RAG keeps relationships between pieces of content intact.
The claimed benefits are better handling of answers split across pages, multi-hop reasoning across documents, and visible reasoning. Snippets, subqueries, and sources can be surfaced in both the API and the product. Julien also says small, fast models are now easy to find, so teams starting today may not need to build specialized ones as Writer did. He closes on the value of the iterative learning process itself.
Graph-based RAG systems outperform vector-only search for enterprise knowledge retrieval because they preserve document relationships and context, requiring a pragmatic approach combining specialized models, search infrastructure, and research-backed techniques like fusion-decoder rather than adhering to any single technology stack.
- Vector search alone fails on dense, homogeneous enterprise data because chunking and nearest-neighbor matching lose relational context, confusing similar items (e.g., Apple products by year)
- Graph structures with fusion-decoder improve retrieval accuracy and efficiency by processing passages independently then jointly aggregating evidence while preserving relationships
- Writer's approach prioritized customer problems over technology hype, building custom components (specialized models, JSON-based search) aligned with team expertise rather than forcing graph database solutions
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The original RAG paper does not discuss using prompt context and questions, which has become the de facto approach in modern RAG implementations, suggesting an alternate design path was overlooked.
“they went back to the original rag paper and if you go back to the original rag paper it doesn't actually ever talk about using prompt context and questions which is super interesting right that's sort of like the deacto way of doing rag now but the the original rag paper actually proposed this whole like two uh component architecture with a retriever and a generator with pre- pre-trained sequence to sequence model never actually talks about prompt and context in questions”
Fusion-Decoder is an alternate RAG architecture that processes passages independently in the encoder for linear scaling (rather than quadratic), then jointly in the decoder for better evidence aggregation, achieving state-of-the-art performance with significant efficiency gains.
“fusion and decoder is this technique that kind of builds upon the original proposal of the original rag paper where it processes the passages independently in the encoder to get linear scaling instead of quadratic scaling but then jointly in the decoder for better evidence aggregation. So big efficiency breakthrough and lots of state-of-the-art performance.”
Enterprise data is inherently dense, specialized, uses very specific language, and is highly clustered with little vocabulary diversity across documents.
“enterprise data is really dense, specialized and massive. So we're often dealing with terabytes of data and it uses very specific language and it's often very clustered together. There's not a lot of diversity in the language used in these documents”
Vector retrieval fails with concentrated/homogeneous enterprise data because when multiple documents use the same specialized vocabulary (e.g., phone specs like megapixels, battery life, cameras), similarity search cannot disambiguate between different entities and context.
“the other big problem that we ran into with vector retrieval was that it was failing with really concentrated data. So if you think about a lot of large enterprises, it's not like they're dealing with documents where like some of them are talking about animals and some of them are talking about fruit, right? Like so if you have a mobile phone company for example and they have thousands and thousands of documents that all use megapixels and cameras and battery life and things like that and you ask the rag system and the LLM to compare two different phone models, it's going to really struggle with that because it's going to find all these answers and have no idea how to make sense of them”
Vector search alone is insufficient for sophisticated retrieval at enterprise scale and the industry is recognizing the need for multiple retrieval strategies beyond simple vector similarity.
“the industry is starting to recognize that vector search alone is just insufficient for sophisticated retrieval and that we're going to need multiple strategies beyond simple vector similarity”
Vector retrieval with naive chunking and nearest-neighbor search produces inaccurate answers because relevant information can be split across chunks in ways that confuse similarity metrics, such as dating different Apple products incorrectly.
“with vector retrieval chunking and nearest neighbors can give inaccurate answers. Uh so if you look at this example of kind of this text about the founding of Apple and the timeline, it's very easy for us as humans to look at these text chunks and pick out the fact that the Macintosh was created in 1984. But when you chunk this text naively and you just give it to a nearest neighbor search, uh it can get confused and it thinks that it was actually in 1983 instead of 1984 because it's in the same chunk as the introduction of the Lisa”
Fusion-Decoder is an alternate RAG architecture (based on the original RAG paper's two-component design) that processes passages independently in the encoder for linear scaling and jointly in the decoder for better evidence aggregation, achieving state-of-the-art performance with improved efficiency.
“fusion and decoder is this technique that kind of builds upon the original proposal of the original rag paper where it processes the passages independently in the encoder to get linear scaling instead of quadratic scaling but then jointly in the decoder for better evidence aggregation. So big efficiency breakthrough and lots of state-of-the-art performance”
Graph-based RAG handles complex data formats where vector retrieval struggles, including answers split across multiple pages or cases where similar terminology doesn't precisely match the user's query, because the graph structure and fusion-decoder provide additional context and relationship information.
“it can handle complex data formats where vector retrieval struggles where an answer might be split into multiple pages or maybe there's a similar term that doesn't quite match what the user is looking for. But we because we have that graph structure and and the and fusion and decoder with the additional context and relationships, we're able to uh formulate these correct answers.”
Because Writer's graph structures preserve relationships and context, they can expose the RAG system's reasoning process by showing snippets, subqueries, and sources in both the API and product interface, improving transparency and explainability.
“for one because like most graph structures we can actually expose the thought process because we have that relationships and the additional context where you can show the snippets and the subqueries and the sources for how the rag system is actually getting the answers and we can expose this in the API to developers as well as in the product”
Graph-based RAG enables multi-hop reasoning across multiple documents and topics without struggles, allowing questions that require reasoning across disparate information sources.
“we can expose this in the API to developers as well as in the product and then we're also to have able to have knowledge graphic sell multihop questions where we can um reason across multiple documents and multiple topics without any struggles.”
Enterprise data is typically dense, specialized, and massive (terabytes), uses very specific language, and is highly clustered with low linguistic diversity, which creates unique retrieval challenges.
“the problem that we are trying to solve kind of constantly as most of us are here is that enterprise data is really dense, specialized and massive. So we're often dealing with terabytes of data and it uses very specific language and it's often very clustered together. There's not a lot of diversity in the language used in these documents.”
Vector search is insufficient for sophisticated retrieval at scale in enterprise RAG systems; the industry is recognizing that multiple retrieval strategies beyond simple vector similarity are necessary.
“the market is starting to catch up that vector search is just not enough for rag at scale”
Graph-based RAG enables exposing the retrieval system's reasoning process by showing retrieved snippets, subqueries, and sources, which can be surfaced in both API and product interfaces for developer transparency.
“because like most graph structures we can actually expose the thought process because we have that relationships and the additional context where you can show the snippets and the subqueries and the sources for how the rag system is actually getting the answers and we can expose this in the API to developers as well as in the product”
Vector retrieval with naive chunking and nearest neighbor search produces inaccurate answers on dense text because semantic boundaries of chunks don't align with logical units—the example shows chunking the Apple timeline causes confusion between Lisa (1983) and Macintosh (1984) dates.
“with vector retrieval chunking and nearest neighbors can give inaccurate answers. Uh so if you look at this example of kind of this text about the founding of Apple and the timeline, it's very easy for us as humans to look at these text chunks and pick out the fact that the Macintosh was created in 1984. But when you chunk this text naively and you just give it to a nearest neighbor search, uh it can get confused and it thinks that it was actually in 1983 instead of 1984 because it's in the same chunk as the introduction of the Lisa.”
Vector retrieval fails catastrophically on concentrated enterprise data where documents use narrow, overlapping terminology (e.g., mobile phone specs all mention megapixels, cameras, battery life), because the system retrieves many superficially similar results and cannot distinguish between them.
“the other big problem that we ran into with vector retrieval was that it was failing with really concentrated data. So if you think about a lot of large enterprises, it's not like they're dealing with documents where like some of them are talking about animals and some of them are talking about fruit, right? Like so if you have a mobile phone company for example and they have thousands and thousands of documents that all use megapixels and cameras and battery life and things like that and you ask the rag system and the LLM to compare two different phone models, it's going to really struggle with that because it's going to find all these answers and have no idea how to make sense of them.”
Context-aware splitting and chunking preserves semantic relationships better than naive chunking, improving the reliability of graph-based RAG systems.
“we did some uh better contextaware splitting and chunking to uh preserve the context and the semantic relationships and this really helped uh preserve the reliability.”
The original RAG paper did not discuss using prompt context and questions—this became the de facto approach later—but instead proposed a two-component architecture with a retriever and generator using pre-trained sequence-to-sequence models.
“they went back to the original rag paper and if you go back to the original rag paper it doesn't actually ever talk about using prompt context and questions which is super interesting right that's sort of like the deacto way of doing rag now but the the original rag paper actually proposed this whole like two uh component architecture with a retriever and a generator with pre- pre-trained sequence to sequence model never actually talks about prompt and context in questions”
Graph-based RAG, which queries a graph database to retrieve relevant documents using keys and generate answers, preserves semantic relationships and provides more context to the model, significantly improving accuracy compared to vector-only retrieval.
“that's what took took us to graph-based rag where instead we would query a graph database and get back the relevant documents using keys uh and generate an answer. Especially powerful if you combine that with like full text and similarity search and things like that. Um and so this really helped us with our accuracy because we were able to preserve the relationships with the text and provide more context to to the model.”
Graph-based RAG enables multi-hop reasoning across multiple documents and topics without the disambiguation struggles of vector search, allowing the system to reason across semantically similar but distinct entities.
“we can actually expose the thought process... we're also to have able to have knowledge graphic sell multihop questions where we can um reason across multiple documents and multiple topics without any struggles”
Graph-based RAG retrieves relevant documents using keys and relationships within a graph database, which preserves relational context and provides more information to the language model compared to vector similarity alone.
“that's what took took us to graph-based rag where instead we would query a graph database and get back the relevant documents using keys uh and generate an answer. Especially powerful if you combine that with like full text and similarity search and things like that. Um and so this really helped us with our accuracy because we were able to preserve the relationships with the text and provide more context to to the model”
Graph-based RAG with fusion-decoder handles complex data formats that vector retrieval struggles with, such as answers split across multiple pages or similar terms that don't exactly match user queries.
“it can handle complex data formats where vector retrieval struggles where an answer might be split into multiple pages or maybe there's a similar term that doesn't quite match what the user is looking for. But we because we have that graph structure and and the and fusion and decoder with the additional context and relationships, we're able to uh formulate these correct answers”
Writer performed context-aware splitting and chunking during the text-to-graph conversion process to preserve semantic relationships and reliability of document structure.
“and we did some uh better contextaware splitting and chunking to uh preserve the context and the semantic relationships and this really helped uh preserve the reliability”
Small, fast language models suitable for fine-tuning on CPU/small GPU inference are now readily available in the market, so teams building graph RAG systems today might not need to build specialized models like Writer did.
“Now if you were to do this now there's probably enough fast small models out there that you could fine-tune something like that. You wouldn't have to build it yourself. But at the time we didn't really have any options like that.”
The learning process of building a retrieval system—iterating through failures and discovering constraints—is almost as valuable as the final technical result, because it teaches teams what tradeoffs exist and where flexibility is needed.
“and also what how you get there and what you learn along the way is actually often very valuable as you're building out your retrieval system uh almost just as valuable as the end result itself.”
The third meta-lesson is letting research challenge assumptions: rather than staying committed to a failing solution, the team stepped back to revisit foundational research (the original RAG paper and fusion-decoder papers) to discover alternative approaches they hadn't initially pursued.
“And so I think this is kind of the final meta point of letting research challenge your assumptions. So rather than staying focused on the solution, you know, step back, look at the research and figure out what you can do to solve the challenges for your customers. So they went back to the original rag paper”
Writer's success in building effective graph-based RAG stems from three meta-principles: focusing on customer needs rather than technology hype, maintaining flexibility based on team expertise, and letting research challenge technical assumptions.
“And so for our team, it was focusing on the customer needs instead of what was hyped, staying flexible based on the expertise of the team and letting research challenge their assumptions”
The process of how to get to graph-based RAG and lessons learned along the way are often as valuable as the end result itself.
“what you learn along the way is actually often very valuable as you're building out your retrieval system uh almost just as valuable as the end result itself”
Let research challenge your assumptions by returning to foundational papers and alternative design paths, rather than staying locked into current practice.
“this is kind of the final meta point of letting research challenge your assumptions. So rather than staying focused on the solution, you know, step back, look at the research and figure out what you can do to solve the challenges for your customers and our specific needs”
Writer's implementation of knowledge graphs with fusion-decoder significantly reduced hallucination rates in their RAG system.
“it really helped our hallucination rate. really drove it down”
Rather than forcing graph database solutions when facing technology constraints, Writer's team returned to their core expertise and built a specialized fine-tuned model to map data into graph structures (nodes and edges) that could scale on CPUs or smaller GPUs.
“So first when it came to converting the data into the graph structure team went back to their expertise and they say what do we know how to do? We know how to build models. So let's build a specialized model that can scale and run on CPUs or smaller GPUs which I think is a really clever solution... the team built it themselves and fine-tuned a model that was trained to map this data into graph structures of nodes and edges”
Writer replaced graph database storage with JSON-based data stored in a Lucene search engine, which handled large enterprise datasets without performance degradation while leveraging the team's existing search infrastructure expertise.
“what we did was instead we stored the data points as JSON in a lucine based search engine. So we take the graph structure we converted into JSON and we put it in the search engine and this allowed us to easily handle the large amounts of data without any performance or speed degradation uh at scale while still being something that the team was really good at”
Knowledge graphs combined with fusion-decoder improves upon the fusion-decoder paper by using knowledge graphs to understand relationships between retrieved passages, which helps with efficiency bottlenecks and cost reduction.
“the next big breakthrough was knowledge graph with fusion and decoder. So you can use knowledge graphs with fusion and decoder uh as a technique and this sort of improves upon the fusion and decoder paper by using knowledge graphs to understand the relationships between the retrieved passages and so it helps with this efficiency bottleneck and improves uh the the process”
Writer's research team is driven by customer insights rather than theoretical research in isolation, which is a key factor in the team's effectiveness.
“They're very focused on solving practical problems for our customers. Uh they're not just sort of like working in isolation uh working on theoretical things. They're actually driven by customer insights. And that's uh really what I would consider like sort of the first meta lesson”
The first meta-lesson of Writer's success is focusing on solving customer problems rather than implementing specific hyped technologies or solutions, which forces pragmatic decision-making and flexibility.
“I think what's really cool about the way our research team works is that they're very focused on solving practical problems for our customers. Uh they're not just sort of like working in isolation uh working on theoretical things. They're actually driven by customer insights. And that's uh really what I would consider like sort of the first meta lesson of what why I think this is working so well for writer. Right now we're really focused on solving the customer problems rather than implementing specific solutions.”
The second meta-lesson is staying flexible based on the team's existing expertise: when facing technical obstacles, Writer chose solutions that aligned with what the team already knew well (model building, search engines) rather than forcing adoption of unfamiliar technologies.
“they decided to stay flexible based on their expertise. So they were running into these problems that I think were not necessarily fundamental to the technology itself, but more like okay, how can we solve the problems for our customers using the expertise that we have on the team?”
Joe Christian Bergam published research on 'the rise and fall of the vector database infrastructure category' and discussed in a Latent Space interview how vector databases experienced a gold rush after ChatGPT's launch but are now recognized as insufficient alone.
“You may have seen this really interesting article by Joe Christian Bergam who is around here somewhere on the rise and fall of the vector database infrastructure category and his subsequent interview on latent space where he talked about how vector databases have experienced this gold rush after ChatP's launch. uh but that the industry is starting to recognize that vector search alone is just insufficient for sophisticated retrieval”
When Writer first adopted graph databases for RAG (a couple years ago), they encountered four specific technical challenges: difficulty converting unstructured data into graph structures at scale, cost issues and expertise limits as graphs scaled, Cypher query language struggling with advanced similarity matching, and LLMs performing better with text-based queries than complex graph structures.
“we were running into these issues at the time. a couple of years ago. And so there were four things that we ran into. First, that converting the data into the structured graph was getting really challenging and costly at scale. Uh as the as the graph database scaled, we were hitting the limits of our team's expertise as well as hitting some cost issues. And then we were running into some problems where cipher was struggling with the advanced similarity matching that we needed. And we were noticing that LLMs were doing better with textbased queries rather than complex graph structures.”
Instead of using traditional graph databases for scaling, Writer built a specialized model (running on CPUs or smaller GPUs) to convert unstructured data into graph structures of nodes and edges, combined with context-aware chunking that preserves semantic relationships.
“So first when it came to converting the data into the graph structure team went back to their expertise and they say what do we know how to do? We know how to build models. So let's build a specialized model that can scale and run on CPUs or smaller GPUs which I think is a really clever solution... the team built it themselves and fine-tuned a model that was trained to map this data into graph structures of nodes and edges and we did some uh better contextaware splitting and chunking to uh preserve the context and the semantic relationships”
Writer stores graph-structured data as JSON in a Lucene-based search engine rather than traditional graph databases, enabling efficient handling of large data volumes without performance or speed degradation at scale while leveraging the team's existing expertise.
“what we did was instead we stored the data points as JSON in a lucine based search engine. So we take the graph structure we converted into JSON and we put it in the search engine and this allowed us to easily handle the large amounts of data without any performance or speed degradation uh at scale while still being something that the team was really good at.”
Knowledge graphs combined with fusion-decoder improves upon fusion-decoder alone by using knowledge graphs to understand relationships between retrieved passages, reducing efficiency bottlenecks and lowering costs.
“the next big breakthrough was knowledge graph with fusion and decoder. So you can use knowledge graphs with fusion and decoder uh as a technique and this sort of improves upon the fusion and decoder paper by using knowledge graphs to understand the relationships between the retrieved passages and so it helps with this efficiency bottleneck and improves uh the the process.”
There are multiple valid implementation approaches to gain the benefits of knowledge graphs in RAG: through graph databases, creative approaches with PostGIS, or search engines, and the specific implementation matters less than leveraging the relationships enabled by knowledge graph structure.
“there are many different ways that you can get the benefits of knowledge graphs in rag and also what how you get there and what you learn along the way is actually often very valuable as you're building out your retrieval system uh almost just as valuable as the end result itself. So, I'm going to weave together these two stories of our journey to graph-based rag... there are many ways that you can get the benefits of knowledge graphs in rag. That could be through a graph database. It could be through doing something creative with posters. It could be through a search engine.”
Many of Writer's clients are Fortune 500 or Global 2000 companies operating in highly regulated industries like healthcare and finance where accuracy and low hallucination rates are critical requirements.
“we've been working with these Fortune 500 and global 2000 companies at scale. Most of them or many of them are in highly regulated industries like healthcare and finance where accuracy and low hallucinations are super important”
Writer identified and discussed the benefits of knowledge graphs and shortcomings of vector databases for enterprise RAG at scale as early as November 2023, positioning the company ahead of industry recognition of this problem.
“if you look at this article from November 2023, which in AI time is like prehistoric times, um we actually talk about the benefits of knowledge graphs and the shortcomings of vector databases and simple similarity search for enterprise rag at scale”
Multiple implementation approaches can deliver the benefits of knowledge graphs in RAG systems, including graph databases, creative use of PostgreSQL, or search engines, so the specific technology choice is less important than understanding and applying relational structure.
“there are many ways that you can get the benefits of knowledge graphs in rag. That could be through a graph database. It could be through doing something creative with posters. It could be through a search engine. Uh but you can you take advantage of the relationships that you can build with knowledge graphs uh in your rag system”
Writer has hired a specialized model and combined it with context-aware text splitting to convert raw data into graph structures with improved reliability.
“the team built it themselves and fine-tuned a model that was trained to map this data into graph structures of nodes and edges and we did some uh better contextaware splitting and chunking to uh preserve the context and the semantic relationships and this really helped uh preserve the reliability”
Writer was an early adopter of graph-based RAG at a time (a couple of years before the talk) when few practitioners were using this approach, which gave the company a competitive advantage.
“And so that's what took took us to graph-based rag where... And this was really interesting because at the time there actually weren't that many people doing graph-based rag o last over the last couple of years. And that's why I think the focus of the team on really trying to solve the problem of the customer rather than chase whatever was uh being hyped up at the time was really important”
Graph database challenges encountered by Writer were not fundamental to the technology itself but contextual to their situation, and different approaches might face different constraints.
“Now again, if you were to do this now, you might not run into those problems, but this is what we ran into historically. And so I think the way that the team approached this is also very interesting where they decided to stay flexible based on their expertise”
Writer's research team encountered four significant challenges with graph databases a couple of years ago: converting data into structured graphs was costly and complex at scale, Cypher query language struggled with advanced similarity matching, LLMs performed better with text-based queries than complex graph structures, and team expertise limitations.
“there were four things that we ran into. First, that converting the data into the structured graph was getting really challenging and costly at scale. Uh as the as the graph database scaled, we were hitting the limits of our team's expertise as well as hitting some cost issues. And then we were running into some problems where cipher was struggling with the advanced similarity matching that we needed. And we were noticing that LLMs were doing better with textbased queries rather than complex graph structures”
Writer built its own implementation of fusion-decoder because the company builds its own language models, allowing custom optimization for RAG tasks.
“the team took all this research and came to came together to build their own um implementation of fusion indicer since we actually build our own models uh to make that kind of the final piece of the puzzle”
Writer benchmarked their retrieval system (using knowledge graphs and fusion-decoder) against seven different vector search systems on Amazon's RoboQA dataset and found their system achieved the best accuracy and fastest response time.
“We used Amazon's robust QA data set and compared our retrieval system with knowledge graph and fusion decoder and everything uh with our with seven different vector search uh systems and we found that we had the the best accuracy and the fastest response time.”
Writer benchmarked their knowledge graph plus fusion-decoder retrieval system against seven different vector search systems using Amazon's RobustQA dataset and found superior accuracy and response time.
“We used Amazon's robust QA data set and compared our retrieval system with knowledge graph and fusion decoder and everything uh with our with seven different vector search uh systems and we found that we had the the best accuracy and the fastest response time”
Facebook/Meta has made a fusion-decoder library available that practitioners can use to experiment with and learn the technique.
“I know there's a super abstract. So if you go to Facebook, they actually have a a fusion and decoder uh library that you can play around with and actually do the steps of fusion and decoder”
Writer's initial approach used algorithmic keyword search to query knowledge bases and pass results to the LLM, but this quickly failed because it was only effective for basic keyword searches and insufficient for advanced similarity search.
“Like most we kind of started out with a regular search of quering a knowledge base using an algorithm and passing that to the LLM. But that quickly sort of like ran out because of you know it was good for basic keyword searches but not really great for that advanced similarity search that we needed.”
Writer's next approach used vector embeddings with chunking and stored results in a database for similarity search, then passed results to the LLM, but this caused two major problems: inaccurate answers from chunking artifacts and failure on concentrated data.
“So then again like most we went to vector embeddings and did chunking and embeddings and put it in a database and then similarity search uh and passing it to the LLM for the end user to query. But we ran into two major problems with this.”
Facebook has a Fusion-Decoder library that allows practitioners to experiment with the fusion-decoder technique steps.
“I know there's a super abstract. So if you go to Facebook, they actually have a a fusion and decoder uh library that you can play around with and actually do the steps of fusion and decoder.”
Writer initially started with basic keyword search (regular search/querying of knowledge bases) before moving to vector embeddings.
“So like most we kind of started out with a regular search of quering a knowledge base using an algorithm and passing that to the LLM. But that quickly sort of like ran out because of you know it was good for basic keyword searches but not really great for that advanced similarity search that we needed”
Vector embeddings with chunking and nearest-neighbor search was Writer's second major retrieval approach before moving to graph-based systems.
“So then again like most we went to vector embeddings and did chunking and embeddings and put it in a database and then similarity search uh and passing it to the LLM for the end user to query. But we ran into two major problems with this”
Because Writer has full-stack ownership (models, RAG system, applications), the research team can work together on all components rather than being isolated, enabling the practical problem-solving approach.
“I think what's really cool about the way our research team works is that they're very focused on solving practical problems for our customers. Uh they're not just sort of like working in isolation uh working on theoretical things. They're actually driven by customer insights.”
A sketch or narrative description of a technical system is more practical and usable for practitioners than a detailed 'blueprint' that attempts to capture all implementation details and moving pieces.
“please consider this a sketch and not a blueprint of what is currently in production. Of course, there are like many moving pieces and many layers to this. Uh, but I want to abstract it enough to make it something that is practical and and usable for people.”
Writer's enterprise customers include Fortune 500 and Global 2000 companies, many in highly regulated industries like healthcare and finance where accuracy and low hallucination rates are critical requirements for deployment.
“as we've been building knowledger graph over the years, it's been an interesting journey as we've been working with these Fortune 500 and global 2000 companies at scale. Most of them or many of them are in highly regulated industries like healthcare and finance where accuracy and low hallucinations are super important.”
Writer has domain-specific models including Palmyra Med (for medical/healthcare) and Palmyra Finn (for finance), indicating their approach to specialized vertical solutions.
“domain specific uh specialization. These are our domain specific models like Palmyra Med and Pomera Finn.”
Writer is an end-to-end agentic platform for enterprises that builds its own models, graph-based RAG system, and software tools for enterprises to build agents and AI applications.
“If you're not familiar with writer, we're this end-to-end agentic platform for enterprises where we build our own models, we build our own graph-based rag system and have this suite of software tools on top of that for enterprises to be able to build agents and AI applications.”
In November 2023, Writer published an article discussing the benefits of knowledge graphs and the shortcomings of vector databases and simple similarity search for enterprise RAG at scale.
“In fact, if you look at this article from November 2023, which in AI time is like prehistoric times, um we actually talk about the benefits of knowledge graphs and the shortcomings of vector databases and simple similarity search for enterprise rag at scale.”
There are additional techniques beyond knowledge graphs, fusion-decoder, and context-aware chunking that Writer uses to improve RAG quality, but they don't have time to cover all of them in this talk.
“there's a few other techniques that we don't have time to go over but point being we're we're assembling together multiple techniques based on research to get the best results we can for our customers”
The problems Writer faced with graph databases (data conversion difficulty, scaling costs, expertise limits, Cypher limitations, LLM preference for text) were not fundamental to graph technology itself but specific implementation challenges at that time.
“Now this is not an indictment of any graph database technology. It's just that we were running into these issues at the time. a couple of years ago.”
Writer's research team works on four main areas: enterprise models (like Palmyra X5), practical evaluations (like FailSafe QA finance benchmark), domain-specific specialization (like Palmyra Med and Palmyra Finn), and retrieval and knowledge integration.
“Our research team, we have a cracked research team at Ryder and they have four main areas of focus. Enterprise models like like our Palmyra X5 model, that's the one powering the chat on the AI engineer website right now. Practical evaluations like our finance uh benchmark called failsafe QA domain specific uh specialization. These are our domain specific models like Palmyra Med and Pomera Finn. And then what our focus is here retrieval and knowledge integration.”
Writer published a white paper describing their findings on the combination of knowledge graphs with fusion-decoder for improved hallucination reduction.
“it really helped our hallucination rate. really drove it down and then we published a white paper with our own findings of it.”
Sam Julene is the Director of Developer Relations at Writer and maintains a personal website at sjulene.com with writing, books, and newsletters.
“I'm Sam Julene. I'm the director of developer relations at Writer and you can find most of my writing and books and newsletters and all of those things at sjulene.com.”
Writer published a white paper with findings on their implementation of knowledge graphs with fusion-decoder for RAG.
“it really helped our hallucination rate. really drove it down and then we published a white paper with our own findings of it”
Writer's research team focuses on four main areas: enterprise models (like Palmyra X5), practical evaluations (like the FailsafeQA finance benchmark), domain-specific specialization (like Palmyra Med and Palmyra Finn), and retrieval and knowledge integration.
“Our research team, we have a cracked research team at Ryder and they have four main areas of focus. Enterprise models like like our Palmyra X5 model, that's the one powering the chat on the AI engineer website right now. Practical evaluations like our finance uh benchmark called failsafe QA domain specific uh specialization. These are our domain specific models like Palmyra Med and Pomera Finn. And then what our focus is here retrieval and knowledge integration”
Writer is an end-to-end agentic platform for enterprises that builds its own models, graph-based RAG system, and software tools for enterprise AI agents and applications.
“we're this end-to-end agentic platform for enterprises where we build our own models, we build our own graph-based rag system and have this suite of software tools on top of that for enterprises to be able to build agents and AI applications”
The speaker will present two intertwined narratives: Writer's journey to graph-based RAG and the first-principles thinking that made the team successful, both of which inform system building.
“I'm going to weave together these two stories of our journey to graph-based rag and sort of the first principles thinking that I think has made our team successful in putting together this system”
The described system is a sketch and not a blueprint of what is currently in production at Writer, with many moving pieces and layers abstracted for practical applicability.
“I want to talk about sort of how we got to this point and where we are now. And I'm just going to put a blanket caveat on here that please consider this a sketch and not a blueprint of what is currently in production. Of course, there are like many moving pieces and many layers to this. Uh, but I want to abstract it enough to make it something that is practical and and usable for people”
Writer is actively hiring across research, engineering, and product roles.
“So if you want to join this amazing team, we're hiring across research, engineering, and product.”
If practitioners were implementing graph RAG now (post-2024), they would likely avoid the problems Writer faced with graph database technology, but the specific historical context of a couple years ago shaped Writer's decisions.
“Now again, if you were to do this now, you might not run into those problems, but this is what we ran into historically.”
The AI Engineer website's chat feature is powered by Writer's Palmyra X5 model.
“Enterprise models like like our Palmyra X5 model, that's the one powering the chat on the AI engineer website right now.”