What this covers

This is a walkthrough of a Harvard-led preprint, presented by a single narrator on the Discover AI channel, on a Knowledge Graph agent for medical question answering. The framing problem is that standard LLMs fail on knowledge-intensive medical questions in three ways: they retrieve incorrect information, miss key information, or drift from scientific and medical knowledge. Retrieval-Augmented Generation (RAG)) is meant to help, but the narrator stresses that it inherits errors from incomplete or wrong repositories and often has no step that verifies what was retrieved. The proposed answer is KGA, which pairs the LLM's uncodified language knowledge with the codified structure of a medical graph.

Most of the video follows the Four-Phase Knowledge Graph Agent Process: generate, review, revise, answer. In generation, the LLM turns a question into triplets of head, relation and tail. UMLS codes map those terms onto the graph. Review checks the triplets against the graph, and for this the narrator explains how embeddings from the LLM and from the graph (using the 2013 Trans-E method, where h + r ≈ t) are aligned through a projection layer into a shared space. Revise then repairs incomplete or incorrect triplets iteratively, and answer combines the validated results.

Some points are more contentious or more speculative. The LLM may propose relations the graph does not contain, which are then validated, and the narrator warns that revision could plateau if the graph is too simple to validate richer triplets. The reported results show Llama-based accuracy rising from about 50% to 70% or more, with larger gains on questions involving more medical concepts. The research used four Nvidia H100 GPUs. The narrator also says they are considering replacing their own RAG systems with this approach, and the closing section goes through the prompts in detail.

Sharpest takeaway

Harvard researchers have developed a knowledge graph-based agent methodology that integrates large language models with structured medical knowledge graphs to significantly improve medical question-answering accuracy by combining unstructured semantic reasoning with verified factual knowledge.

  • Standard LLMs achieve only 30-50% accuracy on expert medical questions due to inability to integrate multiple knowledge sources and verify information
  • The methodology uses knowledge graph embeddings aligned with LLM token embeddings to ground medical reasoning in verified structured knowledge
  • Results show accuracy improvements from 50% to 70%+ on multi-concept medical questions when LLMs are fine-tuned on knowledge graph completion tasks

The claims · ranked15 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.80

Complex medical reasoning requires simultaneous consideration of dependencies across multiple medical concepts and precise, localized domain knowledge of semantically similar concepts that can carry different medical meanings.

causalhigh valueestablishednovelty 2/4durability 4/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“llms face challenges in complex medical reasoning because such reasoning would require here a simultaneous consideration of dependencies across multiple medical Concepts and I will show you those Concepts in a minute plus a precise local IND domain knowledge of semantically similar Concepts that can carry different medical meanings”

0.75

Retrieval-augmented generation (RAG) methods, while attempting to provide multi-source knowledge, are vulnerable to errors because data repositories and knowledge bases contain incomplete and incorrect information, and many RAG methods lack post-retrieval verification mechanisms to validate that retrieved information is factually correct and does not miss key information.

causalhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“Howard discovers that also those R methods can provide multisource knowledge yeah they are really vulnerable to potential errors and Howard goes on and says you know all those data repositories and the knowledge basis those rag malls draw from they contain incomplete and incorrect information leading now here in har Medical School inaccurate retrieval for their specific cases and how it goes on you know further many rack based methods simply lack a post- retrieval verification mechanism to validate The Retreat information is factually correct and does not miss some key information especially here in medicine”

0.75

Harvard researchers discovered that standard large language models suffer from three critical problems in medical applications: incorrect retrieval of information, missing key information, and misalignment with scientific and medical knowledge.

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“llms have suffer from incorrect retrieval missing key information and misalignment with in scientific and medical knowledge”

0.69

The quality of triplets generated depends heavily on the capabilities of the underlying LLM: better LLMs produce more complex and accurate triplets, which can increase system intelligence when integrated with knowledge graphs.

causalhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“the better your llm the more the llm is able to think and reason and detect and create complex more complex triplets the more intelligent the system could be”

0.66

The use of UMLS codes as an intermediary between LLM-generated concepts and knowledge graphs allows for consistent and accurate mapping of medical terms, ensuring that LLM outputs align with standardized medical terminology.

causalhigh valueestablishednovelty 1/4durability 4/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“the use of these umls codes as an intermediary between the llm generated Concepts and the knowledge graph allows for consistent and accurate mapping of medical terms”

0.60

The Trans-E embedding method (from 2013, developed by CNRS and Google) is a translation-based model that learns vector embeddings of entities and relationships in multi-relational data by optimizing a simple objective: representing each triplet such that h + r ≈ t in a low-dimensional vector space.

factualhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“this is from 2013 this is the first time it happened we have a lot of um embeddings functionalities I would recommend you this particular paper this is the original trans e paper for translating EMB addings for modeling multi- relational data from cnrs and Google 2013 about 2013 and they face you the problem of embeding the entities in relationship of a multi- relational data into low dimensional Vector spaces”

0.52

The LLM is not constrained to existing relationships in the knowledge graph and can propose novel connections based on its language understanding, which are then validated against the knowledge graph.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“the llm is not constrained now to existing relationship in the knowledge graph it can propose your novel connection based on its language understanding”

0.52

There is a potential performance plateau when iterative revision reaches a validation threshold that the knowledge graph cannot exceed; if the knowledge graph has insufficient complexity to validate more sophisticated triplet structures that the LLM can generate, the revision process may reach an equilibrium where further refinement is blocked by the knowledge graph's complexity limits.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“you have to make sure that the complexity level of both of your system the llm and the knowledge growth are about the same or you must know hey my llm has a complexity level of 6 out of 10 but my knowledge graph has eight out of 10 you have to know what procedures to follow”

0.50

The 'revise' phase iteratively adjusts incorrect or incomplete triplets by proposing new relations or entities based on related medical concepts, validated repeatedly against the knowledge graph until triplets are validated, enabling error correction through iterative refinement.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“this happens if the triplets that were generated are found to be incomplete those triplets are found to be incorrect so what we do theoretically just throw them away but Harvard tells us hey maybe there is some hidden information in there so what we do we keep those but we get them a special treatment and now since we have this super llm the llm plus attempts now to adjust incorrect triplets by proposing here new relations or new entities in our triplet structure based on related medical Concepts”

0.47

The methodology's 'generate' phase is designed to prompt the LLM to follow specific procedures for generating relevant triplets from medical questions, where a triplet consists of a head entity (H), a relationship (R), and a tail entity (T), representing medical concepts and their relationships.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“at first we are at a generate level this is designed here to prompt the llm to follow different procedures and I will show you these procedures for generating relevant triplets and you know triplets from graph Theory so we have here as I told you two kind of questions the choice aware questions and the non-choice aare questions”

0.47

The methodology must achieve alignment between LLM token embeddings (from the language model's understanding of text) and knowledge graph structural embeddings (from graph theory), by adding a projection layer that transforms the dimensionality and structure of LLM embeddings to match knowledge graph embeddings, creating a shared geometric space where both types of knowledge can be combined.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“the alignment between the llms token embeddings and the knowledge graph structural embeddings this is NOW essential to combine here exactly also the unstructured n of the llm with the structured domain specific medical knowledge found in a medical Knowledge Graph so how we do this well it is easy we simply add a projection layer for example now we do this on the token embeddings which will adjust their dimensionality and their structure to match here the format of the knowledge graph embeddings”

0.45

The speaker is considering replacing all their existing RAG systems with the Knowledge Graph Agent methodology because it is more powerful than RAG, suggesting the methodology has implications beyond medical applications.

normativehigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“I immediately have five other ideas how we can build on this and further improve this but the goal of this video is just to make you familiar with this study by Howard and fiser...I thought about replacing here all my rag systems also with this new methodology because it is so much more powerful and if I don't need rack maybe that's maybe not a bad idea if I can substitute it with a more powerful methodology”

0.39

On multiple-choice medical questions with Llama 3.8B, applying the KGA methodology improves accuracy from approximately 50% to 70% or higher, with larger improvements on questions with more medical concepts.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“if you take the same llama 38b and you apply this new method ology you see that you go from I don't know 50% to about 70% plus”

0.20

The methodology is called 'Knowledge Graph Agent' or 'KGA' (per the Harvard publication), though the speaker notes uncertainty about what the acronym fully stands for.

factualspeaker onlynovelty 0/4durability 3/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“Howard calls this methodology the kga Ravon so Knowledge Graph a ravion don't ask me for what it stands”

0.13

The research was conducted using a cluster of 4 Nvidia H100 GPUs, demonstrating that the computational requirements for implementing the KGA methodology are not prohibitively high.

factualspeaker onlynovelty 0/4durability 1/4· Unidentified Speaker — Harvard Presents NEW Knowledge-Graph AGENT (MedAI) [Fm68I-phaiY]

“Harvard writes here they used here a cluster 4 Nvidia I think it was an h100 GPU configuration so it was not that complex to do all of this calculations here in this demo”