
The Future of Knowledge Graphs in a World of Large Language Models
What this covers
This is a conference keynote, given at the Knowledge Graph Conference 2023 in New York, by a knowledge graph practitioner who frames the talk around a single question: what happens to the field now that ChatGPT has arrived? It opens with the shock of knowledge engineers who met at Dagstuhl two months before the release, then works through a running example. Google and Wikipedia disagree about where the actress Ena Begović was born. The speaker uses the case to show how knowledge ecosystems get contaminated when sources copy one another, and then to probe how ChatGPT guesses at the answer.
The middle of the talk is a catalogue of what large language models) do badly. They are inconsistent across facts and across languages, cannot reliably tell what they know from what they don't, and show no awareness of contradicting themselves after correction. They also handle simple math poorly while spending 175 billion parameters on the attempt. The speaker calls the autoregressive transformer a Turing tar pit. Cost is a second thread. Per-query figures put a ChatGPT-style answer at roughly 50 times the price of a Wikidata query. The talk also covers training costs, Sam Altman's claim that the age of giant models is over (a contested forecast), and the way a leaked Meta model was quickly squeezed onto consumer hardware.
The constructive part is where the talk turns. LLMs are described as the best knowledge-extraction tool in decades, well suited to populating a knowledge graph. The graph in turn supplies an editable, auditable store that is meant to "overfit for truth." The speaker favors augmented models, such as Toolformer-style plugins, that look facts up or call functions rather than memorizing everything. One concrete proposal is a third special value, it's complicated), for cases a graph cannot yet express. The talk closes with Wikifunctions and Abstract Wikipedia.
Knowledge graphs and large language models are complementary systems: LLMs excel at language understanding and knowledge extraction, but knowledge graphs provide the accurate, auditable, cost-efficient ground truth that LLMs need to avoid hallucination and maintain factual integrity.
- LLMs are 50x more expensive per query than knowledge graphs and struggle with consistency, math, and fact verification
- Knowledge graphs can be edited, audited, and curated to provide reliable ground truth; LLMs should be augmented with external knowledge lookup rather than internalizing all facts in parameters
- LLMs are exceptionally good at knowledge extraction to populate knowledge graphs, creating a symbiotic relationship where each system solves the other's core weaknesses
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Google and Wikipedia/Wikidata provided different answers about the birthplace of Croatian actress Ena Begović, with Wikipedia claiming Split while the correct answer is likely Trpanj, demonstrating how knowledge ecosystems can become contaminated when sources copy from each other.
“I was surprised that Google and Wikipedia/Wikidata were giving different answers so I tried to figure out 'The Truth' and fix it. Here's a screenshot of Google answering the question, Begović was born in Trpanj. Wikipedia used to say Split. After a bit of hunting through sources I figured out that it mostly likely is Trpanj and that a lot of the sources were copying from Wikipedia and Wikidata. It became contaminated by Wikipedia and Wikidata.”
ChatGPT reached 100 million users two months after launch, setting a record for the fastest technology adoption in history.
“ChatGPT, a large language model, reached 100 million users two months after launch. This is a record in how fast a technology is spreading.”
ChatGPT cannot reliably determine what it knows versus doesn't know; when asked about mayors born after 1998 (making them younger than 23 at ChatGPT's 2021 knowledge cutoff), it correctly noted this was implausible, but when asked in different ways or via BingChat, it claimed no knowledge of such mayors despite them existing in Wikidata.
“GPT isn't particularly good with knowing what it knows. For example, here we are asking for cities with Mayors born after 1998... ChatGPT correctly points out that that would make them younger than 23 at its cutoff date of 2021 so it is unlikely that anyone would be mayor at that age... Let's check Wikidata. I again use ChatGPT to help me write a query... And indeed: Wikidata has an answer. It tells me about Kristan von Waldenfels who was born in April 2000, Mayor of Lichtenberg in Bavaria, Germany.”
Research on how knowledge is stored in large language model parameters is beginning to understand why models must be so large; Stable Diffusion 1 has only 890 million parameters (200x smaller than GPT-3's 175 billion) because it doesn't need to internalize knowledge of individual entities.
“There is work that is trying to understand how knowledge is stored in the parameters of a large language model. And when we look at this work we start to understand why large language models are large. Do they really have to be this large? Let's compare with Stable Diffusion 1... GPT-3 had 175 billion parameters. How many parameters do you think are in Stable Diffusion 1? Knowing all these people, being able to generate them? 890 million parameters. Now, I think 890 million is a lot, but GPT-3 is about 200 times larger than Stable Diffusion 1.”
Sam Altman, CEO of OpenAI, describes the compute cost of ChatGPT as 'eye-watering' and John Hennessy, chairman of Google and former Stanford computer science professor, estimates a ChatGPT-like Google would be 10 times more expensive than current Google infrastructure.
“Sam Altman the CEO of OpenAI, the company making ChatGPT, describes the compute cost of ChatGPT as eye-watering... John Hennessy, chairman of Google and former computer science professor in Stanford, so he really knows what he is talking about, said that running a ChatGPT-like Google would be 10 times more expensive.”
Fine-tuning existing open-source language models is considerably cheaper than training models from the foundation, making this approach economically viable for organizations that cannot afford training from scratch.
“Good news is you probably don't have to do this training but you can just hopefully reuse an existing open source model that you can fine-tune to your task. This will be considerably cheaper than to train a model from the foundation on.”
ChatGPT cannot reliably generate code to fix problems in itself because understanding and modifying neural networks is fundamentally opaque, making it impossible for LLMs to explain what they need to change to fix mistakes.
“which isn't great if you have to explain to someone 'hey sorry, I cannot fix your problem. The thing is making a mistake but I don't have a clue how to make it better'”
Knowledge graphs provide ground truth for language models, solving the problem of hallucination and enabling fact verification.
“The knowledge graph provides you with the ground truth for your language models.”
When Meta's LLaMA model was leaked, engineers quickly adapted it to run on consumer hardware within days, then on a phone, and subsequently on a Raspberry Pi, demonstrating that even large models are still more expensive to run than knowledge graphs.
“when LLaMA was leaked within days someone managed to run it on consumer hardware. A few days later we got it running on a phone. And then a Raspberry Pi. It was very slow but it ran. And by the way as ridiculous as that sounds this still means that these giant models are more expensive to run than a knowledge graph.”
Current state-of-the-art large language models cost millions of dollars to train, with GPT-4 costing more than $100 million and GPT-3 costing around $4 million, representing prohibitive barriers to entry for model development.
“Each of these models also need to be trained and current state-of-the-art language models cost millions of dollars to train. For GPT-4 the cost was given as more than $100 million. The cost for GPT-3 was around $4 million.”
Sam Altman believes the age of large language models is already over due to diminishing returns of further reading after consuming a million books worth of text, and OpenAI is shifting from GPT-5 development toward exploring new ideas.
“some folks like Sam Altman think that the age of large language models is already over. OpenAI says it's because of diminishing returns of further reading. Well, guess what, after reading a million books worth of texts you don't seem to learn too much new stuff. He said that OpenAI is not working on GPT-5 but that they want to explore new ideas.”
Augmented language models (like Toolformer and ChatGPT plugins) can enrich large language models with additional systems that are efficient at specific tasks like math, function calls from Wikifunctions, or fact lookup in knowledge graphs like Wikidata.
“One possible solution to this capability gap are so-called augmented language models, Toolformer being a particularly well-known example. Or for ChatGPT that's what ChatGPT plugins are there for. The idea is that we can enrich large language models with additional systems which are good and efficient at specific tasks such as math or other functions from Wikifunctions or looking up facts or query results in a knowledge graph such as Wikidata.”
Large language models are exceptional tools for knowledge extraction and are likely the best tool for knowledge extraction developed in a decade or two, making them ideal for populating knowledge graphs.
“the other way around is also true: large language models can be an amazing tool to speed up the creation of a knowledge graph. They are probably the best tool for knowledge extraction we have seen developed in a decade or two.”
Large language models are inconsistent whether about individual facts or across different languages and are 'not yet great' at mathematical operations that could be performed in single CPU operations, yet they use 96 layers and 175 billion parameters to perform these tasks.
“Large language models are not yet great at being consistent whether about individual facts or across different languages. They are also not always very good at math. But even if they were good at math: we have to answer the same question that we do for looking up facts in a knowledge graph: Why would you ever use a 96 layers 175 billion parameters model to do multiplication when that's something you can do in a single operation on your CPU?”
Knowledge should be extracted into symbolic form where the system is designed to 'overfit for truth'—stored in a system that can be edited, audited, and curated rather than a system trained to return knowledge with probability estimates that hallucinate.
“We want to extract knowledge into a symbolic form. We want the system to overfit for truth. And this is why it makes so much sense to store the knowledge in a symbolic system. One that can be edited, audited, curated, understood, where we can cover the long tail by simply adding new nodes to the knowledge graph, one we don't train to return knowledge with a certain probability, to make stuff up on the fly, but one where we can simply look it up.”
A manager at Google used to say 'you can machine learn Obama's birthplace every time you need it but it costs a lot and you're never sure it is correct', capturing the fundamental problem with storing knowledge in LLMs rather than knowledge graphs.
“my manager at Google used to say: you can machine learn Obama's birthplace every time you need it but it costs a lot and you're never sure it is correct.”
Wikidata and similar knowledge graphs should introduce a third special value: 'it's complicated,' to explicitly mark cases where answers cannot be stated with current knowledge graph expressivity, preventing dangerous or wrong answers and flagging areas where LLMs should be trusted instead.
“My suggestion is to introduce a third special value: 'it's complicated'. I usually get people laughing when I make the suggestion but I'm really serious... 'It's complicated' is what you would use if the answer cannot be stated with the expressivity of your knowledge graph. This helps with maintaining the graph to mark difficult spots explicitly. This helps with avoiding embarrassing, wrong, or flat out dangerous answers and given the interaction with LLMs this can in particular mark areas of knowledge where we say 'Don't trust the graph! Can we instead train the LLM harder on this particular question and assign a few extra parameters for that?'”
ChatGPT's SPARQL query for Croatian actors had a subtle error—it returned actors born in places within Croatia rather than Croatian actors—and when executed against Wikidata found 445 results, with 154 (one-third) born in Zagreb, providing the conditional probability basis for ChatGPT's guess of Zagreb as Ena Begović's birthplace.
“The query can be copied and pasted right into the Wikidata Query Service and it runs giving 445 answers. And you can already see in the screenshot that the first few are all in Zagreb. So now we can see that of the 445 Croatian actors for place of birth 154 are born in Zagreb, about a third. This gives a pretty good conditional probability for the birthplace of Croatian actress. ChatGPT is guessing the place of birth for Ena Begović in Croatian even though it knows it in English.”
Autoregressive Transformer models like ChatGPT are Turing complete and represent 'a very expensive reiteration of a Turing tar pit'—you can do everything with them but shouldn't.
“Autoregressive Transformer Models such as ChatGPT are Turing complete and they are just a very expensive reiteration of a Turing tar pit. You can do everything with them but it doesn't mean you should.”
ChatGPT stores knowledge not in a language-independent way but separately within each language, which is why asking the same factual question in different languages produces different answers.
“knowledge in ChatGPT seems to be not stored in a language independent way but is stored within each individual language. Depending on the language I ask the question in I receive a different answer.”
ChatGPT can generate SPARQL queries for knowledge graph databases like Wikidata with correct QIDs (identifiers) with zero-shot learning, and can even generate tables of EU member countries with their correct QIDs, demonstrating memorized knowledge of thousands of structured identifiers.
“But still, this is fascinating. ChatGPT got out of the box all of the QIDs right in this query: for Croatia, for actor, you got the property ID right, it can make SPARQL queries that are syntactically right, and all of that with zero shot extra training... you can totally ask ChatGPT to make you a table of all the countries of the European Union and their QIDs. It has a lot of QIDs memorized.”
Answering a six-token question with a 60-token ChatGPT response costs approximately 50 times more than answering via Wikidata query service; a thousand queries cost $7.56 via GPT-4 Azure pricing versus 14 cents for Wikidata cloud hosting.
“if I look at Azure pricing for GPT-4 a thousand queries as asked before would cost $7.56. So that's 14 cents compared to more than seven dollars or a factor of 50. 50 times more expense can make a difference.”
Knowledge engineers and practitioners met in Dagstuhl, Germany in September 2022 (two months before ChatGPT's release) to discuss the role of Knowledge Engineering in the 21st century and were in shock about the implications for their field.
“Last year in September, that's two months before the release of ChatGTP, a number of researchers and practitioners in the knowledge graph field met in Dagstuhl in Germany. Our goal was to discuss the role of Knowledge Engineering in the 21st century. We were almost under a kind of shock, talking about it, processing and trying to figure out what it really means for us.”
Even when LLMs are corrected with additional information and guided to the right answer, they show no awareness of the contradiction between their previous incorrect answer and the corrected answer, demonstrating a fundamental limitation in consistency awareness.
“Now that we know the answers we can guide BingChat and get the Mayor of Lichtenberg and his age and obviously this answer is inconsistent with its previous answer but ChatGPT or BingChat are both completely unaware of this inconsistency and don't care.”
ChatGPT runs on the fastest and most expensive hardware you can buy, Google runs on the fastest and most expensive hardware you cannot buy, and Wikidata runs on 'abandonware on a server somewhere', illustrating different scales of infrastructure investment.
“ChatGPT is running on the fastest and most expensive hardware you can buy. Google is running on the fastest and most expensive hardware you cannot buy. And the Wikidata query service is running abandonware on a server somewhere.”
ChatGPT took about 5 seconds to answer the query 'Who created the School of Athens?' including nearly 1 second for parsing, while Google took 0.5 seconds including parsing and results retrieval, and Wikidata took about 0.25 seconds to generate the answer.
“here's one thing ChatGPT took about five seconds to answer my question... Google took half a second including parsing... Wikidata took about a quarter of a second to produce the answer.”
The speaker worked at Google about six to seven years ago on the Knowledge Graph and was asked whether the system's costs in resources, money, and manpower could be justified or if Google should stop investing.
“About six or seven years ago I was working at Google on the Knowledge Graph, back then when knowledge graph wasn't necessarily a generic term yet. And there was a big question Google had to ask itself: the Knowledge Graph costs a lot of resources, a lot of money, a lot of manpower to run and maintain.”
A knowledge graph represents things and relations usually stored in a graph database, with Wikidata being the largest publicly available free knowledge graph that anyone can edit.
“What is a knowledge graph? A knowledge graph represents things and relations usually stored in a graph database. Wikidata is an example. Wikidata is the largest publicly available free knowledge graph that anyone can edit.”
ChatGPT's answer to 'Who created the School of Athens?' was 'brilliant' and 'absolutely correct'—providing not only the right answer (Raphael) but also valuable context about the fresco, when it was painted, and its artistic significance.
“ChatGPT answers 'The School of Athens is a famous fresco painted by the Italian Renaissance artist Raphael...' This answer is not only correct, it is brilliant! There is absolutely nothing to complain about. It gives context, it answers the question, it's great.”
A large language model is a neural network with nodes connected by weights that are trained on natural language texts; GPT-3 specifically has 96 layers and 175 billion parameters.
“What is a large language model? Model here means that we have a neural network, nodes connected with each other with weights. The weights in this neural networks are trained on some input and desired outputs in order to create good outputs on novel inputs. They are language models because they are trained on natural language texts. And it is a large language model because it is trained on lots and lots of natural language text... GPT-3 has 96 such layers as in the picture and it has 175 billion parameters, that is, weights.”
The speaker's brother, who lives in a small Croatian village of 180 people, called to discuss Stable Diffusion, an image generation model, demonstrating that AI technologies are reaching even isolated, non-technical populations.
“my brother who lives in a small village of 180 people on a Croatian Island, he is roughly here on this chart — but he called me last fall to talk about Stable Diffusion, the image generation model.”
The speaker was invited to keynote the Knowledge Graph Conference in May 2023 and was asked by François Scharffe to speak about the relationship between knowledge graphs and large language models based on the topic people most frequently ask them about.
“I was invited by François Scharffe to keynote one of the days of the Knowledge Graph Conference in May 2023 in New York, NY... François asked me what topic I want to talk about... there's one topic everyone talks to me about and this is: what about knowledge graphs and large language models? And he said perfect, do that!”
The speaker acknowledges they were 'most certainly in doubt' about what to say despite having spoken and thought about the knowledge graphs and LLM topic for a decade with very smart people, but they are certain 'something big is happening' with AI technologies.
“It's May now, and I'm still full of doubts about what to say. I don't know what the future of knowledge graphs is. But I have been talking about this for a decade now with a lot of very smart people so I hope that what I say will be worth your time. There's one thing I have no doubts about: something big is happening.”
The talk is focused only on the technical relationship between knowledge graphs and large language models, and explicitly excludes discussion of ethical questions, blunders, economic impact, job effects, copyright, or existential risk.
“this is a focused talk. It is only about knowledge graphs and large language models. It is not about the ethical questions of large language models, it's not about blunders of current LLMs, it's not of the effect that those systems will have, about the impact on jobs and the economy, it is not about questions of copyright or whether LLMs are a path to the destruction of the human race.”
The speaker's second disclaimer is that nothing in the talk is generated by AI unless explicitly marked or shown as a screenshot from AI, and this is not one of those talks that ends with a revelation that everything was AI-generated.
“My second disclaimer: nothing in this talk is generated by an AI unless it is explicitly marked so or you see a screenshot from an AI. This is not one of those talks where we have a reveal in the end that all of this was AI generated.”