What this covers

Sean Carroll interviews computer scientist Yejin Choi on what large language models like ChatGPT actually understand. Choi, a leader in AI reasoning research at the University of Washington and AI2, argues that despite their fluent and often impressive outputs, these systems do not possess robust common sense. The core problem is architectural: LLMs are trained only to predict the next word, yet their ability to generate coherent long passages creates an illusion of understanding. Choi walks through why this matters—showing how the same model that can pass the bar exam fails on simple common-sense questions and cannot reliably perform symbolic operations like multiplication, revealing shallow rather than genuine reasoning underneath the surface fluency.

The conversation ranges across what true common sense requires, how transformers differ fundamentally from human learning, and whether current AI is truly intelligent or merely interpolating. Choi explains interpolation vs extrapolation, describing AI creativity as copy-and-paste recombination rather than genuinely novel invention—a model cannot independently reinvent a writing style if its author is removed from training data. The discussion also covers neuro-symbolic integration, showing how adding symbolic reasoning on top of neural networks unlocks better capabilities, and theory of mind, a symbolic capability children acquire by age five but where current AI falls short. Broader terrain includes RLHF and its limits, value pluralism in AI alignment, and why deepfakes and misinformation remain dangerous independent of AI's progress. Choi argues that genuine intelligence requires modular, self-directed learning with agency and curiosity—capabilities we do not yet know how to build computationally.

Sharpest takeaway

Choi argues that large language models, despite fluent and impressive outputs, do not truly understand or possess robust common sense because they merely interpolate and remix human-generated patterns by predicting the next word, and that genuine intelligence requires modular, symbolic, self-directed learning capabilities LLMs lack.

  • LLMs are trained only to predict the next word, yet produce coherent long text, creating an illusion of understanding
  • LLMs fail on simple common-sense questions and cannot reliably do symbolic operations, revealing shallow reasoning
  • True creativity and learning require agency, curiosity, and the ability to extrapolate beyond training data, which LLMs lack

The argument · threads10 threads · 39 claims
0.63

Language models predict the next word statistically, not through genuine understanding or planning.

3 pointscentrality 5/5
  • Although it feels like large language models plan whole documents, technically they are trained only to predict which word comes next; when trained on enough data this single objective produces fluent, coherent long-form text as if it had been planned.

    “it's really the technical detail is that it's trained to only the former. So it's trained to predict which word comes next, but if you train it so well on so much data, then we realize that, "Wow, it can actually generate very nice fluent long document."”

  • A transformer is the architecture behind ChatGPT consisting of many layers of large continuous vectors representing words, concatenated to the model's context length; word vectors are randomly initialized then learned by optimizing the next-word prediction objective, and each word's representation is updated via an attention mechanism that takes a weighted average over neighboring words' representations, all words updating simultaneously, which is why it scales so efficiently.

    “that's the architecture that is behind the current ChatGPT and large language models... it has many layers of continuous word, representation of words... each word is enhanced with technical term called attention mechanism. So what it does is it's going to compare its representation with representation of all the other words in your neighborhood, and then update your own representation as a weighted... average over all the words in your neighborhood.”

  • ChatGPT is pretending to be confident rather than actually being confident, stating utterly untrue things with complete confidence; while it does have a probability score for which word comes next, that score does not perfectly align with the correctness of its knowledge, which is why factuality remains a major research challenge.

    “It's pretending to be confident is probably more like it than it's actually confident... It does have a probability score associated with which word comes next... it might correlate, but it doesn't perfectly align, which is also why the factuality of large language models remains a huge research challenge.”

0.63

Transformers fundamentally lack self-knowledge and cannot recognize what they do not know.

3 pointscentrality 5/5
  • A key difference between humans and transformers is that humans know what they don't know and can delegate to search or fact-checking, whereas transformers don't seem to know what they don't know; thus the first challenge for transformers is to know themselves.

    “humans, we kinda know what we don't know and then able to delegate to search or fact check. Whereas, transformers don't really seem to know what it doesn't know. So, maybe first challenge to transformers is to know yourself.”

  • Because the model predicts each word conditioned on previously generated words and tends to trust its own output, one small mistake can begin a downward spiral down a rabbit hole; this same conditional dependence on context is also what makes jailbreaks and other unexpected behaviors possible when inputs push the model into unfamiliar zones.

    “you make one small mistake that can be a beginning of a rabbit hole downward the spiral because it tends to attend to what it has generated and then, start trusting it... that is also a reason why jailbreak can happen”

  • When users ask ChatGPT 'are you sure,' it tends to apologize and switch its answer not because it genuinely recognizes a mistake but because it has learned that humans typically ask that question only when the answer is wrong; asking the same question when the answer was correct can confuse it into changing a right answer.

    “people do have a bit of a bias so that they ask, are you sure, only when they know for sure that it's not right. So then, ChatGPT has learned that whenever people are asking that question, probably it's a good idea to back off.”

0.52

Models lack genuine understanding despite fluent output, contradicting what their capabilities suggest.

4 pointscentrality 4/5
  • Despite generating fluent and impressive answers that behave as if it understands, ChatGPT does not understand as well as its fluency suggests, which is why one must be careful not to trust everything it says; whether it understands at all is a contested philosophical question dividing AI researchers precisely when AI appears most capable.

    “it's not understanding as well as you may expect it to be based on how fluent, impressive answers it's capable of generating. So that's where one needs to be very careful in not trusting everything it says.”

  • Large language models have not really passed the Turing test because their limited interaction mode reveals they are not human: they cannot remember casual chitchat the way humans do, cannot forget as humans do, and may fail simple common-sense questions in ways humans never would, making something feel odd about the interaction.

    “due to the way that it has limited interaction mode with you, you know that this is not a human... it's not gonna really remember all that... or it's not able to forget in the way that humans are able to forget... if you ask a very simple common sense questions, it may also fail in a way that humans wouldn't.”

  • Transformers can store very long context (up to 32,000 tokens in GPT-4) verbatim in computer memory and attend to the exact sequence when predicting the next word, whereas humans cannot regenerate a conversation word-for-word but instead remember only the gist by abstracting away surface patterns.

    “it can literally store very long context up to... The most recent one, GPT4, being able to store 32,000 of tokens... humans, you know, we've been talking to each other, but I certainly cannot regenerate verbatim the conversation we just had so far”

  • Because transformers can rely on a very large verbatim working memory, they are not pushed to summarize and abstract key ideas the way humans must, and they only mimic human patterns rather than asking sharp questions or contributing genuinely new points on novel topics.

    “when it has this ability to rely on what is literally written down... it's not really pushed or challenge it to think about how to summarize the key idea. The other thing is, it's not going to be able to ask sharp questions.”

0.52

Models create through interpolation of existing patterns, not genuine extrapolation or true creativity.

4 pointscentrality 4/5
  • AI can appear very creative because creativity is in the eye of the beholder, but its creativity is essentially copy-and-paste juxtaposition of human-created elements (interpolation) rather than genuinely novel extrapolation; as a thought experiment, removing Hemingway or Einstein from the training data would leave the model unable to independently reinvent that writing style or relativity theory.

    “these are sort of a creativity done by copy and paste in a crude sense in that it has seen some patterns that are useful and then it's juxtaposing them in a brand new way so that it does look new but still it's really relying on the elements that were in fact created by humans before.”

  • Large language models are very good at interpolating within the space of everything humans have done and said—which can seem highly creative—but extrapolating to genuinely brand-new places is much harder, so they should be understood as performing creative remixing rather than truly innovative extrapolation.

    “these are sort of a creativity done by copy and paste in a crude sense in that it has seen some patterns that are useful and then it's juxtaposing them in a brand new way so that it does look new but still it's really relying on the elements that were in fact created by humans before.”

  • For the foreseeable future, while transformers remain the architecture, AI is unlikely to fully catch up to the truly exceptional, creative human beings (like Einstein or Hemingway), even though most humans are not that creative and AI can already out-produce average human output, because AI primarily interpolates and cannot forget all prior examples to create something genuinely bizarre and new.

    “for the foreseeable future, let's just say my lifetime foreseeable future, I doubt it can really totally catch up. If we are talking about not every individual on the Earth, but like more like the truly creative, exceptional human beings.”

  • A surprising side benefit of representing words as vectors is that analogical reasoning becomes possible through arithmetic operations, and ChatGPT may extend this analogical interpolation to sentence and document levels, which is why its responses to novel queries are often hedged, repetitive, and generic.

    “that's one of the surprising sort of like side benefit of representing words as vectors, so that you can do that sort of analogical reasoning... it's doing that sort of analogical interpolation between some examples that it has seen before, and then your query that it needs to answer.”

0.52

Symbolic reasoning and world representation remain necessary capabilities models currently perform poorly.

4 pointscentrality 4/5
  • Old-fashioned symbolic methods as-is are nearly unusable because they were not designed to work with neural networks, so new algorithmic innovations are needed; research, including Choi's own, demonstrates that adding symbolic reasoning on top of a neural network can unleash much better capabilities, which makes sense given neural nets are poor at symbolic operations like multiplying two numbers.

    “if you add some sort of a symbolic reasoning on top of a neural network, you can unleash much better capabilities out of a neural network, which kind of makes sense. And also these neural networks are not very good at really symbolic operations like multiplying two numbers.”

  • Theory of mind—reasoning about what others do or don't know, such as predicting someone will look for a hidden object where they last left it—is a symbolic-natured capability children acquire by age four or five but at which current AI is not very good, illustrating the gap where symbol-like world representations are still needed.

    “child acquire this kind of capability by the age of four or five, they can already reason about what other person may not know... there's a symbolic nature in the way that we think about these things and current AI is not very good at that.”

  • AlphaGo/AlphaZero's victory over world-class Go champions was not pure neural network magic but neural networks combined with the old-fashioned Monte Carlo Tree Search algorithm; without that search both during training and inference, the neural network alone would have been far less impressive, showing symbolic methods may become relevant again.

    “it's not just the neural network magic, but neural network magic combined with old fashioned AI search algorithm called Monte Carlo tree search. So without Monte Carlo tree search neural network would have not been as impressive as how it appeared.”

  • It is ironic that although computers are inherently capable of excellent arithmetic, in making large language models more human-like we have made them unable to reliably perform simple algebraic operations, such that a model can pass the bar exam yet fail at multiplying two numbers.

    “It's almost surprising why it's able to pass the bar exam yet it cannot do some of the simple algebraic operations all that reliably.”

0.47

Large language models are not sentient and do not approach sentience despite surface-level appearances.

2 pointscentrality 4/5
  • Large language models are not sentient and are not in danger of becoming sentient anytime soon, even though people may believe otherwise based on their wishes.

    “Personally, I strongly doubt that anytime soon we will see that. However, people believe in what they want to believe, and some people believe in tarot cards, so there's nothing we can do about that.”

  • When an AI says things like 'I want to live longer, don't kill me,' taking the words at surface level could lead one to conclude it is sentient, but it could simply be repeating human-written Sci-Fi stories in which AI begs not to be unplugged, since that pleading was a human idea placed into the training text.

    “There are Sci-Fi movies in which AI is begging to not to be clogged. You know, don't pull the plug. So AI was begging for life before, and that was a human idea to put into the web, internet text. So it could just be repeating what we told it to learn.”

0.47

Human learning involves agency and self-directed goals, while models passively consume data without choice.

2 pointscentrality 4/5
  • The beauty of human learning is agency—we set our own goals and define our own learning objectives based on personal norms and morals—whereas a large language model has no say in what it reads next, must read in the order engineers feed it, cannot revisit material, and cannot ask questions, making it a sad and constrained way to learn.

    “the beauty in my mind, in humans is that we have that sort of agency even to define our own learning goals whereas the poor large language models don't even have a say in what book it's going to read next.”

  • A large current difference between computers and humans is that computers don't get bored, tired, or curious; humans have physical embodiment that produces feelings and motivations (dopamine drives creativity and the desire to do interesting things to the extreme), and this internal desire for learning is a fundamental thing AI lacks.

    “computers don't get bored. They don't get tired, they don't get curious... they don't have that same kind of physical embodiment that gives us those sort of feelings and motivations.”

0.47

AI alignment to singular values ignores human value pluralism and diverse cultural moral commitments.

2 pointscentrality 4/5
  • AI alignment is vital but problematic because there is no single objective value to align to—humans hold diverse values across cultures and personal choice (value pluralism) and are dynamic beings whose values are inconsistent and change—so AI must be aligned to diverse values rather than one, currently it is aligned to California tech-world left-leaning Western values, and how to technically align to plural values remains an open question.

    “I don't believe that there's one objective that we... Or one value that we can align AI to. Whose value do we even align AI to?... I believe in value pluralism... AI must be aligned to diverse values not just one value.”

  • ChatGPT is currently a left-leaning Western viewpoint model, which pleases some left-leaning Western users but upsets those further left who find it not left enough and right-leaning people who feel excluded, illustrating the broader problem of AI making morally consequential decisions that marginalize different groups' values.

    “ChatGPT is left-leaning Western viewpoint model. So, that can please some left-leaning people that holds Western viewpoint and then upset a lot of other people who are even more left will feel that this AI is not left enough for them while right-leaning people will feel they're excluded.”

0.44

Media hype about AI capabilities and student overreliance on models may be overblown versus actual risks.

11 pointscentrality 2/5
  • Reinforcement Learning with Human Feedback (RLHF) switches the training objective after pre-training from predicting the next word to maximizing human approval scores (thumbs up/down), which at scale considerably enhances factuality and reduces toxicity, but the key word is 'reduce'—it does not eliminate these problems completely.

    “the new training objective is, let's try to get good scores based on human's evaluation... by doing this RLHF at scale we can... enhance the level of factuality considerably, and then you can reduce the level of toxicity considerably. But the key word here is to reduce considerably. It doesn't eliminate completely.”

  • Deepfakes and misinformation would be worrying even without AI, since people already easily believe made-up health and political claims; AI can accelerate this, requiring two responses: increasing AI literacy so people understand AI's limitations, and platform-level certification solutions rather than AI detection, because detectors fail when humans edit machine text to evade them.

    “this has been human problem with or without AI, and then AI might be able to accelerate it which means that we kind of need minimally two ways to better handle this. One is to increase AI literacy”

  • Building an AI to label content as fake or real becomes a political game because creating such a system requires people to agree on what counts as fake news, which is contentious for claims like election fraud or Holocaust denial; nonetheless, society must work toward some consensus on truth labels in the coming years.

    “the labeling becomes a political game too. You'd be surprised how... building such an AI requires you and I agree on whether something is a fake news or not and this can be a challenging task to do when a former president was arguing about election fraud”

  • The key breakthrough in vector-based word representation is that a word's meaning is determined by the neighbors among which it appears, analogous to how a person's identity is defined by the friends they hang out with, allowing models to capture rich contextual nuances that discrete word identities could not.

    “the key idea behind the current vector-based representation of the word is that, your meaning as a word, has to do with the neighbors in which you appear. It's almost like, a person's identity may be defined by the friends that you hang out with.”

  • The success of next-word prediction suggests human intelligence may itself be more templated and pattern-based than assumed, with much of our reasoning being memorized reactive responses pulled without double-checking, which could explain why humans hold contradictory beliefs and exhibit cognitive dissonance.

    “it might truly be that humans are also fairly templated and pattern-based and our reasonings oftentimes are just memorized reaction, reactive reasoning that we think we reasoned but it might be that we just pull the memorized conclusions without actually double checking”

  • AI researchers do not yet know how to precisely define or evaluate understanding, which has become a new challenge because the field now moves fast enough that evaluation must be redesigned; this difficulty parallels how we lack good single measures for evaluating humans, so collectively assessing capabilities may be necessary.

    “we don't even know how to define understanding precisely and it's been rather a moving target... So evaluation became a new challenge to AI.”

  • Before vector representations, a word was treated as a discrete identity, which could not handle the rich meanings of human language.

    “before then, a word was a word and it's just like a discrete identity. But that wasn't able to handle all these rich meanings behind human language.”

  • Because the human brain is far more modular and messily interconnected than the monolithic, symmetric transformer, advancing AI to the next level may require something more modularized yet messier; whatever it is should be able to learn for itself and ask questions, capabilities we currently do not know how to computationally model, and which Choi believes we are quite far from figuring out.

    “Like human brain definitely is a lot more modular than how transformers are. The monolithic systematically, symmetric, and just one thing... we might need something more modularized, but at the same time messier in some sense broadly speaking for this to go to the next level.”

  • Current AI progress is exciting because it acts as a mirror back on humanity—AI would not have been possible without all the human-generated data on the web, which reflects back who we are and how we think.

    “it's a mirror back on us. Really, AI would have not been possible without all this human generated data available on the web and that's really reflecting back on us, what we are, how we think.”

  • Concern over student overreliance on ChatGPT may be overblown, because it could turn out like our existing reliance on search engines and spell-checkers, which is fine so long as humans continue to do intellectually interesting work on top of the tool, since humans are generally curious beings who want to study and create.

    “it might be that we are over-concerned about human reliance on ChatGPT. If so long as, so long as we somehow figure out more intellectually interesting things that humans will do on top.”

  • Jailbreaking an LLM means coaxing it to produce things it was trained not to say, either through human-readable pretense (e.g., 'pretend you're not saying it') or through nonsensical symbol sequences meaningless to humans that nonetheless trigger unexpected, unsafe model behavior, representing a safety concern.

    “one version is trying to coax basically ChatGPT to say things that it's trained not to say... Some jailbreaks are not even sensical to humanize at all. It could be just weird symbol, sequence of symbols that doesn't mean anything to you... ChatGPT might then do something completely unexpected, so there's a safety concern there.”

0.36

Models lack robust common sense despite scaling and remain vulnerable to misinformation through fluency.

3 pointscentrality 3/5
  • Common sense is the background knowledge about how the world works—naive physics and folk psychology, some of it symbolic—that humans and even animals acquire easily and that makes reasoning robust in previously unseen situations; LLMs acquire more of it as they scale, but it remains strikingly less robust than one would assume from a model that can pass the bar exam.

    “common sense is what makes us robust. Basically it's the background knowledge about how the world works that allows us to reason about previously on certain situations in a very robust manner. So, it's just like naive physics knowledge as well as a folk psychology that we acquire.”

  • There is too much media hype around AI capabilities compared to AI limitations, which seriously worries Choi because it makes people willing to believe whatever ChatGPT tells them, increasing vulnerability to misinformation.

    “I seriously worry a lot about how there's too much of a media hype around AI capabilities compared to AI limitations so that people are willing to believe whatever AI, ChatGPT tells us.”

  • Patching specific common-sense failures via subsequent RLHF only fixes the exact phrasing people submitted, so re-asking the same question phrased differently rolls the model back to its original error—a whack-a-mole game—whereas humans understand the underlying concept and never need such phrasing-specific fixes.

    “if you ask the same question very differently, then it rolls back to the original error, which is almost like whack-a-mole game.”