What this covers

This episode of Intercom on Product, recorded the week OpenAI released ChatGPT, has Dez Traynor, an Intercom co-founder, questioning Fergal Reid, the company's Director of Machine Learning (transcribed as Frugal). It begins with the technical background: the Transformer architecture from Google's "Attention Is All You Need" paper, its advantages over LSTMs), and why next-token prediction at scale yields sonnets, working code, and a passable first-round coding interview. Reid argues that the old claim that these models merely match patterns now has to say which pattern is being matched, and that, unlike earlier ML breakthroughs, they don't stop looking magical once you understand them.

The middle of the conversation is about limits. Hallucination gets extended treatment: Reid says accuracy has improved sharply over one and three years, but believable wrong answers make direct use in medicine or law risky. He suggests people judge plausibility with fast System 1 thinking, which these models exploit. He also offers a heuristic that generative AI fits best where answers are cheap to verify, or where correctness doesn't matter, and where producing them is laborious. Other threads include chain-of-thought prompting, a model running Python to catch and fix its own errors, and a comparison to AlphaGo-style self-play. Whether this matches the internet in significance is flagged as contested, and Dez wonders whether the real effect will be exposing performative "work work."

The final stretch turns to customer support and Intercom's own products. Resolution Boss deliberately uses medium-sized models and outputs only customer-curated content. Smart replies show how small, constrained suggestions drive adoption. Reid describes showing a generated answer beside its source article so agents can check it, and warns about over-trust, which Dez ties to "normalization of deviance." They also weigh the operational cost of feedback-driven learning loops, and close by considering which other industries might be affected.

Sharpest takeaway

Large language models like GPT-3.5 and ChatGPT represent a fundamental capability shift comparable to the internet itself, but their deployment in production requires careful constraint through augmentation, verification loops, and domain-specific architectures rather than naive end-to-end usage.

  • Scaling transformer models with massive training data produces unexpected emergent capabilities in reasoning, creativity, and code generation that were previously thought impossible
  • Hallucination and factual errors remain a critical barrier to direct deployment, but human-in-the-loop systems with verification affordances and constrained response sets can unlock productivity gains
  • The optimal deployment pattern mirrors verification-complete problem classes: leveraging AI for tasks where humans can cheaply verify correctness or where no single correct answer exists

The claims · ranked32 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.75

GPT models trained at scale with simple objectives (next-token prediction) can accomplish creative tasks like writing sonnets, generating poetry, and producing stylistically varied content that were long assumed to require human-level creativity and semantic understanding

factualhigh valueestablishednovelty 2/4durability 3/4· Dez (Intercom co-founder)

“a lot of things that we thought were you know the domain only of human intelligence oh you just need tons of training data and a big model yeah so it's a remarkable time because a lot of things that we thought were you know the domain only of human intelligence oh you just need tons of training data and a big model”

0.75

OpenAI demonstrated a model executing code, detecting errors through Python execution, and iteratively fixing code until it works correctly—a closed-loop system where the model can verify its own outputs and self-correct

factualhigh valueestablishednovelty 2/4durability 3/4· Frugal (AI expert guest)

“they gave it a natural language and problem a bit like our intercom programming problem and then you know ask the system to synthesize code code but then put the code into a python prompt and when it got it wrong system saw all of that so try to execute the code it's all it was wrong and then took another go at another go until it got it right right”

0.74

Another safe pattern is to have the AI search the knowledge base and compose answers from it, then show both the AI-generated text and the original source article side-by-side so the support agent can verify the model's reasoning before sending it to the customer

normativehigh valueestablishednovelty 1/4durability 4/4· Frugal (AI expert guest)

“and side by side with that here is the the original Source article so that the customer support trap can like look at them side by side and see if it goes up yeah and see if it adds up so there's an angle like where the the AI explains like it's epistemological basis for how it concludes this”

0.74

Learning loops where human feedback on AI suggestions is used to improve the model are valuable in principle but operationally complex: they require ongoing monitoring, retraining infrastructure, and careful versioning—sometimes it's simpler to lock models at a stable point

normativehigh valueestablishednovelty 1/4durability 4/4· Frugal (AI expert guest)

“there's a lot of trade like it's very easy to say oh we want a system that will like learn in production but then it's like okay who has to maintain that who has to be bugged out sometimes it's easier and to to get it to like a stable stage and then like lock it where it just you know so it depends”

0.74

The core principle for deploying generative tech in support is that it must reduce work and stress on agents without introducing new risks or annoying them with incorrect suggestions, and optimization metrics must reflect this multi-dimensional goal

normativehigh valueestablishednovelty 1/4durability 4/4· Dez

“we do believe that this sort of generative Tech it needs to be shaped it needs to be made good so that it's not annoying so that it's not giving you wrong things misleading you certainly never pushing more work or more stress on you than than you would without it we do believe that its time is coming and we're trying to figure out like the best ways to uh to really make people more efficient and and you know yeah to to leverage it to put it in a production setting that actually works for people”

0.74

A critical danger in human-AI collaboration is that overworked humans can over-trust appealing but incorrect AI suggestions and fail to do due diligence, a phenomenon called 'normalization of deviance' documented in nuclear reactor disasters

causalhigh valueestablishednovelty 1/4durability 4/4· Dez (Intercom co-founder)

“you've got to be ready to take over at any point but there may be areas in that regime where like the system as a whole with the human is actually worse than just people actually over trust people can over trust and like you know what do they call it like these not desensitization like normalization of deviance like yeah you know people study this in the context of like nuclear reactor disasters and stuff”

0.69

ChatGPT and GPT-3.5 cannot be directly deployed in domains where high accuracy is required (medical diagnosis, legal analysis) because they produce well-written, believable-sounding but potentially incorrect answers with unknowable accuracy rates, creating significant risk if used without human verification

causalhigh valueestablishednovelty 1/4durability 3/4· Frugal (AI expert guest)

“it's not going to directly um you can't deploy it directly today in a lot of areas where there's a definitive right answer especially if the risk of giving the the wrong answer is pretty high so like you don't want this thing consuming medical records and spitting at diagnoses because I can guarantee you the diagnosis will be really well written really believable sounding to a lay person”

0.69

For a long time, it was assumed AI would struggle with semantic understanding and linguistic reasoning tasks like understanding pronoun references and semantic modifiers that change meaning—but ChatGPT now handles these well in many domains, making it difficult to claim 'the model just does pattern matching' without addressing what pattern it's actually matching

factualhigh valueestablishednovelty 1/4durability 3/4· Frugal (AI expert guest)

“the the the tractor went down the road and turned into a field right like please explain what happened in that field you know yeah and computer is what we're bad at that historically you know and uh the magic tractor went down there Road and turned into a field you know it's like modifier like that changes that changes the meaning it's getting really good at that in some domains”

0.69

Intercom's Resolution Boss product uses medium-scale language models (not GPT-scale models) with embeddings and is designed with a conscious constraint: it never outputs anything not explicitly curated by customers, preventing hallucinations but limiting capability—a conservative, production-safe approach to generative AI in customer support

factualhigh valueestablishednovelty 1/4durability 3/4· Frugal (AI expert guest)

“we have resolution boss we have it in production and it's you know it's not using language models that are as large as maybe medium language models um you know embeddings and so on and you know it it gets very good accuracy at the sort of thing it does and it's designed to you know the customer has to go and like create a specific answer it will never we made a conscious design decision very early on yeah that it would never say anything that hasn't been like explicitly curated by by the team”

0.69

The ability to play with ChatGPT without friction (no credit card needed) matters more for adoption and policy response than the marginal improvement in underlying capabilities, because friction matters more than absolute quality for network effects

causalhigh valueestablishednovelty 1/4durability 3/4· Frugal

“I don't underestimate the ability of low friction being able to like go in play with something everyone can do that and I think it's like a Sputnik moment which is like people who kind of look at this and go wow something's arriving here”

0.63

ChatGPT passes the first-stage coding interview screen (array-based algorithmic questions) in a way that suggests it could perform at a senior engineer level on whiteboard problems, demonstrating a qualitative leap in programming capability compared to previous AI systems

factualhigh valueestablishednovelty 2/4durability 2/4· Dez (Intercom co-founder)

“if you ask it to write you like a function even if you'd like and you made up out of C add a sample one like it might give you totally the wrong answer yeah but the chances of it giving you something useful have like gone way up”

0.63

Intercom's existing 'smart replies' feature demonstrates the success of constrained AI suggestion: automatically suggesting greetings (easy to detect context, high value, low friction) has been well-received and drives adoption, showing that incremental augmentation works better than full automation

factualhigh valueestablishednovelty 0/4durability 3/4· Frugal (AI expert guest)

“smart replies we started out with like greetings it was just such an obvious thing to do we can very easily tell when you're probably going to want to greeting uh you're coming to your conversations new conversation no one said anything to the end user before it's very obvious and I wasn't low-hanging piece of fruit people really liked the user interface like you just press tab yeah boom it's done and it's easy it's low friction”

0.63

Computer programmers spend significant time not writing code from scratch but searching for existing solutions (Stack Overflow, Google), copy-pasting and modifying them—a workflow where Copilot can intervene, automating the search step while programmers verify and modify

factualhigh valueestablishednovelty 0/4durability 3/4· Frugal (AI expert guest)

“a lot of the times like if you look at the workflow of a computer programmer a lot of the time it's like ah I know how to do this but I don't remember exactly how to use this live I'm going to Google for it I'm going to go to stack Overflow I'm going to copy paste the answer number three from stock overflow put that back in change it a bit”

0.60

Chain-of-thought prompting—prompting a model to 'think step by step'—can significantly improve accuracy on tasks like math problems by forcing the model to output intermediate reasoning, reducing the compounding effect of single-step errors

factualhigh valueestablishednovelty 0/4durability 4/4· Frugal (AI expert guest)

“people discovered that these models uh while they would get things wrong a lot you could prompt them with and the classic one is let's think step by step so you had a model you could ask it like a simple math question... these things struggle with with basic maths and and so like it would often like get things like that wrong but you could say something like let's think step by step that kind of like forced it to Output its reasoning step by step away like along the way and then accuracy rates went up”

0.60

The Transformer architecture, introduced by Google in the paper 'Attention Is All You Need,' represents a breakthrough in sequence modeling by learning which parts of a sentence to attend to, enabling faster training, more efficient scaling, and superior performance compared to LSTMs and previous sequence models

factualhigh valueestablishednovelty 0/4durability 4/4· Frugal (AI expert guest)

“Google published this paper you know attention is all you need and a pretty revolutionary paper with a pretty big thesis which was like hey um instead of like these sort of traditional sequence models here's a new way of doing as a new model which they call the Transformer model or the Transformer architecture”

0.60

Hallucination—the phenomenon where language models generate plausible-sounding false information—is a significant and persistent problem in current models, but accuracy has also improved dramatically compared to one and three years ago, meaning we should track both the improvement trajectory and the remaining error rate

factualhigh valueestablishednovelty 1/4durability 2/4· Frugal (AI expert guest)

“the accuracy has gone up like you know this versus where state of the art was a year ago versus where it was three years ago like it's absolutely better at giving you the right answer a lot of the time yeah”

0.56

ChatGPT capability represents as significant a change as the internet itself, and the total trajectory of improvement in capability over the past year and likely next 1-2 years represents a potential paradigm shift in technology, though significant work remains in productization and constraining models to specific domains

forecasthigh valuecontestednovelty 2/4durability 2/4· Frugal (AI expert guest)

“I feel that like the change and capability that we've seen over the last kind of like year and certainly if you extrapolate forward another year or two I personally feel that's as big as the internet I think that's gonna like the potential and we're gonna have to figure out how to productize these things”

0.56

The domains most likely to benefit from generative AI are those where verification is cheap (you can easily check if an answer is right/wrong) OR where correctness doesn't matter (creative tasks), AND where creating answers is laborious or complex—a pattern similar to computational complexity classes where verification is easy but generation is hard

definitionhigh valuespeaker onlynovelty 2/4durability 4/4· Frugal (AI expert guest)

“anything where there's um like structure in the task and where it is like a human approver who's able to discern whether an answer is right or wrong um this is gonna seem like weird intuition but you know in computer science or in cryptography we pay attention to the certain types of problems where you know it's easy to verify an answer is correct but hard to go and like find that answer yeah problems that P equals MP”

0.52

Humans judge creative outputs based primarily on plausibility at first glance using System 1 (intuitive) thinking rather than deliberate System 2 (analytical) reasoning, and ChatGPT exploits this by generating outputs that appear correct on surface inspection even when factually wrong or logically flawed upon scrutiny

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Frugal (AI expert guest)

“I think you know it's like people talk about like humans and human brains and like is this model of like you know system one and system two we have this intuitive part it's like really good at like recognizing patterns and then we have the kind of logical analytical reasoning part that's slower and more precise yeah and like this thing seems like it's it's very good at that sort of intuitive piece and it's very good at fooling our intuitive piece”

0.52

The potential transformative value of ChatGPT may not be in automation itself but in forcing honest conversations about which work has real value versus which is performative 'work work,' potentially leading to elimination of busywork and revealing the kernel of genuinely valuable activities

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Dez (Intercom co-founder)

“it's possible this is a good thing it's possible that like you know [ __ ] jobs like things that like okay the person doing it feels has got no value and so like this performance review is just work work so I'm gonna pan that off to GPT and then like you know after a while everyone kind of realizes that's what's happening and pressing the other side well I'm gonna hand it off the GPT to analyze it and like maybe then we can have an honest conversation about like hey what's the kernel that's actually really valuable”

0.52

Medical diagnosis is a domain where high hallucination tolerance might eventually be acceptable if accuracy improves sufficiently (from 70% to 99% to 99.9%), because diagnostic workflows include verification steps (tests) that can catch and correct errors, similar to how programming works with code review

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Frugal (AI expert guest)

“if we assume like let's be like pessimistic and say like giving it like a five paragraph description of a patient it can give you a 70 accurate diagnosis uh immediately um and in most of those diagnoses questions there's some quick test that can then verify whether or not that's true like as in sounds like you have blah here's the quick test for blah turns out as rock turns out I was wrong that's still a massive like productivity step change”

0.52

The 'dark matter' risk of deploying large language models in production is that the system may give wrong answers to questions that no one observes or analyzes, creating a hidden error class where neither the company nor the customer realizes a problem occurred

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Dez

“the risk there is like the they'll be like kind of almost like a dark matter like an unknown unknown so if someone goes to your business and asks the question that you didn't see because GPT dealt with it gave it the wrong answer and the customer goes off and does the wrong thing no one actually knows what's happened there if you know what I mean except for the bot and the bot doesn't even know it's wrong because it doesn't know if it's boofing or not”

0.52

Language model natural language processing is beginning to resemble the AlphaGo paradigm of self-play and iterative improvement, suggesting that the limits of these systems are harder to define than previously thought and that continued scaling and iteration may produce further unexpected breakthroughs

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Frugal (AI expert guest)

“could you combine that with the self-improvement thing you know I think this is a very interesting World here where you know um language models in NLP is starting to look a bit more like the alphago world where and self-play and improved and so yeah I think it's uh it's very exciting times and we don't really it's very hard to say what the limits are here”

0.52

The excitement about ChatGPT among non-experts is often rooted in the metaphor of a 'talking dog'—the remarkable fact of linguistic capability itself, overshadowing legitimate concerns about accuracy. The hallucination problem is real but shouldn't obscure the genuine breakthrough in language understanding

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Frugal (AI expert guest)

“this metaphor I don't remember who who kind of came up with it but it's a metaphor out there on the internet of like you know there's a talking dog and someone tells you they want you to go meet their new talking dog and you're like dogs can't talk yeah you get to the dog and the dog has a conversation with you and everyone starts talking about how the dog's grammar isn't very good very important yeah don't lose fat out of the fact the dog is talking it's grammar the hallucinations thing for me is is like that”

0.52

Unlike previous machine learning breakthroughs, modern large language models resist the pattern of 'looking magical until you understand how they work' because the internal mechanisms remain extremely difficult to understand even with deep investigation

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Frugal

“it's very hard to say oh like you know up until about two three years ago and whenever I saw a new machine Learning System whenever we like really got into it like it always looked magical and amazing at the start whenever you got into it you got underneath the hood you're like oh it's just logistic regression or like now I understand that it's much less impressive and um it's doing something extremely basic and I'm struggling to do that here maybe that's just because it's so hard to understand but um the complexity of the model but I think I think you know these things are I feel like qualitatively different capabilities than we've had before”

0.47

Despite risks of over-trust and system degradation, deploying language models for healthcare triage in regions without doctor access might still be valuable on net, because the alternative is no medical guidance at all, making the comparison not just 'AI + doctor' vs 'doctor alone' but 'AI + whatever medical resources exist locally' vs 'no AI'

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Frugal (AI expert guest)

“when we're ever thinking about like things like medical things you know some portion of the world doesn't have access to a doctor yeah right I'm like maybe there's some future thing where you know is is like it's the classic thing it's like looking up medical things in the Internet it's like well if you can't afford or you're in a country doesn't have a doctor so so I don't know where where to draw that where to draw that boundary”

0.45

The cognitive load bottleneck in customer support is not just writing responses but finding the right article, reading it, understanding the answer, copy-pasting it, and reformatting it—multiplying the time of newer or part-time support agents

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Dez

“we obviously we study customer support flow very carefully and very closely and we absolutely have some support agents where it's like okay got the question they have to go and find an article themselves some expert ones they know it they're instantly there and you know exactly where to go maybe they've got a macro that that does it yeah but then maybe someone is like newer in the company and they're still being trained in or maybe it's only part of their job and they have to go and they've got to find the article themselves and then they gotta read it and they got to check the answer and then they gotta copy paste it and then they gotta reformat it and maybe so maybe it is like maybe there's like a you know a multiple productivity like boost and we can make someone twice as efficient or something”

0.45

The rapid evolution of language models means product decisions and roadmap predictions become outdated within weeks or months, so companies building on these technologies must be adaptable rather than committed to fixed plans

normativehigh valuespeaker onlynovelty 1/4durability 2/4· Dez (Intercom co-founder)

“this is being great we'll check in and I guess in two more weeks when the whole world's changed again yeah exactly but if not it could be a few months thanks very much by the time this is up on the web I'm sure it'll be out of date Mary look foolish but that's the nature of this being that's absolutely that's why you're working that's why we're working that's why it's exciting”

0.45

For customer support, there are several levels of augmentation: pre-filling the composer with suggested text, suggesting macros contextually, formatting/style buttons (make more formal, add apology), and flow-finding (understanding recurring patterns in support conversations)

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Dez

“we're looking at stuff like should we be suggesting those micros people proactively maybe it's not enough to maybe when we don't want to be pre-filling the composer maybe we want to like just show some macro suggestions that are contextual there's a lot of flows that are repetitive we've been working on things like flow finding like trying to like understand the kind of the common steps people will go through and you know but I mean I guess the big message is um we do believe that this sort of generative Tech it needs to be shaped it needs to be made good so that it's not annoying so that it's not giving you wrong things misleading you certainly never pushing more work or more stress on you than than you would without it”

0.41

OpenAI released ChatGPT as an open beta on November 30th via email invite, packaging GPT-3.5 in a more accessible way without requiring credit card entry, which caused it to go viral and achieve massive public adoption

factualestablishednovelty 0/4durability 4/4· Dez (Intercom co-founder)

“you didn't need to put a credit card into it and so I think everyone just saw that word has been a huge change in capability here recently and it went viral it went wild”

0.38

Prompt engineering—the practice of carefully crafting input prompts to improve model outputs—is becoming a real discipline with practitioners developing expertise, suggesting that how you ask the model matters significantly for performance

factualestablishednovelty 1/4durability 2/4· Dez (Intercom co-founder)

“we're joking and machine learning team and intercom is going to prompt engineering team and we're kind of joking about that as we play with it and but there are people out there who you know at some point that laughter stops I'm sure yeah yeah yeah yeah who you know who who really swept the prompts and gotten really good at prompted engineering so it's a real thing”

0.30

ChatGPT can perform tasks with immediate zero-second latency and handle complex multi-step requests like translating content into different languages, adjusting tone, and reformatting in real time without perceptible delay

factualestablishednovelty 1/4durability 2/4· Dez (Intercom co-founder)

“you can double back it and say no give me that in the style of like a 1940s gangster I never say it in German I know and it translates to German into Spanish but also add in like more you know more anger or whatever and it does all these things like both immediately like as in you know pretty much zero second delay”