YouTube13m· Mar 2024· cataloged

What's next for AI agentic workflows ft. Andrew Ng of AI Fund


What this covers

This is a 14-minute talk by Andrew Ng at Sequoia Capital's AI Ascent, with a brief introduction from an event host. Ng starts from how most people use language models today: type a prompt, get an answer in one pass, which he likens to asking someone to write an essay without ever pressing backspace. He contrasts this with an agentic workflow, in which the model outlines, researches, drafts, critiques its own work, and revises. The evidence he offers is the HumanEval coding benchmark, where GPT-3.5 scores 48% and GPT-4 scores 67% with zero-shot prompting. GPT-3.5 inside an agentic loop surpasses GPT-4, and GPT-4 inside one does better still.

The middle of the talk walks through four design patterns. Reflection has a model check its own code, or explain why a unit test failed, and then produce a better version. Tool use covers web search, code execution, and domain-specific tools, and he traces its early development to the computer vision community. Planning lets an agent choose its own sequence of steps and recover from failures. He illustrates this with an image-transformation task that chains several specialized models, and he admits that planning agents are finicky. Multi-agent collaboration is shown through ChatDev, an open-source system where role-playing agents (CEO, designer, engineer, tester) build software, and through multi-agent debate and simple coder-and-critic pairs.

The more contested claims come near the end. Ng suggests that agentic reasoning on earlier models may match what people are waiting for from GPT-5-class models on some applications. He also argues that fast token generation from a slightly weaker model may matter more than slow generation from a stronger one, because agents loop so many times. He names user impatience as a practical obstacle, since agentic tasks take minutes rather than the sub-second responses people expect from search. He mentions that he already delegates research to an agent in his own routine, and he frames the whole approach as one modest step on a long road toward AGI.

Sharpest takeaway

Andrew Ng argues that agentic AI workflows—where language models iteratively plan, reflect, and revise—dramatically outperform zero-shot prompting and represent a critical design pattern for building practical AI applications that will expand the range of tasks AI can perform.

  • GPT-3.5 wrapped in an agentic workflow outperforms GPT-4 using zero-shot prompting on coding benchmarks (HumanEval)
  • Four concrete design patterns (reflection, tool use, planning, multi-agent collaboration) are replicable and deliver productivity boosts when implemented
  • Agentic reasoning is a stepping stone toward AGI that requires patience for extended execution but enables autonomy and failure recovery

The claims · ranked26 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.69

Early work in tool use originated from the computer vision community because before large language models could process images, the only option was for LMs to generate function calls that could manipulate images (generate, detect objects, etc.).

causalhigh valueestablishednovelty 1/4durability 3/4· Andrew Ng

“it turns out that a lot of the early work in two use turned out to be in the computer vision Community because before large language models lm's you know they couldn't do anything with images so the only option was that the LM generate a function called that could manipulate an image like generate an image or do object detection or whatever so if you actually look at literature it's been interesting how much of the work um in two years seems like it originated from Vision because LMS would blind to images before you know gp4 and and and lava and so on”

0.69

On the HumanEval coding benchmark, GPT-3.5 with zero-shot prompting achieves 48% accuracy, GPT-4 achieves 67% accuracy, but GPT-3.5 wrapped in an agentic workflow actually outperforms GPT-4, and an agentic workflow around GPT-4 performs even better.

factualhigh valuecontestednovelty 3/4durability 2/4· Andrew Ng

“it turns out that if you use GPT 3.5 uh zero shot prompting it gets it 48% right uh gp4 way better 607 7% right but if you take an agentic workflow and wrap it around GPT 3.5 I say it actually does better than even gbd4 um and if you were to wrap this type of workflow around gb4 you know it it it also um does very well”

0.68

This agentic workflow result has significant consequences for how practitioners approach building AI applications, suggesting that architectural pattern and iteration strategy matter as much or more than raw model capability.

causalhigh valuecontestednovelty 2/4durability 3/4· Andrew Ng

“I think this has and this means that this has signant consequences fighting how we all approach building applications”

0.68

Agentic workflows with language models involve iterative processes where the model writes an outline, conducts web research, drafts content, reviews its own work, identifies revision needs, and revises iteratively—delivering remarkably better results than non-agentic approaches.

causalhigh valuecontestednovelty 2/4durability 3/4· Andrew Ng

“with an agentic workflow this is what it may look like have an AI have an LM say write an essay outline do you need to do any web research if so let's do that then write the first draft and then read your own first draft and think about what parts need revision and then revise your draft and you go on and on and so this workflow is much more iterative where you may have the L do some thinking um and then revise this article and then do some more thinking and iterate this through a number of times and what not many people appreciate is this delivers remarkably better results”

0.65

Generating more tokens quickly from a slightly lower-quality LM might deliver comparable results to generating fewer tokens slowly from a higher-quality LM in agentic workflows, suggesting quality-speed tradeoffs are different in iterative architectures.

causalhigh valuecontestednovelty 3/4durability 2/4· Andrew Ng

“I think that um generating more tokens really quickly from even a slightly lower quality LM might give good results compared to slower tokens from a better LM maybe it's a little bit controversial because it may let you go around this Loop a lot more times kind of like the results I showed with gbd3 and an agent architecture on the first slide”

0.65

The typical non-agentic workflow with language models involves typing a prompt and generating an answer in a single pass, analogous to asking a person to write an essay without ever using backspace.

definitionhigh valueestablishednovelty 1/4durability 3/4· Andrew Ng

“today the way most of us use Lish models is like this with a non- agentic workflow where you type a prompt and generates an answer and that's a bit like if you ask a person to write an essay on a topic and I say please sit down to the keyboard and just type the essay from start to finish without ever using backspace”

0.65

When an LM's code fails a unit test, prompting the LM to explain why and suggesting changes enables it to generate improved versions (version 3) that work better, creating a feedback loop.

causalhigh valueestablishednovelty 1/4durability 3/4· Andrew Ng

“if you let it run unit test if it fails a unit test then he why do you fail the unit test have that conversation and be able to figure out fail the unit test so you should try changing something and come up with V3”

0.61

The HumanEval benchmark tests coding problems such as given a list of integers, return the sum of all elements at even positions, which humans would solve through an iterative drafting and testing process rather than writing code in a single pass.

factualhigh valueestablishednovelty 1/4durability 3/4· Andrew Ng

“this says coding problems like given the nonent list of integers return the sum of all the all elements are an even positions and it turns out the answer is you code snipper like that so today lot of us will use zero shot prompting meaning we tell the AI write the code and have it run on the first spot like who codes like that no human codes like that just type out the code and run it maybe you do I can't do that”

0.60

ChatDev is an open-source multi-agent system that runs locally where agents with different organizational roles (CEO, designer, product manager, engineer, tester) collaborate to develop software, and when given a task like 'develop a game', it spends several minutes writing code, testing, and iterating to generate surprisingly complex programs.

factualhigh valueestablishednovelty 1/4durability 2/4· Andrew Ng

“Chad Dev is open source it runs on my laptop and what Chad Dev doeses is example of a multi-agent system where you prompt one LM to sometimes act like the CEO of a software engine company sometimes Act designer sometime a product manager sometimes I a tester and this flock of agents that you built by prompting an LM to tell them you're now Co you're now software engineer they collaborate have an extended conversation so that if you tell it please develop a game develop a GOI game they'll actually spend you know a few minutes writing code testing it uh iterating and then generate a like surprisingly complex programs doesn't always work I've used it sometimes it doesn't work sometimes it's amazing”

0.60

Multi-agent debate where different agents (e.g., ChatGPT and Gemini) debate each other results in better overall performance than single-agent systems.

causalhigh valuecontestednovelty 2/4durability 2/4· Andrew Ng

“so just one of design pattern it turns out that multi-agent debate where you have different agents you know for example could be have ch GPT and Gemini debate each other that actually results in better performance as well so having multiple simulated air agents work together has been a powerful design pattern as well”

0.59

Planning algorithms in agentic systems enable AI agents to autonomously decide on sequences of actions, recover from failures, and reroute around obstacles without human intervention.

factualhigh valuecontestednovelty 1/4durability 3/4· Andrew Ng

“and then planning you know for those of you that have not yet played a lot with planning algorithms I I feel like a lot of people talk about the chat GPT moment where you're wow never seen anything like this I think if not used planning alums many people will have a kind of a AI agent wow I couldn't imagine the AI agent doing this I've run live demos where something failed and the AI agent rerouted around the failures”

0.56

Language models today can use tools including web search, code generation and execution, and various domain-specific tools for analysis, information gathering, and personal productivity.

factualhigh valueestablishednovelty 1/4durability 2/4· Andrew Ng

“the second design pattern is to use many of where already have seen you know LM based systems uh uh using tools on the left is a screenshot from um co-pilot on the right is something that I kind of extracted from uh gp4 but you know LM today if you ask it what's the best coffee maker web search for some problems um will generate code and run code um and it turns out that there are a lot of different tools that many different people are using for analysis for gathering information for taking action for personal productivity”

0.52

Multi-agent systems can be evolved from single agentic agents by creating two agents with different prompting roles—a coder agent and a critic agent—who review each other's work, which is easy to implement and gives significant performance boosts.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Andrew Ng

“one Natural Evolution of this idea is instead of a single code agent you can can have two agents where one is a coder agent and the second is a Critic agent and these could be the same base LM model but that you prompt in different ways where you say one your expert coder right code the other one say your expert code review to review this code and this Tye of workflow is actually pretty easy to implement I think it's such a very general purpose technology for a lot of workflows this would give you a significant boost in in the performance of LMS”

0.50

Reflection design pattern involves prompting an LM to check its own generated code for correctness, efficiency, and good construction practices, allowing it to identify and fix bugs in its own output.

definitionhigh valuespeaker onlynovelty 2/4durability 3/4· Andrew Ng

“reflection here's an example let's say ask a system please write code for me for a given task then we have a coder agent just an LM that you prompt to write code to say you def du task write a function like that um an example of self-reflection would be if you then prompt the LM with something like this here's code intended for a toas and just give it back the exact same code that they just generated and then say check the code carefully for correctness sound efficiency good construction CRI just write prompt like that it turns out the same l that you prompted to write the code may be able to spot problems like this bug in line Five May fix it by blah blah blah”

0.50

Fast token generation is important for agentic workflows because agents iterate repeatedly, with the LM generating tokens for the agent to read, and the ability to generate tokens much faster than humans read enables more efficient looping.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Andrew Ng

“one other important Trend fast token generation is important because with these agented workflows we're iterating over and over so the LM is generating tokens for the elm to read so be able to generate tokens way faster than any human to read is fantastic and I think that um generating more tokens really quickly from even a slightly lower quality LM might give good results compared to slower tokens from a better LM”

0.49

Multi-agent collaboration can be implemented via multi-agent systems where a single language model is prompted to adopt different roles (CEO, designer, product manager, engineer, tester) and these agents collaborate through extended conversation.

definitionhigh valuespeaker onlynovelty 2/4durability 2/4· Andrew Ng

“the final design pattern multi- Asian collaboration this is one of those funny things but uh um it works much better than you might think uh uh but on the left is a screenshot from a paper called um chat Dev uh which is completely open which actually open source many of you saw the you know flashy social media announcements of demo of a Devon uh uh Chad Dev is open source it runs on my laptop and what Chad Dev doeses is example of a multi-agent system where you prompt one LM to sometimes act like the CEO of a software engine company sometimes Act designer sometime a product manager sometimes I a tester and this flock of agents that you built by prompting an LM to tell them you're now Co you're now software engineer they collaborate have an extended conversation”

0.49

There are four broad design patterns within agentic AI systems: reflection (robust and reliable), tool use (expanding model capabilities), planning (emerging and variable but impressive), and multi-agent collaboration (emerging).

definitionhigh valuespeaker onlynovelty 2/4durability 2/4· Andrew Ng

“I want to be a bit concrete and share of you um the broad design patterns I'm seeing in agents it's a very messy chaotic space tons of research tons of Open Source there's a lot going on but I try to categorize um bit more concretely what's going on agents reflection is a tool that I think many of us should just use it just works uh to use I think it's more widely appreciated but actually works pretty well I think of these as pretty robust technology when I use them I can you know almost always get them to work well um planning and multi-agent collaboration I think is more emerging”

0.49

If practitioners focus on running applications on future models (GPT-5, Claude 4, GPT-5, Gemini 2.0) via zero-shot prompting, they may be able to achieve comparable performance on some applications using agentic reasoning on earlier models.

causalhigh valuespeaker onlynovelty 2/4durability 2/4· Andrew Ng

“and I think that um if you're looking forward to running your thing on gp5 zero shot you know you mayble to get closer to that level performance on some applications than you might think with agenting reasoning um but on an early model I think I I I I think this is an important Trend”

0.47

A critical challenge with agentic workflows is human impatience with response latency: modern users expect instant responses (sub-second, like web search), but agentic workflows require patience for extended execution times (minutes to hours), which conflicts with human nature and management instincts.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Andrew Ng

“one thing that it's actually difficult people to get used to is when we prompt an LM we want to response right away um in fact a decade ago when I was you know having discussions around at at at Google on um it called a big box search we type a long prompt one of the reasons you know I failed to push successfully for that was because when you do a web search you one of responds back in half a second right that's just human nature we like that instant grab instant feedback but for a lot of the agent workflows um I think we'll need to learn to dedicate the toss and AI agent and patiently wait minutes maybe even hours uh to for a response but just like I've seen a lot of novice managers delegate something to someone and then check in 5 minutes later right and that's not productive um I think we need to it be difficult we need to do that with some of our AI agents as well”

0.46

In a complex planning example, an AI agent can autonomously decompose an image transformation task (given a girl reading a book, create an image of a boy reading a book in the same pose) by: determining the boy's pose, finding a pose extraction model, finding a pose synthesis model, converting the image to text, and applying text-to-speech—coordinating multiple specialized models sequentially.

factualhigh valuespeaker onlynovelty 2/4durability 2/4· Andrew Ng

“you say this general image where the girls read where a girl is reading a book and it posts the same as a boy in the image example. jpack and please subscribe the new image for your voice so give an example like this um today we have ai agents who can kind of decide first thing I need to do is determine the post of the boy um then you know find the right model maybe on hugging face to extract the post then next need to find a post image model to synthesize a picture of a of a girl of as following the instructions then use image to text to and then finally use text of speech”

0.45

Using the four design patterns (reflection, tool use, planning, multi-agent collaboration) in practical work can deliver quick productivity boosts.

normativehigh valuespeaker onlynovelty 1/4durability 2/4· Andrew Ng

“just to summarize I think these are the these are the the the uh patterns of seen and I think that if we were to um use these uh uh patterns you know in our work a lot of us can get a prity boost quite quickly”

0.43

The path to AGI is a journey rather than a destination, and agentic workflows represent a small but meaningful step forward on this very long path.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Andrew Ng

“honestly the path to AGI feels like a journey rather than a destination but I think this typ of agent workflows could help us take a small step forward on this very long journey”

0.42

Planning agents are not yet fully reliable or predictable—they are 'finicky' and don't always work—but when they do work, the results are remarkable and recovery from earlier failures is possible through agentic loops.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Andrew Ng

“today we actually have agents that I don't want to say they work reliably you know they're kind of finicky they don't always work but when it works is actually pretty amazing but with agentic loops sometimes you can recover from earlier failures as well”

0.42

Andrew Ng personally uses research agents as part of his workflow where instead of conducting time-consuming web searches himself, he delegates the task to an agent, waits a few minutes, and reviews the results—even though the agent sometimes fails, it is already part of his routine.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Andrew Ng

“so I find myself already using research agents for some of my work where one of piece of research but I don't feel like you know Googling myself and spend a long time I should send to the research agent come back in a few minutes and see what it's come up with and and it sometimes works sometimes doesn't right but that's already a part of my personal workflow”

0.40

Agentic reasoning design patterns will expand the set of tasks that AI can perform dramatically during the year, becoming an important capability lever.

forecasthigh valuespeaker onlynovelty 2/4durability 1/4· Andrew Ng

“I think that um agentic reasoning design patterns are going to be important uh this is my small slide I expect that the set of T AI could do will expand dramatically this year uh because of agentic workflows”

0.29

Andrew Ng is a famous computer science professor at Stanford who was early in neural networks development with GPUs, creator of Coursera and DeepLearning.ai, and founder and early lead of Google Brain.

factualestablishednovelty 0/4durability 3/4· Unknown Speaker (event host/emcee)

“all of you uh know Andreu in as a famous uh computer science professor at Stanford was really early on in the development of neural networks with gpus of course a creator of corsera and popular courses like deeplearning.ai also the founder and Creator uh and early lead of Google brain”