What this covers

This is a roughly half-hour Lightcone conversation among the YC partners, Garry, Harj, Diana and Jared, drawing on what they've seen from hundreds of founders building on LLMs. It starts from a real production prompt, the one Parahelp published, and uses it to look at how serious prompts are structured. The layers it describes are a system prompt that defines how the company operates in a customer-agnostic way, a developer prompt that adds each customer's context, and a user prompt for end-user products. From there the talk moves to the practice around prompts: metaprompting, worked examples, longer-prompt tricks, and evals.

The middle of the episode is the most practical. One recurring problem is that models want to be helpful and will hallucinate a formatted answer rather than admit missing context. The remedies discussed are an explicit escape hatch that tells the model to refuse and ask, and a debug info field that lets the model complain about underspecified instructions, which then becomes a to-do list for the developer. Jared describes prompt folding, where a classifier prompt generates specialized prompts from the previous query. The group also covers scoring with a rubric, which needs detailed guidance for separating adjacent scores and still always has exceptions. Metaprompting is likened to managing a person, and to the manufacturing principle of kaizen, where the people doing the work improve the process.

The second half turns to business. Evals are called the real moat, more valuable than the prompts they justify, and they can only be gathered by sitting beside domain experts such as a regional tractor sales manager. That leads to the forward-deployed engineer model, which began at Palantir. Engineers rather than salespeople embed with clients and build working software quickly. The speakers argue vertical AI founders can beat incumbents like Salesforce, Oracle and Booz Allen this way, though they note the tension between customization and turning into a consultancy. The episode also touches on differences in model personalities and on founders with unusual domain knowledge, with Flexport's Ryan Peterson as the example.

Sharpest takeaway

Prompt engineering and metaprompting have become critical competitive tools for AI startups, requiring founders to master both technical prompt design and deep domain knowledge of their customers' workflows through a 'forward deployed engineer' model similar to Palantir's approach.

  • Best prompts use hierarchical structures (system/developer/user), XML formatting, and escape hatches to prevent hallucinations
  • Evals (evaluation datasets) are more valuable than prompts themselves because they encode customer domain knowledge
  • Founders must personally embed with customers to capture their workflows and preferences, then rapidly iterate demos to close enterprise deals

The claims · ranked61 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.69

The 'forward-deployed engineer' role originated at Palantir, where engineers (not salespeople) were stationed with clients to understand their workflows and directly build solutions, rather than relying on sales processes and contracts.

factualhigh valueestablishednovelty 1/4durability 3/4· Gary

“the forward deployed engineer title was specifically how do you sit next to literally the FBI agent who's um investigating domestic terrorism. How do you sit right next to them in their actual office and see what does the case coming in look like? What are all the steps?”

0.69

Palantir's founding insight was that the world's top technologists almost never sat in rooms with Fortune 500 companies or government agencies, yet those organizations faced multi-billion-dollar problems around data understanding that required world-class engineers.

factualhigh valueestablishednovelty 1/4durability 3/4· Gary

“One of the key recognitions that Peter Teal and Alex Karp and Stefan Cohen and Joe Lansdale, Nathan Gettings, like the original founders of Palunteer had was that uh go into anywhere in the Fortune 500, go into any government agency in the world, including the United States, and nobody who understands computer science and technology at the level that you at the highest possible level would ever even be in that room.”

0.69

Palantir's core business was helping organizations make sense of their massive datasets and find signals in noise, because enterprises are drowning in data but have no understanding of what's happening—a problem that has only become more acute over 20+ years and is increasingly addressable with LLMs.

factualhigh valueestablishednovelty 1/4durability 3/4· Gary

“you can go and find the world's best technologists who know how to write software to actually make sense of the world. You know, you have these pabytes of data and you don't know how do you find the needle in the haststack. Um and you know the wild thing is going on uh something like 20 22 years later it's only become more true that we have more and more data and we have less and less of an understanding of what's going on and uh it's no mistake that actually now that we have LLMs like we actually it is becoming much more tractable”

0.65

LLMs want to be helpful so strongly that if asked for output in a specific format without enough context, they will hallucinate rather than admit uncertainty; the solution is to provide an explicit 'escape hatch' telling the LLM to refuse and ask for clarification rather than guess.

causalhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“the model really wants to actually help you so much that if you just tell it give me back output in this particular format even if it doesn't quite have the information it needs it'll actually just tell you what it thinks you want to hear and it's literally a hallucination. So, one thing they discovered is that you actually have to give the LLM's a real escape hatch. You need to tell it if you do not have enough information to say yes or no or make a determination, don't just make it up. Stop and ask me.”

0.65

Best practice for scoring output from LLMs is to provide detailed rubrics explaining how to differentiate between adjacent scores (e.g., 80 vs 90), though these rubrics are never perfect and always have exceptions.

normativehigh valueestablishednovelty 1/4durability 3/4· Harj (Unknown Last Name - YC team)

“it's certainly best practice to give um LLM's rubrics, especially if you want to get a numerical score as the output. You want to give it a rubric to help it understand like how should I think through and what's like a 80 versus a 90. But these rubrics are never perfect. there's often always exceptions”

0.63

Kaizen, the manufacturing principle that created Japanese automotive excellence in the 1990s, works by making the people doing the actual work responsible for improving the process, and metaprompting is the LLM version of this principle—the people improving the prompt are the ones actually using it.

causalhigh valuespeaker onlynovelty 3/4durability 4/4· Gary Tan

“there's this aspect of Kaizen, you know, this um this manufacturing technique that created really really good cars for Japan in the '90s. Uh and that principle actually says that the people who are the absolute best at improving the process are the people actually doing it. That's literally why uh Japanese cars got so good in the '90s. And that's metaprompting to me. So, I don't know. It's a brave new world. We're sort of in this new moment.”

0.59

For YC vertical AI and SaaS companies, evals can only be obtained by sitting literally side-by-side with people doing the actual knowledge work (like a regional tractor sales manager), understanding their reward function and what outcomes they care about, then codifying those specific outcomes into evals.

causalhigh valuespeaker onlynovelty 2/4durability 4/4· Gary Tan

“you can't get the eval unless you sitting literally side by side with people who are doing X Y or Z knowledge work. you know, you need to sit next to the tractor sales regional manager and understand, well, you know, this person cares, you know, this is how they get promoted. This is what they care about. This is that person's reward function. And then you know what you're doing is taking these in-person interactions sitting next to someone in Nebraska and then going back to your computer and codifying it into uh very specific evals like this particular user wants this outcome after they you know after this invoice comes in we have to decide whether we're going to honor the you the warranty on this tractor.”

0.57

Kaizen (the Japanese manufacturing technique that made Japanese cars exceptional in the 1990s) operates on the principle that the people actually doing the work are the best at improving the process, and metaprompting is the LLM equivalent of this principle applied to AI development.

definitionhigh valuespeaker onlynovelty 3/4durability 3/4· Gary Tan

“like there's this aspect of Kaizen, you know, this um this manufacturing technique that created really really good cars for Japan in the '90s. Uh and that principle actually says that the people who are the absolute best at improving the process are the people actually doing it. That's literally why uh Japanese cars got so good in the '90s. And that's metaprompting to me.”

0.57

Prompt architecture is emerging with three layers: system prompt (defines the high-level API of how the company operates, customer-agnostic), developer prompt (adds context of specific customer implementations), and user prompt (for end-user facing products), with each layer containing different levels of customization.

definitionhigh valuespeaker onlynovelty 3/4durability 3/4· Gary Tan

“So this is concept of uh defining the prompt in the system prompt. Then there's a de developer prompt and then there's a user prompt. So what this mean is uh the system prompt is basically almost like defining uh sort of the highle API of how your company operates. In this case the example of parhel is very much a system prompt. There's nothing specific about the customer. And then as they add specific instances of that API and calling it then they stuff all that in into more the developer prompt which is not shown here and that's adds all the context of let's say working with perplexity there's certain ways of how you handle rack questions as opposed to working with bold is very different right and then I don't think parhelp has a user prompt because their product is not consumed directly by an end user but a end user prompt could be more like replet or a zero right where users need to type is like generate me a site that that has these buttons this and that that goes all in the user prompt.”

0.57

Prompt folding is a metaprompting technique where a classifier prompt dynamically generates specialized prompts based on the previous query, allowing the LLM to improve its own prompts rather than requiring manual rewriting.

definitionhigh valuespeaker onlynovelty 3/4durability 3/4· Jared

“So you know basically one prompt can dynamically generate better versions of itself. So a good example of that is a classifier prompt that generates a specialized prompt based on the previous query. And so you can actually go in take uh the existing prompt that you have and actually feed it more examples where maybe the prompt failed or didn't quite do what you wanted and you can actually instead of you having to go and rewrite the prompt, you just put it into um you know the raw LLM and say help me make this prompt better.”

0.57

YC internal work with agents developed an alternative escape hatch approach: giving the LLM a 'debug_info' parameter in the response format that allows it to complain to the developer about confusing or underspecified information, and this debug_info becomes a to-do list for the agent developer to fix.

factualhigh valuespeaker onlynovelty 3/4durability 3/4· Jared

“We came up with a different way which is in the response format to give it the ability to have part of the response be essentially a complaint to you the developer that like you have given it confusing or underspecified information and it doesn't know what to do. And then the nice thing about that is that we just run your LLM like in production with real hoser data and then you can go back and you can look at the outputs that it has given you in that like output parameter. Um we we call it debug info internally. So like we have this like debug info parameter where it's basically reporting to us things that we need to fix about it and it literally ends up being like a to-do list that you the agent developer has to do.”

0.56

The challenge for vertical AI agent companies is how to balance flexibility (allowing customer-specific customization) against operational scaling (avoiding becoming a consulting company that builds a new prompt for every customer).

factualhigh valuespeaker onlynovelty 2/4durability 4/4· Jared

“has a really interesting thing that I see the vertical AI agent companies talking about a lot which is like how do you have enough flexibility to make special purpose logic without turning into a consulting company where you're building like a new prompt for for for every customer.”

0.56

The best founders in the world combine being cracked engineers/technologists with deep understanding of some part of the world that very few people understand, and the weirder that part of the world, the greater the opportunity (example: Ryan Peterson from Flexport was the third biggest importer of medical hot tubs for an entire year).

causalhigh valuespeaker onlynovelty 2/4durability 4/4· Gary Tan

“the best founders in the world they're you know sort of really great cracked engineers and technologists and uh just really brilliant and then at the same time they have to understand some part of the world that very few people understand and then there's this little sliver that is you know uh the founder of a multi-billion dollar startup you know I think of Ryan Peterson from Flexport, you know, really really great person who understands how software is built, but then also I think he was the third biggest uh importer of medical hot tubs for an entire year like you know a decade ago. So you know the weirder that is the more of the world that you've seen that nobody else who's a technologist has seen uh the greater the opportunity actually.”

0.56

Palantir institutionalized and scaled forward deployment as a business process, which is probably why they were able to secure consistent seven, eight, and nine-figure contracts—not through salespeople building relationships, but through engineers delivering working software quickly based on observed needs.

causalhigh valuespeaker onlynovelty 2/4durability 4/4· Gary Tan

“I think what happened there, the reason why they were able to get these sort of seven and eight and now nine figure contracts very consistently is that uh instead of sending someone who's like hair and teeth and they're in there and you know, let's go to the let's go to the uh steakhouse. You know, it's all like relationship. and you'd have one meeting uh they would really like the salesperson and then through sheer force of personality you'd try to get them to give you a seven-figure contract and like the time scales on this would be you know 6 weeks 10 weeks 12 weeks like 5 years I don't know it's like and the software would never work uh whereas if you put an engineer in there and you give them uh you know Palunteer Foundry which is what they now call sort of their core uh data viz and data mining suites instead of the next meeting being reviewing 50 pages of you know sort of sales documentation or a contract or a spec or anything like that. It's literally like, "Okay, we built it." And then you're getting like real live feedback within days.”

0.56

The biggest opportunity for startup founders is to do what forward deployed engineers at Palantir do: beat established competitors (like Salesforce, Oracle, Booz Allen) by showing something they've never seen before—demonstrate software that makes the customer feel understood, heard, and seen.

normativehigh valuespeaker onlynovelty 2/4durability 4/4· Gary Tan

“That's honestly the biggest opportunity for startup founders. If startup founders can do that and uh that's what forward deployed engineers are sort of used to doing that's how you could beat a Salesforce or an Oracle or you know a Booze Allen or literally any company out there that has a big office and a big fancy you know you have big fancy salespeople with big strong handshakes and it's like how does a really good engineer with a weak handshake go in there and beat them? It's actually you show them something that they've never seen before and like make them feel super heard. You have to be super empathetic about it. Like you actually have to be a great designer and product person and then you know come back and you can just blow them away. Like the software is so powerful that you know the second you see something that you know makes you feel seen you want to buy it on the spot.”

0.55

Metaprompting—where one prompt dynamically generates better versions of itself—is turning out to be a very powerful tool that everyone in AI startups is now using, and it feels like learning how to manage a person because you need to communicate the things they need to know to make good decisions.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Gary Tan

“Metarprompting is turning out to be a very very powerful tool that everyone's using now. It kind of actually feels like coding in you know 1995 like the tools are not all the way there. We're you know in this new frontier. But personally it also kind of feels like learning how to manage a person where it's like how do I actually communicate uh you know the things that they need to know in order to make a good decision.”

0.52

A major startup opportunity exists in automating the creation of worked examples: using an agent to automatically extract the best examples from customer datasets and inject them into the prompt pipeline, reducing manual engineering.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“anyone who's done prompt engineering knows that the examples and worked examples are really important to improving the quality of the output. And so then if you take like power as an example, they really want good worked examples that are specific to each company. And so you can imagine that as they scale, you almost want that done automatically. Like in your dream world, what you want is just like a an agent itself that can pluck out the best examples from like the customer data set and then software that just like ingests that straight into like wherever it should belong in the pipeline”

0.52

Metaprompting—where one prompt dynamically generates better versions of itself—is a powerful technique; a practical example is a classifier prompt that generates specialized prompts based on previous queries, allowing iterative improvement without manual rewriting.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely Jared)

“one prompt can dynamically generate better versions of itself. So a good example of that is a classifier prompt that generates a specialized prompt based on the previous query. And so you can actually go in take uh the existing prompt that you have and actually feed it more examples where maybe the prompt failed or didn't quite do what you wanted and you can actually instead of you having to go and rewrite the prompt, you just put it into um you know the raw LLM and say help me make this prompt better.”

0.52

YC has developed an alternative escape hatch mechanism: including a 'debug_info' parameter in response format that allows LLMs to report confusing or underspecified information directly back to the developer, creating a prioritized to-do list of prompt improvements.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Jared

“We came up with a different way which is in the response format to give it the ability to have part of the response be essentially a complaint to you the developer that like you have given it confusing or underspecified information and it doesn't know what to do. And then the nice thing about that is that we just run your LLM like in production with real hoser data and then you can go back and you can look at the outputs that it has given you in that like output parameter. Um we call it debug info internally.”

0.52

The analogy of prompt engineering to coding in 1995 captures the frontier nature of the field: tools are incomplete, many things are unspecified, and developers are charting new territory without established best practices.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely Gary based on earlier pattern)

“the analogies I think even we use to discuss this is uh it's kind of like coding. It kind of actually feels like coding in, you know, 1995. Like the tools are not all the way there. There's a lot of stuff that's unspecified. We're, you know, in this new frontier.”

0.52

Great founders often have unusual domain expertise—sometimes called 'weird' knowledge—that few other technologists possess, like Ryan Peterson importing medical hot tubs before founding Flexport, which provides genuine competitive advantage.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Gary

“the best founders in the world they're you know sort of really great cracked engineers and technologists and uh just really brilliant and then at the same time they have to understand some part of the world that very few people understand and then there's this little sliver that is you know uh the founder of a multi-billion dollar startup you know I think of Ryan Peterson from Flexport, you know, really really great person who understands how software is built, but then also I think he was the third biggest uh importer of medical hot tubs for an entire year like you know a decade ago.”

0.52

Metaprompting is turning out to be a very powerful tool that everyone in AI startups is using now, and it feels like learning how to manage a person—communicating what they need to know to make good decisions.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely host/primary speaker)

“Metarprompting is turning out to be a very very powerful tool that everyone's using now. It kind of actually feels like coding in you know 1995 like the tools are not all the way there. We're you know in this new frontier. But personally it also kind of feels like learning how to manage a person where it's like how do I actually communicate uh you know the things that they need to know in order to make a good decision.”

0.52

The system prompt defines the high-level API of how a company operates (company-wide, customer-agnostic rules), the developer prompt adds customer-specific context and variations, and the user prompt is for end-user input in directly-consumed products; these layers allow prompt reuse across customers while enabling customization.

definitionhigh valuespeaker onlynovelty 2/4durability 3/4· Diana

“this is concept of uh defining the prompt in the system prompt. Then there's a de developer prompt and then there's a user prompt. So what this mean is uh the system prompt is basically almost like defining uh sort of the highle API of how your company operates. In this case the example of parhel is very much a system prompt. There's nothing specific about the customer. And then as they add specific instances of that API and calling it then they stuff all that in into more the developer prompt which is not shown here and that's adds all the context of let's say working with perplexity there's certain ways of how you handle rack questions as opposed to working with bold is very different right”

0.52

To build valuable evals for vertical AI, founders must sit in person next to domain experts (e.g., regional tractor sales managers) to understand their reward functions, workflows, and actual decision criteria, then codify these as specific eval cases.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Gary

“you can't get the eval unless you sitting literally side by side with people who are doing X Y or Z knowledge work. you know, you need to sit next to the tractor sales regional manager and understand, well, you know, this person cares, you know, this is how they get promoted. This is what they care about. This is that person's reward function. And then you know what you're doing is taking these in-person interactions sitting next to someone in Nebraska and then going back to your computer and codifying it into uh very specific evals like this particular user wants this outcome after they you know after this invoice comes in we have to decide whether we're going to honor the you the warranty on this tractor.”

0.52

Evals (evaluation datasets) are the true competitive moat for AI companies, more valuable than prompts themselves, because evals explain why a prompt was written a certain way and enable continuous improvement.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Jared

“like evals are the crown jewels because without the evals you don't know why the prompt was written the way that it was. Um and it's very hard to improve it.”

0.52

Beating large incumbent vendors (like Salesforce, Oracle, Booz Allen) with a weak-handshake engineer is possible by showing them something they've never seen before, making them feel deeply heard, being a great designer/product person, and delivering software so powerful that they want to buy it immediately.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Gary

“how does a really good engineer with a weak handshake go in there and beat them? It's actually you show them something that they've never seen before and like make them feel super heard. You have to be super empathetic about it. Like you actually have to be a great designer and product person and then you know come back and you can just blow them away. Like the software is so powerful that you know the second you see something that you know makes you feel seen you want to buy it on the spot.”

0.52

The best prompts start by setting up the role of the LLM (e.g., 'You're a manager of a customer service agent'), then clearly specify the task, break down the high-level plan step-by-step, outline important constraints and guardrails, and use XML tag formatting because LLMs were post-trained on XML input and produce better results with that structure.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Diana

“The big thing that a lot of the best prompts start with is this concept of uh setting up the role of the LLM. You're a manager of a customer service agent and it breaks down into bullet points what it needs to do. Then the big thing is telling the the task which is to approve or reject a tool call because it's orchestrating agent calls from all these other ones. And then it gives it a bit of the highle plan. It breaks it down step by step. You see steps one, two, three, four, five.”

0.52

The core competency required of founders building vertical AI today is being deeply and maniacally obsessed with understanding their user's specific workflows and having the software actually work for those people—this obsession is the true moat.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely Gary or Jared)

“if you are out there in particular places understanding that user better than anyone else and having the software actually work for those people that's the moat is that is like such a perfect depiction of like what is the core competency required of founders today? Like literally like the thing that you just said like that's your job as a founder of a company like this is to be really good at that thing and like maniacally obsessed with like the details of the regional tractor sales manager workflow.”

0.52

Parahelp powers AI customer support for Perplexity, Replit, Bolt, and other top AI companies by orchestrating agent calls, and they were willing to open-source their six-page system prompt because the real crown jewel IP is not the prompt itself but the evals (evaluation datasets).

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Jared

“Parahelp does AI customer support. There are a bunch of companies who who are doing this, but Parhel is doing it really really well. They're actually powering the customer support for Perplexity and Replet and Bolt and a bunch of other like top AI companies now.”

0.52

Gemini 2.5 Pro's ability to reason about exceptions to rubrics is valuable for investor evaluation because some investors might be slow to respond but have great track records, and a rigid rubric would penalize them when they deserve a 91 instead of 89.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Harj and Gary Tan

“just thought was interesting cuz that it's just like when you're training a person, you're trying to you give them a rubric like you want them to use a rubric as a guide, but there are always these sort of edge cases where you need to sort of think a little bit more deeply. Um, and I just thought it was interesting that the models themselves will handle that differently, which means they sort of have different personalities, right? Like 03 felt a little bit more like the soldier sort of like, okay, I'm definitely like check, check, check, check, check. Um, and Gemini Pro 2.5 felt a little bit more like a a high agency sort of employee was like, "Oh, okay. I think this makes sense, but this might be an exception in this case," which was um just really interesting to see.”

0.52

For complex tasks that are hard to specify exactly, giving the LLM a few working examples is often more effective than trying to write precise instructions, because examples help the LLM reason around complicated tasks and steer it better than exact parameters can.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Jared

“I think this pattern of sometimes when it's too hard to even kind of write a pros around it let's just give you an example that turns out to work really well because it helps LM to reason around complicated tasks and steer it better because you can't quite kind of put exact act parameters and it's almost like um unit testing programming in a sense like test-driven development is sort of the LLM v version of that.”

0.52

LLMs have a strong tendency to hallucinate when asked to provide output in a specific format—they will make up information to fit the format even if they don't have enough information—so you must give them an explicit escape hatch by telling them 'if you don't have enough information, don't make it up, stop and ask me' rather than relying on them to volunteer uncertainty.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Jared

“the model really wants to actually help you so much that if you just tell it give me back output in this particular format even if it doesn't quite have the information it needs it'll actually just tell you what it thinks you want to hear and it's literally a hallucination. So, one thing they discovered is that you actually have to give the LLM's a real escape hatch. You need to tell it if you do not have enough information to say yes or no or make a determination, don't just make it up. Stop and ask me.”

0.52

Founders must themselves be the forward-deployed engineers—they can't farm out this work. They need to be technical, be great product designers, be ethnographers, and personally do the demos and customer work that lead to deals.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely Jared, replying)

“Is a good way of thinking about it that founders should think about themselves as being the four deployed engineers of their own company. Absolutely. Yeah. Like you definitely can't farm this out. Like literally the founders themselves, they're technical. They have to be the great product people. They have to be the ethnographer. They have to be the designer. You want the person on the second meeting to see the demo you put together based on the stuff you heard. And you want them to say, Wow, I've never seen anything like that. And take my money.”

0.52

Some companies use metaprompting with a large, expensive model (like Claude 3.5 or GPT-4o) to generate high-quality prompts, then distill those prompts into smaller, faster models (like a smaller open-source model) to reduce latency for production use cases like voice AI agents where response time is critical for passing the Turing test.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“I think it's a common pattern sometimes for companies when they need to get um responses from element elements in their product a lot quicker. They do the meta prompting with a bigger beefier model any of the I don't know hundreds of billions of parameter plus models like uh I guess cloud 4 3.7 or your uh GPD 03 and they do this meta prompting and then they have a very good working one that then they use into the distilled model. So they use it on uh for example an FRO and it ends up working pretty well specifically sometimes for uh voice AI agents companies because uh latency is very important to uh get this whole touring test to pass because if you have too much pause be before the agent responds I think humans can detect something is off.”

0.51

The primary competitive concern for vertical AI startups—whether they are 'just wrappers'—is actually resolved through superior understanding of user workflows and ensuring the software works exceptionally well for those specific people, which is where the actual moat is built.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Gary Tan

“everyone's really worried about um are we just rappers and you know what is going to happen to startups and I think this is literally where the rubber meets the road where um if you you know if you are out there in particular places understanding that user better than anyone else and having the software actually work for those people that's the moat is that is like such a perfect depiction of like what is the core competency required of founders today? Like literally like the thing that you just said like that's your job as a founder of a company like this is to be really good at that thing and like maniacally obsessed with like the details of the regional tractor sales manager workflow.”

0.50

For complex tasks that are hard to specify exactly, providing worked examples (especially those involving hard cases only expert programmers can solve) is more effective than trying to write precise prompt instructions, functioning like test-driven development for LLMs.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely Jared)

“they feed a bunch of really hard examples that only expert programmers could do let's say if you want to find an N plus1 query it's actually hard for today for even like the best LMS to find those and the way to do those is they find parts of the code then they add those into the prompt a meta prompt that's like hey this is an example of n plus1 type of error and then that works it out and I think this pattern of sometimes when it's too hard to even kind of write a pros around it let's just give you an example that turns out to work really well”

0.50

YC has used LLMs to help founders decide which investors to take money from by creating scoring rubrics (0-100 scale) and testing models like GPT-4o Turbo versus Gemini 2.5 to understand how different models interpret and apply the same rubric.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Gary Tan

“Well, one of the things we've been using uh LLMs for internally is actually helping founders figure out who they should take money from. And so in that case, sometimes you need a very straightforward rubric, a zero to 100. zero being never ever take their money and 100 being take their money right away. Like they actually help you so much that you'd be crazy not to take their money.”

0.50

The emergence of LLMs has made data intelligence and sense-making more tractable, because language models can work with unstructured data (documents, conversations, narratives) in ways that traditional data mining couldn't.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Unknown Speaker (likely Gary)

“it's no mistake that actually now that we have LLMs like we actually it is becoming much more tractable and then the forward deployed engineer title was specifically how do you sit next to literally the FBI agent”

0.49

Vertical AI agent startups can close six and seven-figure enterprise deals very quickly (even the next day after initial meetings) because founders can sit with end buyers, understand their context, embed that in prompts, and return with a demo—something that would have taken traditional consulting teams weeks.

factualhigh valuespeaker onlynovelty 2/4durability 2/4· Jared

“the incredible thing about this model is this is why we're seeing a lot of the vertical AI agents take off is precisely this because they can have these meetings with the end buyer and champion at these big enterprises. They take that context and then they stuff it basically in the prompt and then they can quickly come back in a meeting like just the next day maybe with Palunteer would have taken a bit longer and a team of engineers here. It could be just the two founders go in and then they would close this six, seven figure deals which we've seen and with large enterprises which has never been done before”

0.49

Parahelp powers AI customer support for top companies like Perplexity, Replit, and Bolt, and they shared their actual production prompt (6 pages long) showing how best-in-class prompts are structured with role definitions, task specifications, step-by-step reasoning, output formatting, and examples.

factualhigh valuespeaker onlynovelty 2/4durability 2/4· Jared

“Parahelp does AI customer support. There are a bunch of companies who who are doing this, but Parhel is doing it really really well. They're actually powering the customer support for Perplexity and Replet and Bolt and a bunch of other like top AI companies now.”

0.49

One startup opportunity is building automation to automatically extract and insert the best worked examples from customer datasets into the appropriate place in the prompt pipeline, rather than manual prompt ingestion.

normativehigh valuespeaker onlynovelty 2/4durability 2/4· Unknown Speaker (likely Gary or Jared)

“anyone who's done prompt engineering knows that the examples and worked examples are really important to improving the quality of the output. And so then if you take like power as an example, they really want good worked examples that are specific to each company. And so you can imagine that as they scale, you almost want that done automatically. Like in your dream world, what you want is just like a an agent itself that can pluck out the best examples from like the customer data set and then software that just like ingests that straight into like wherever it should belong in the pipeline without you having to manually go out and plug that all and ingest it in all of yourself.”

0.48

A forward-deployed engineer's job involves converting file-cabinet and fax-machine-era business processes into clean software—the ideal is making business processes as frictionless as posting a lunch photo to Instagram.

definitionhigh valuespeaker onlynovelty 1/4durability 3/4· Gary

“what you do as a forward deployed engineer is take these sort of you know file cabinet and fax machine things that people have to do and then convert it into really clean software. So you know the classic view is that it should be as easy to actually do uh an investigation at a threeletter agency as going and taking a photo of your lunch on Instagram and posting it to all your friends.”

0.48

Some investors like Benchmark or Thrive have immaculate processes, respond quickly, and never ghost—making them obvious 'take their money' cases—while others are overwhelmed and slow but have strong track records, creating legitimate edge cases where debug info matters.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Gary

“You know, sometimes you have investors like a Benchmark or a Thrive, it's like, "Yeah, take their money right away. Their process is immaculate. They never ghost anyone. They answer their emails faster than most founders. It's, you know, very impressive. And then, uh, one example here might be, you know, there are plenty of investors who are just overwhelmed and maybe they're just not that good at managing their time. And so, they might be really great investors and their track record bears that out, but they're sort of slow to get back. They seem overwhelmed all the time. They accidentally, probably not intentionally ghost people. And so this is legitimately exactly what an LLM is for.”

0.48

Giger ML (customer support and voice AI company) differentiated from competitors by innovating their RAG pipeline to achieve both accurate and very low-latency voice responses, a technically challenging combination that allowed them to win deals through superior demo performance rather than sales process.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Jared (YC AI)

“they've um innovated a bit on the rag pipeline so that they can um have their voice responses be both accurate and very low latency. sort of like a technically challenging thing to do, but I just feel like in the like pre sort of the current LLM rise, you couldn't necessarily differentiate enough in the demo phase of sales to beat out incumbent.”

0.48

Metaprompting feels like coding in 1995: the tools are not yet complete, there is much that's unspecified, and we are in a new frontier where the field is still being built.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Jared (YC AI)

“it kind of actually feels like coding in you know 1995 like the tools are not all the way there. We're you know in this new frontier.”

0.47

Giger ML are software engineers who forced themselves into the forward-deployed engineer role despite not being natural salespeople, and they closed major deals with Zapier by delivering impressive demos with technically optimized voice response generation (low latency + accuracy via RAG pipeline).

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“I think of a company I mentioned before on the podcast like Giger ML who do customer another customer support and especially a lot of voice support and it's just classic case of two extremely um talented software engineers not natural sales people but they force themselves to be essentially forward deployed engineers and they closed a huge deal with Zeppto and then a couple of other companies they can't announce yet”

0.46

The forward deployed engineer role originated at Palantir, where founders (Peter Thiel, Alex Karp, Stefan Cohen, etc.) recognized that Fortune 500 and government agencies had multi-billion to trillion-dollar problems from data overload ('world is awash in data') but no world-class technologists in those organizations to solve them.

factualhigh valuespeaker onlynovelty 1/4durability 4/4· Gary Tan

“the whole thesis of Palunteer at some level was that um if you look at Meta back then it was called Facebook or Google or any of the top software startups that everyone sort of knew back then. One of the key recognitions that Peter Teal and Alex Karp and Stefan Cohen and Joe Lansdale, Nathan Gettings, like the original founders of Palunteer had was that uh go into anywhere in the Fortune 500, go into any government agency in the world, including the United States, and nobody who understands computer science and technology at the level that you at the highest possible level would ever even be in that room.”

0.46

GPT-4 03 felt like a 'soldier' following orders strictly (check, check, check), while Gemini 2.5 Pro felt like a high-agency employee asking 'Why is this an exception?' and making discretionary judgments—a meaningful difference in how to train/interact with each model.

factualhigh valuespeaker onlynovelty 2/4durability 2/4· Unknown Speaker (likely Harj)

“Um, and I just thought was interesting that the models themselves will handle that differently, which means they sort of have different personalities, right? Like 03 felt a little bit more like the soldier sort of like, okay, I'm definitely like check, check, check, check, check. Um, and Gemini Pro 2.5 felt a little bit more like a a high agency sort of employee was like, Oh, okay. I think this makes sense, but this might be an exception in this case, which was um just really interesting to see.”

0.45

Companies using metaprompting with large models (like Claude 3.5, GPT-4o) to refine prompts often then distill the better prompts into smaller models (like Llama), achieving latency-critical performance (especially for voice AI agents) while maintaining quality from the refined prompts.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“I think it's a common pattern sometimes for companies when they need to get um responses from element elements in their product a lot quicker. They do the meta prompting with a bigger beefier model any of the I don't know hundreds of billions of parameter plus models like uh I guess cloud 4 3.7 or your uh GPD 03 and they do this meta prompting and then they have a very good working one that then they use into the distilled model. So they use it on uh for example an FRO and it ends up working pretty well specifically sometimes for uh voice AI agents companies because uh latency is very important to uh get this whole touring test to pass because if you have too much pause be before the agent responds I think humans can detect something is off.”

0.45

Happy Robot builds AI voice agents for logistics brokers and has closed seven-figure contracts with the top three largest logistics brokers by using the forward-deployed engineer model, starting from six-figure deals just a couple months prior.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Jared

“Happy Robot who has sold seven figure contracts to the top three largest logistic brokers in the world. They build AI voice agents for that. They are the ones doing the forward deploy engineer model and talking to like the CIOS of these companies and quickly shipping a lot of product like very very quick turnaround. And it's been incredible to see that take off right now. And it started from six figure deals now doing closing and seven figure deals which is crazy. This is just a couple months after.”

0.44

When evaluating investor quality for founders, best practice is to use LLMs with rubrics (0-100 scoring scales) to standardize evaluation, but GPT-o3 strictly adheres to rubrics while Gemini 2.5 Pro flexibly reasons about exceptions.

factualhigh valuespeaker onlynovelty 2/4durability 2/4· Unknown Speaker (likely Harj based on 'we found')

“it's certainly best practice to give um LLM's rubrics, especially if you want to get a numerical score as the output. You want to give it a rubric to help it understand like how should I think through and what's like a 80 versus a 90. But these rubrics are never perfect. there's often always exceptions and you tried it with uh 03 versus Gemini 2.5 and you found this this is what we found really interesting is that um 03 was very rigid actually like it really sticks to the rubric”

0.43

As prompts get longer and become large working documents, a useful technique is to keep notes in a Google Doc of outputs that aren't working as intended or ideas for improvement, then give those notes plus the original prompt to an LLM like Gemini Pro and ask it to suggest edits to incorporate these improvements—it does this quite well.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“like as the prompt gets longer and longer like it becomes a a large working doc um one thing I found useful is as you're using it if you just note down in a Google doc things that you're seeing just um the outputs not being how you want or not ways that you can think of to improve it. you can just write those in note form and then give Gemini Pro like your notes plus the original prompt and ask it to suggest a bunch of edits to the prompt um to incorporate these in well and it does that quite well.”

0.42

The debug_info parameter output from LLM evaluations (with scores like 91 vs 89) is often the most interesting and useful part, revealing nuanced reasoning about edge cases that the rubric alone wouldn't capture.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Gary Tan

“the debug info on some of these are very interesting to see like you know maybe it's a 91 instead of like an 89. We'll see.”

0.42

The best prompts use XML tag formatting (like programming) rather than plain English, because LLMs were post-trained with RLHF on XML-type inputs and this produces better results.

causalhigh valuespeaker onlynovelty 1/4durability 2/4· Diana

“it looks more like programming than writing English because it has this uh XML tag kind of format to specify sort of the plan. We found that it makes it a lot easier for LMS to follow because a lot of LMS were post-trained in LHF with kind of XML type of input and it turns out to produce better results.”

0.42

Different LLM models (Claude, Llama 4) have different personalities: Claude is more 'happy' and human-steerable, while Llama 4 requires more steering and is 'rough to work with,' potentially because it hasn't had as much RLHF (reinforcement learning from human feedback) applied to it.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“One of the things that's kind of interesting about uh each model is that they each seem to have their own personality. And one of the things the founders are really realizing is that you're going to go to different people for different things. Actually, one of the things that's known a lot is Claude is sort of the more happy and more human steerable model. And the uh other one is Lama 4 is one that needs a lot more steering. It's almost like talking to a developer and part of it could be an artifact of not having done as much RL RHF on top of it. So is a bit more rough to work with, but you could actually steer it very well if you actually are good at actually doing a lot of prompting and almost doing a bit more RLHF, but it's a bit harder to work with actually.”

0.42

YC has built various workbenches for debugging and testing prompts, but sometimes it's better to just use gemini.google.com directly and drag-and-drop JSON files rather than working in a special container, and this approach also works in ChatGPT.

normativehigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“Jared and the software team at YC has actually built this um you know various forms of workbenches that allow us to like do debug and things like that. But to your point like sometimes it's better just to use gemini.google.com directly and then drag and drop you know literally JSON files and uh you know you don't have to do it in some sort of special container like it you know seems to be totally something that works even directly in you know chat GPT itself.”

0.39

Different LLM models have distinct personalities: Claude is described as more 'happy' and human-steerable, while Llama 4 requires more steering and is rougher to work with, partly due to less extensive RLHF training, but can be steered effectively with good prompting.

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“one of the things that's known a lot is Claude is sort of the more happy and more human steerable model. And the uh other one is Lama 4 is one that needs a lot more steering. It's almost like talking to a developer and part of it could be an artifact of not having done as much RL RHF on top of it. So is a bit more rough to work with, but you could actually steer it very well if you actually are good at actually doing a lot of prompting and almost doing a bit more RLHF, but it's a bit harder to work with actually.”

0.22

YC itself is becoming a place where founders learn to be forward-deployed engineers, learning these techniques alongside building their AI companies.

factualspeaker onlynovelty 0/4durability 2/4· Jared

“we've uh, talked about evals for going on a year now”

0.17

Eric Bacon, YC's head of data, has been instrumental in helping YC companies apply metaprompting and use Gemini Pro 2.5 effectively as a debugging tool.

factualspeaker onlynovelty 0/4durability 2/4· Gary

“I would give a shout out to YC's head of data, Eric Bacon, who's um, helped us all a lot a lot of this metaparrotting and using Gemini Pro 2.5 as a effectively a ripple.”

0.17

YC has built various forms of workbenches for debugging and testing prompts, but direct use of Gemini's web interface or ChatGPT with drag-and-drop JSON files often works just as well without needing specialized containers.

factualspeaker onlynovelty 0/4durability 2/4· Unidentified Speaker — State-Of-The-Art Prompting For AI Agents [DL82mGde6wo]

“Jared and the software team at YC has actually built this um you know various forms of workbenches that allow us to like do debug and things like that. But to your point like sometimes it's better just to use gemini.google.com directly and then drag and drop you know literally JSON files and uh you know you don't have to do it in some sort of special container like [14:00] it you know seems to be totally something that works even directly in you know chat GPT itself.”

0.13

YC's head of data Eric Bacon has been instrumental in helping YC companies learn metaprompting and use Gemini Pro 2.5 as an effective tool for prompt development.

factualspeaker onlynovelty 0/4durability 1/4· Unknown Speaker (likely Gary or Jared)

“I would give a shout out to YC's head of data, Eric Bacon, who's um, helped us all a lot a lot of this metaparrotting and using Gemini Pro 2.5 as a effectively a ripple.”