What this covers

Russ Roberts interviews Adam D'Angelo, CEO of Quora, about how the question-and-answer platform operates at scale. The conversation centers on two operational commitments that D'Angelo argues are essential: disciplined controlled experimentation and machine learning-driven personalization. Rather than rely on product intuition or feature accumulation, D'Angelo describes how Quora tests changes systematically, with particular rigor around pre-registration of hypotheses to avoid the Multiple Comparisons Problem. The interview ranges across Quora's specific design choices—enforcing real names to bring expert credibility, routing questions to qualified answerers, consolidating duplicate questions into canonical versions—and explains how each one connects to the underlying mission of aggregating and distributing knowledge.

The discussion expands into broader economic and strategic terrain. D'Angelo contends that complexity tax from accumulating features eventually throttles a product's speed and maintainability, and that even millisecond delays in responsiveness measurably degrade usage across millions of users because Transaction Costs matter profoundly. He argues that Personalization as Market Force quietly reshapes entire industries—consolidating fragmented markets like magazines or television into a small number of dominant products. The conversation also touches on tensions: how signaling theory applies to answer ranking, whether learning experiments can be gamed by metrics that rise for the wrong reasons, and how personalization enables Price Discrimination that may harm consumers even when it benefits the company. Throughout, D'Angelo emphasizes that most product intuitions fail under testing, making experiments the only trustworthy path to improvement.

Sharpest takeaway

D'Angelo argues that running a high-quality knowledge-sharing platform at scale depends less on intuition and feature-building than on disciplined controlled experimentation and machine-learning-driven personalization, which together quietly reshape entire markets.

  • Real names plus personalization create an environment where experts participate and the right answers reach the right users
  • Most product intuitions fail when tested, so controlled experiments are the only reliable way to improve
  • Personalization lets a single product serve diverse needs, consolidating fragmented markets

The argument · threads9 threads · 36 claims
0.81

Knowledge locked in experts' heads or paywalled research must be extracted and redistributed as deduplicated canonical answers.

7 pointscentrality 5/5
  • A huge amount of valuable knowledge exists only in people's heads or is locked in research papers requiring domain-specific language to navigate, creating an opportunity to build a platform that extracts and shares this knowledge with the world.

    “there is a huge amount of knowledge that is out in people's heads, that's not on the Internet; it's not very easy to get access to. There's a huge amount of knowledge that I think is locked in research papers where you need to know a certain amount of very domain-specific language to even navigate”

  • Almost all questions people have were asked before, so Quora deliberately avoids having multiple versions of each question and concentrates all answering energy into one canonical place, which generates the best possible definitive answers.

    “we don't want to have 20 versions of each question. We want to have one version; and that allows us to concentrate all the answers' energy into that one place. And then that helps to generate the best possible answers.”

  • Quora routes questions to potential answerers based on signals like whether their past answers were on the same topic, and withholds questions that already received a great answer, because generating duplicate answers works against the mission of sharing all the world's knowledge.

    “we don't want to have 20 versions of each question. We want to have one version; and that allows us to concentrate all the answers' energy into that one place. And then that helps to generate the best possible answers.”

  • Unlike Google which indexes existing internet knowledge well, Quora's distinct challenge is getting new information onto the internet in the first place, and prior question-and-answer products failed because quality degrades at scale into a mess with no expert participation.

    “The challenge we're interested in is: How do you get more information onto the Internet in the first place?”

  • Most internet services today are not set up with concern for the future, focusing on what is happening right now and treating content even a week old as irrelevant, whereas Quora has a strong focus on long-term archiving of contributed knowledge.

    “I think it's actually something that most Internet services today are not really set up with any concern for the future. It's about what's happening right now; and then the content that's even a week later is not really relevant. So, we have a big focus on this.”

  • Quora has about 100 million monthly unique visitors, most of whom are reading questions others asked rather than asking their own.

    “we have about 100 million monthly unique visitors--that was the last number we announced.”

  • Quora uses a machine learning algorithm to detect whether two questions are likely the same and suggests existing duplicates at ask-time, plus an offline machine learning process and a user-driven merge process with quality review, to consolidate questions.

    “we have a machine learning algorithm that can look at two questions and determine whether, make a guess at whether those are likely the same question or not.”

0.74

Product decisions must be validated through controlled experiments, not intuition, because untested theories cause destructive failures.

5 pointscentrality 5/5
  • Personal usage is good for generating theories and ideas, but those ideas should ideally be tested with controlled experiments and cross-checked against surveys and other data sources, because acting on untested intuition can send a product in a crazy, destructive direction.

    “it's good for generating theories and ideas. Usually I'd like to test those theories and ideas on some, ideally on a controlled experiment... I try to make sure that the ideas I generate from my own usage are consistent with the data. Because if you don't do that, you can really just go off in a direction that's crazy.”

  • Quora enforces that every experiment either declares a hypothesis in advance or explicitly declares it a learning experiment; if a learning experiment surfaces a promising effect, a follow-up experiment must be run with that as the hypothesis, because otherwise you run into statistical significance problems.

    “every experiment ahead of time, it either has to, either declare a hypothesis, or you have to declare that there is explicitly no hypothesis and you are just doing it to learn... If you have what we call a learning experiment, and you pick up on some...consequence of that experiment...we require you to run another experiment, where that's the hypothesis. So--because otherwise you run into, you know, statistical significance problems.”

  • Most product experiments fail to confirm their hypotheses, and Quora's success rate is around a third, which is actually better than rates D'Angelo hears from other companies; this is counterintuitive and demoralizing to newcomers because nearly everything they did before usually appeared to work.

    “It's most of the time... It's maybe a third of-- ... I hear numbers from other companies that are actually more negative.”

  • Many companies operate in a dreamland believing their changes are working, when the real reason the product grows may be network effects or external factors like more people gaining internet access, and biases like wishful thinking plus career consequences push people toward irrational, non-objective interpretations of results.

    “a lot of other companies are just operating on this mindset where they are just in, like, dreamland, where they think that all these things they are doing are working. But the real reason that the product is working is some kind of network effect”

  • As CEO, D'Angelo deliberately uses Quora like a normal user rather than artificially writing many answers and following many topics, because behaving abnormally would give him an experience nothing like a normal user's and bias the information he gets from his own usage.

    “I actually try to behave more like what I think a normal user behaves... I just try to like act like a normal user so that the information I get isn't so biased.”

0.66

Personalization algorithms must tailor content based on user behavior and context to optimize both engagement and relevance.

3 pointscentrality 5/5
  • Quora's personalization algorithm takes as input the topics you follow, the people you follow, your past reading and answering behavior, and questions you've followed, and predicts what content will interest you and which questions you'll want to answer, so that two users following the same topic get very different experiences depending on factors like where they live.

    “We have the topics that you've followed. We have the people who you are following. We have all your behavior from what you've read in the past and questions that you've answered in the past, questions you've followed. And we have an algorithm that takes all that information as input and then uses that to try to predict what's going to be interesting to you”

  • Personalization is a powerful technology that reshapes markets: because the internet allows infinite content tailored to each user, formerly fragmented markets like magazines collapse into a small number of products that reach much bigger audiences, exemplified by a single Google search engine and Netflix replacing many TV channels.

    “personalization is a really powerful technology as far as its effect on markets... with the Internet, you can have, sort of an infinite amount of content; and you can have every user getting a totally different experience. And so that's meant that markets that used to be fragmented, like, say, the magazine industry, now in the Internet you can just have a very small number of products that are able to reach a much bigger audience.”

  • The power of personalization is under most people's radar, including industry insiders, because each user only sees one experience and finds it unintuitive that others see totally different things; one of machine learning's biggest economic impacts will be enabling ever-better personalization, making products much better per user and spreading personalization into markets where it never existed.

    “one thing that's under a lot of people's radar, including people in the industry, is just the power of personalization... one of the biggest impacts that the machine learning has on the economy is through enabling better and better personalization, which allows these products to get much, much better for any given user than they used to be... you are going to see personalization in all these other markets where it never existed before.”

0.63

Real names and authority-weighted voting are essential to overcome experts' reluctance to build reputation from zero online.

2 pointscentrality 5/5
  • When ranking answers, the identity of the voter is a very important signal beyond mere vote count, so an upvote from someone known to have authority on a particular topic carries more weight than a generic vote.

    “The votes are a very important signal, especially not just like the number of votes but who was the voter. So, did you get a vote from someone who we know really has authority on some particular topic? That's an important signal to us.”

  • Requiring real names lets people with real-world reputations bring credibility into the platform from the start, which is crucial because forcing experts to build reputation from zero in an anonymous online system is discouraging and put competing Q&A platforms at a disadvantage.

    “it lets someone with a real-world reputation bring that reputation into the platform and start out with this credibility. And so, these other platforms at the time that we started didn't really use real names. And so that just put them at a disadvantage. And it's very discouraging to an expert to have to start from zero in an online system.”

0.53

Feature proliferation imposes compounding complexity costs that eventually paralyze product development and confuse users.

3 pointscentrality 4/5
  • Adding features imposes a recurring complexity tax: each new feature must interact with all others, be controlled for in future experiments, have its impact measured, be maintained and debugged, and be explained to users, eventually reaching a point where even employees don't know all the features, which slowed Quora down significantly.

    “The complexity tax just has hurt us over and over... every feature you are going to add has to interact with that feature. And you are going to have to control for the effect of that feature in some future experiment; and you are going to have to go measure the impact of that feature; and you have know when that feature breaks and fix bugs in it; and you have to explain to users how to use it. And then you get to this point where you have so many features and people inside the company don't even know all of them.”

  • There is a finite surface area users can consume, defined as the full description of everything that can happen in the product, so a mature product should make the existing surface area better rather than continually adding more, which is very hard to do well without running experiments.

    “there's sort of a finite surface area that users can consume. So we've built the product and the surface area of the product can't really expand that much. And so at some point you don't want to keep adding surface area: you want to make the existing surface area better. And it's actually very hard to do that without running experiments.”

  • It is much easier to prevent the addition of a complexity-increasing feature than to remove an existing one, and because Quora will never delete any knowledge a user has shared, it is especially hard to remove features tied to user-contributed information.

    “it's much easier to prevent the addition of another feature that's going to make things more complex than it is to remove an exiting feature. Because one thing that's very important to us is, we never want to remove anyone's knowledge that they've shared”

0.53

Exponential growth creates technical and operational demands requiring dedicated platform infrastructure and data science capability.

3 pointscentrality 4/5
  • Quora's exponential growth pattern, with many more users each week than the prior week, creates internal challenges requiring infrastructure to handle the load, improving personalization technology, and the ability to run more experiments as more data accumulates.

    “the growth basically follows this exponential growth pattern, so that there's a lot more people using the product every week than the week before. And that creates a lot of challenges internally.”

  • A dedicated platform team building tools and infrastructure produces a multiplicative effect on product development, because the progress they make on shared building blocks lets other teams take on challenges that would have been infeasible at smaller scale, helping avoid diminishing returns to experimentation.

    “we have a big fraction of the engineering team, all they work on is tools and infrastructure to make the rest of the engineering team more efficient... we call this the Platform Team--the progress that they make on blocks, efforts that then other teams can take on. And so you have this really multiplicative effect within the product development team”

  • Quora built an experiment framework about four years ago, earlier than most companies do, and since then has run about 2000 controlled experiments, with roughly 30 running concurrently at any time, staffed by 13 data scientists out of a 130-person company.

    “if you are just counting experiments that have happened since the framework, I think we've run about 2000 experiments... at any given time we have about 30 experiments running concurrently... the whole company is about 130 people. There are 13 Data Scientists.”

0.44

Regulatory threats and market competition create incentives for transparency and ethical behavior, though personalization raises welfare concerns.

8 pointscentrality 2/5
  • Much of what motivates people to participate in social networks like Twitter, Facebook, Instagram, and Snapchat is a desire to signal information about themselves that is useful to others, making these products one of the first chances to apply signaling theory to product design.

    “I think a lot of what motivates people to participate in these social networks, especially things like Twitter, Facebook, Instagram, Snapchat is a desire to signal things about themselves--information that's going to be useful to other people. So, it's one of the first chances that we've had to really apply signaling theory to product design.”

  • Even products in seemingly different markets compete for users' discretionary online time, so Netflix effectively competes with Facebook and Quora competes with Hulu, because the better any one product gets, the more of a user's limited time it captures.

    “effectively Netflix is in competition with Facebook even though you think about them as totally different markets. People have this discretionary time that they are going to spend online; and the better Netflix gets, the more time is going to Netflix; and the better Facebook gets, the more time is going to Facebook.”

  • Quora believes it can make enough money to sustain the business by running ads on a small percentage of pages without significantly worsening the user experience, following Google's model where most searches show no ads but commercial searches show more.

    “We think we can make a lot of money, enough to sustain the business, without having to make the user experience a lot worse. I think a good example of this is Google. You do a search; most of the time you don't see any ads. If you happen to search for something that's commercial, then you'll see more ads.”

  • Even the mere threat of future regulation can protect consumers, because companies have an incentive to disclose how their systems work (as Facebook did with newsfeed personalization) and to behave well, to get ahead of regulators and prevent misconceptions that would lead to worse regulation.

    “Facebook came out and disclosed how they think about personalizing their newsfeed... part of the motivation there is to get ahead of regulators. Because they'd rather have the public understand what they are trying to do than have these misconceptions out there that then lead to regulation... even the threat of regulation I think is good for protecting consumers.”

  • A machine learning technique that looks great in a published research paper often fails to work as well as a much simpler technique once it has to be engineered into production code, because production realities can make the fancier method underperform.

    “someone will publish a research paper and it looks great, where you have to get it into the actual production code where you have to engineer it, it turns out that there are some things that make it not actually work as well as a much simpler technique that was able to be implemented better.”

  • For products that are not ad-supported, personalization enables price discrimination and companies may push higher-margin items, so personalization is not always welfare-improving for consumers even if it is for the company; Amazon has taken a strong stance against price discrimination, but that could reverse.

    “in these companies that are selling a product that are not ad-supported--I think personalization enables price discrimination... And I think Amazon has actually taken a very strong stance against price discrimination. But, you know, that could reverse at some point.”

  • Quora tends not to trust the results of other companies' product changes much, because those companies often make mistakes and have different user bases, strategies, and market positions, so what is right for them may not be right for Quora.

    “I think we tend not to trust the results of other companies' product changes very much... The companies have different user bases than us; they have different strategy, a different position in the market.”

  • Keeping a company mission front and center is achieved less by reciting the phrase than by hiring people who genuinely care about the mission, screening candidates for whether the mission is their motivation rather than viewing the company as interchangeable with ten others.

    “getting employees in here in the first place who care about knowledge is sort of more important than just sort of reciting the phrase over and over... is that their motivation to come work here? Or, how much does that compare to--or do they just view us as one of 10 different companies”

0.39

Product quality is measured by behavioral outcomes and infrastructure speed, not surveys, but qualitative data remains essential.

3 pointscentrality 3/5
  • Product speed has a surprisingly large effect on usage, with differences down to milliseconds causing measurable increases when averaged over millions of users, because any friction anywhere greatly decreases how much something gets used—transaction costs are a huge issue.

    “One thing is just the important of speed--of making the product fast for users... how much, down to milliseconds, it will cause an increase in usage. And when you average these things out over millions of users, you can really pick up on these very small differences.”

  • Behavioral outcomes like whether users write more and better answers are usually a more reliable indicator of whether a product change improved the experience than surveys, but surveys remain essential to capture qualitative reasons and detect causes that metrics cannot explain, such as bad mobile connections or government filters.

    “do you write more answers, do you write better answers?... that's usually a better, a more reliable indicator of whether we made the product better than a survey. But surveys are also useful.”

  • A change can increase short-term usage for the wrong reasons; for example, making it harder to find questions to answer might make a recruiter spend more time and write more answers to achieve the same goal, so naive engagement metrics can misrepresent whether the product actually got better.

    “maybe you'll come back more in the short term. Because you--let's say, your objective was to recruit people for your company... If we make it very hard for you to find questions to answer you might actually spend more time looking for answers. You might have to write more answers to achieve the same goal.”

0.21

Learning behavior on knowledge platforms triggers intrinsic reward signals, and aggregated user sentiment creates historical record.

2 pointscentrality 2/5
  • Reading behavior on a knowledge platform functions like the mind rewarding the user for acquiring information that might be helpful later, producing a short-term payoff where it feels good to learn new things and express curiosity.

    “I almost think of the reading behavior as your mind is rewarding you for getting some information that might be helpful to you later. So you get this sort of short-term payoff that it feels good to learn new things and to express your curiosity.”

  • Aggregated knowledge platforms could let future historians reconstruct what most people actually thought and felt about events, overcoming the traditional limitation that history is colored by the single person writing it or a lone diary, because the internet creates hundreds of millions of diaries of public sentiment.

    “you get this interpretation of history that's very colored by the person who is writing the history... in a hundred years from now people will be able to look back, and through Quora and other services will be able to get a sense for, really, what did people think of Donald Trump: like, what was the true reaction to this, this candidate?”