Sheela
Sheela
Home
Add
Questions
Thinkers
Claims
People
Concepts
Jargon
Sources
Health
Predictions
Collections
Domain Landscape
Contribute
Review
Dedup
Topic requests
Settings
Help

← Jargon

Jargonsearch technology· 1 claim· 1 episode

crawling the web

The automated process by which a search engine systematically fetches and indexes web pages.

As defined in Paul Buchheit on Google, Friendfeed, and Start-ups 09/21/2009


Where it comes up · 1
  • Paul Buchheit on Google, Friendfeed, and Start-ups 09/21/20091 claim

Most important claims · 1 of 1Search all claims for “crawling the web” →
  • Email infrastructure requires fundamentally different reliability assumptions than web search—losing a crawled web page is recoverable, but an email system can never lose data, and this constraint propagates down to the lowest-level technical decisions in the stack.Paul Buchheit on Google, Friendfeed, and Start-ups 09/21/2009