crawling the web
The automated process by which a search engine systematically fetches and indexes web pages.
As defined in Paul Buchheit on Google, Friendfeed, and Start-ups 09/21/2009
Where it comes up · 1
Most important claims · 1 of 1Search all claims for “crawling the web” →
- Email infrastructure requires fundamentally different reliability assumptions than web search—losing a crawled web page is recoverable, but an email system can never lose data, and this constraint propagates down to the lowest-level technical decisions in the stack.Paul Buchheit on Google, Friendfeed, and Start-ups 09/21/2009