MapReduce
A programming model for processing large datasets by distributing computation across many machines and aggregating results.
As defined in Balaji Srinivasan
Where it comes up · 4
- Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]3 claims
- Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker3 claims
- Making Decisions with Data1 claim
- Balaji Srinivasan1 claim
Most important claims · 8 of 8Search all claims for “MapReduce” →
- MapReduce was fundamentally inefficient compared to distributed database systems, and Google's adoption of eventual consistency as a concurrency control mechanism was architecturally flawed because it cannot maintain integrity constraints like stock levels remaining non-negative.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- The core innovation of MapReduce was to move the code that processes data to the machines where the data sits, because data volumes were too large to transfer over the network without collapsing it.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- Hadoop with MapReduce became the first operating system for the distributed data center and enabled processing of any amount of data in parallel, but every computation had to be expressed through just two functions, map and reduce, which made programs cumbersome and limited writing to sophisticated programmers.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- MapReduce was ridiculously inefficient, and Stonebraker (along with others like Dave Dewitt) understood distributed databases well enough to beat the heck out of Hadoop with a distributed database system, which was the central claim of their 2011 paper.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Google eventually abandoned both eventual consistency and MapReduce in favor of conventional transactional systems when Jeff Dean implemented Spanner, recognizing that data correctness was necessary for their use cases.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Peter Thiel's contrast between blockchain's decentralizing aspects and AI's centralizing aspects is complicated by recent developments allowing machine-learning coefficients to be calculated locally on phones via a MapReduce-style approach where data stays local and the central server is just a coordinator that doesn't store the data, as in Google's published research.Balaji Srinivasan
- When building data teams, hire people trained in the scientific method and hypothesis-driven inquiry rather than those who only have technical data science skills, since foundational scientific thinking is harder to teach than tools.Making Decisions with Data
- The original assumption of distributed data systems, that network bandwidth is the scarce resource and code must travel to data, has become obsolete because networks and cloud infrastructure now outpace data growth.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]