Hadoop
Open-source implementation of MapReduce enabling distributed processing on clusters; Stonebraker argues it is architecturally inferior to distributed database systems
As defined in Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
Where it comes up · 4
- Databricks: Data Based Decisions3 claims
- Databricks: From Data to Decisions2 claims
- Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]2 claims
- Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker2 claims
Most important claims · 9 of 9Search all claims for “Hadoop” →
- Spark originated at UC Berkeley when the machine learning group competing in the Netflix recommendation contest found Hadoop too slow for their highly iterative algorithms, which reload and rewrite data each pass, so the systems group proposed keeping the data in memory and iterating there.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- MapReduce was fundamentally inefficient compared to distributed database systems, and Google's adoption of eventual consistency as a concurrency control mechanism was architecturally flawed because it cannot maintain integrity constraints like stock levels remaining non-negative.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Cloudera, built on Hadoop's batch processing, was the dominant big-data platform until Databricks overtook it by accelerating in-memory processing, which makes analysis much faster and more versatile than organizing data one book at a time.Databricks: Data Based Decisions
- In the early 2010s, companies stored large volumes of data on the assumption it was valuable, but then hit Gartner's trough of disillusionment because they could not extract value from it, and that unmet need expanded Databricks' addressable market.Databricks: From Data to Decisions
- Hadoop with MapReduce became the first operating system for the distributed data center and enabled processing of any amount of data in parallel, but every computation had to be expressed through just two functions, map and reduce, which made programs cumbersome and limited writing to sophisticated programmers.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- MapReduce was ridiculously inefficient, and Stonebraker (along with others like Dave Dewitt) understood distributed databases well enough to beat the heck out of Hadoop with a distributed database system, which was the central claim of their 2011 paper.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Earlier data lake companies such as Cloudera, built on Hadoop, failed to establish the data lake market because the underlying technology was simply not good enough, not because the demand was absent.Databricks: From Data to Decisions
- Cloudera was the biggest data and infrastructure company at the time, built on Hadoop and poorly suited to real-time streaming, and Databricks overtook it early on, moving on to provide a value-added layer atop Microsoft and Amazon clouds.Databricks: Data Based Decisions
- Being Hadooped describes getting locked into the Hadoop-based batch-processing ecosystem, which was great for structured data but poorly suited to real-time streaming, and Cloudera had been the dominant player built on that architecture.Databricks: Data Based Decisions