Apache Spark
An open-source in-memory distributed computing engine originally built at UC Berkeley for fast iterative processing such as machine learning.
As defined in Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
Where it comes up · 3
- Databricks: Data Based Decisions4 claims
- Databricks: From Data to Decisions1 claim
- Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]1 claim
Most important claims · 6 of 6Search all claims for “Apache Spark” →
- The cloud paradigm shift of scale-out architecture allowed use of commodity hardware to address larger amounts of compute, storage, and data processing, which enabled the data explosion and made distributed data processing feasible, as Apache Spark did by applying distributed compute to data processing.Databricks: From Data to Decisions
- Databricks charges customers by usage in increments as small as one second on a pay-as-you-go basis, with revenue coming from three product lines: the Spark-based analytics engine, the lakehouse, and an AI product added with the Mosaic acquisition.Databricks: Data Based Decisions
- Databricks took open-source Spark, built a proprietary, more efficient version with faster performance and lower cost, and by reducing infrastructure spend so much, the software effectively felt free, creating value that exceeded what open source alone could capture.Databricks: Data Based Decisions
- Spark did not get industry adoption as a purely academic project at UC Berkeley, so by 2012 the team concluded it would need a company around it to gain traction, which led to forming Databricks.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- Spark originated in the Berkeley lab of Ion Stoica, and Databricks was founded around 2009 by roughly seven or eight people, including Matei Zaharia, all of whom are still executives at the company.Databricks: Data Based Decisions
- Spark's open-source platform came out of UC Berkeley, where Ion Stoica's lab produced the technology that became Databricks.Databricks: Data Based Decisions