Spark
A distributed computing framework for large-scale data processing, created by Matei Zaharia, providing APIs in multiple languages and support for interactive and batch processing
As defined in Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
Other definitions
Chilupati's term for triggering event that converts a flammable item into a crisis; examples include new competitor, commodities price movement, earnings slowdown, or market shock
No. 1 Forensic Accountant: The Coming AI Collapse | Anthony Scilipoti
Where it comes up · 4
- Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]6 claims
- Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development1 claim
- No. 1 Forensic Accountant: The Coming AI Collapse | Anthony Scilipoti1 claim
- Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker1 claim
Most important claims · 9 of 9Search all claims for “Spark” →
- Spark originated at UC Berkeley when the machine learning group competing in the Netflix recommendation contest found Hadoop too slow for their highly iterative algorithms, which reload and rewrite data each pass, so the systems group proposed keeping the data in memory and iterating there.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- Databricks' schedulers from OS researchers couldn't scale to scheduling a million Spark jobs, so they solved the problem by putting all the scheduling data in a Postgres database and running a Postgres application to do the scheduling.Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
- Databricks deliberately open-sourced every element of its platform where lock-in would occur, including Spark for data access, Delta for data quality and performance, MLflow for end-to-end machine learning operations, and Redash for visualizations, so customers choose it for value rather than for lock-in.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- In the early years, the Spark team told incumbent Hadoop-era companies that Spark was 100 times faster, much easier to use, and supported machine learning and real-time computation, but industry ignored it as an academic project from 2009 to 2012.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- A three-stage forensic analysis process identifies risks: (1) understand the business and control environment (compensation structure, life cycle stage, accounting policies), (2) identify flammable items (suspicious metrics that could be problems under certain conditions, like negative cash flow that could be positive sign or negative sign depending on context), (3) identify the spark (trigger event that converts flammable item into crisis).No. 1 Forensic Accountant: The Coming AI Collapse | Anthony Scilipoti
- From 2009 to 2012, Spark spread slowly, and enterprise companies using Hadoop dismissed it as an academic project, concerned that students might leave and leave them with unsupported software.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- Even the spark scenario, in which about four years of progress happens in the first year after automating AI research, would be comparable to the jump from GPT-3.5 to a GPT-6-level model in 12 months.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
- Spark is now only a small portion of what Databricks does in the cloud, and the company's core products have moved to layers such as Delta on top of it.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- The Databricks workflow runs from ingesting data with Spark into a lake, organizing it with Delta Lake for quality and speed, iterating on machine learning models with MLflow tracking and access control, and finally serving predictions to live applications.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]