Databricks' schedulers from OS researchers couldn't scale to scheduling a million Spark jobs, so they solved the problem by putting all the scheduling data in a Postgres database and running a Postgres application to do the scheduling.

causalpending

Speaker

Mike Stonebraker

Evidence Quote

“He said there was no we tried all the all the schedulers written by the OS folks and they they couldn't they didn't scale. So we put all the scheduling data in a Postgress database and basically a Postgress application was doing scheduling”

Source

Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker— Ryan Peterman
Created: 8/12/2026, 6:09:46 PM

Your notes

Loading notes...