Databricks' schedulers from OS researchers couldn't scale to scheduling a million Spark jobs, so they solved the problem by putting all the scheduling data in a Postgres database and running a Postgres application to do the scheduling.
causalpending
Speaker
Mike StonebrakerEvidence Quote
“He said there was no we tried all the all the schedulers written by the OS folks and they they couldn't they didn't scale. So we put all the scheduling data in a Postgress database and basically a Postgress application was doing scheduling”
Source
Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker— Ryan PetermanCreated: 8/12/2026, 6:09:46 PM
Your notes
Loading notes...