Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
What this covers
Patrick talks with Ali Ghodsi, CEO of Databricks and a Berkeley researcher, in a conversation that begins with hardware history and ends with how a founder's habits have to change as a company grows. The first half is a technical genealogy. CPUs stalled around 2005, computation moved into data centers, and cloud providers began renting out thousands of machines. Hadoop and MapReduce became the first operating system for that world. Enterprises then filled data lakes and measured success in petabytes rather than business results. Ghodsi also tells how Spark grew out of Berkeley's Netflix-prize machine learning work, which found Hadoop too slow for iterative algorithms. He credits David Patterson's shared open office for seating a machine learning researcher beside Matei Zaharia. Industry dismissed Spark as academic from 2009 to 2012, despite the team's claim that it was 100 times faster.
The middle of the conversation turns to the enterprise. Ghodsi describes data management vendors with governance but no AI, AI vendors with no governance, and the two camps split between Java and Python. His forecast that the Lakehouse Paradigm will merge them is flagged as contested. He explains why Databricks open-sourced Delta, MLflow, and Redash, how its cloud service differs from downloadable open-source versions, and why the free Community Edition lets sales focus on heavy users. He lists five traits of enterprises that succeed with data and AI, and argues that the big, familiar data companies are coasting on roughly decade-old innovations.
The last stretch is about management. Ghodsi describes company phases from product-market fit to scaling to optimization, and says the founder who personally picks up trash must eventually install processes. He argues that continually cannibalizing one's own revenue is Databricks' real edge against the Innovator's Dilemma.
Ghodsi argues that distributed computing was forced by the end of CPU speed gains, that the bottleneck has shifted from moving data to bridging the divide between data management and AI, and that open source and an end-to-end platform are how enterprises escape lock-in and turn data into value.
- CPU stagnation around 2005 forced computation onto distributed data centers, and networks later became fast enough to make moving code to data unnecessary
- Enterprises avoid proprietary lock-in, so open-sourcing every lock-in point is both a customer requirement and Databricks' competitive moat
- The next barrier is organizational and technical: IT data management and line-of-business AI teams are split, and successful firms unify them under one leader
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Spark originated at UC Berkeley when the machine learning group competing in the Netflix recommendation contest found Hadoop too slow for their highly iterative algorithms, which reload and rewrite data each pass, so the systems group proposed keeping the data in memory and iterating there.
“And we said, 'Yeah, memory is getting pretty cheap, why don't we just figure out a way where we can just load all of this data into the memory and then we can just do the iterations super fast in memory'”
Around 2005 CPU speeds stagnated at the so-called Moore's Wall, so computers stopped getting faster and computation and data moved into data centers with hundreds or thousands of machines, because demand for processing kept growing exponentially.
“around 2005, the CPU speeds stagnated, computers were not getting any faster anymore. Computers have basically been three gigahertz since then. So that started meaning that we'd probably have to distribute.”
Falling storage prices, combined with the example of Google extracting value from data, led enterprises to collect everything into data lakes, and early success was measured by data volume (one petabyte, then two) rather than by business outcomes.
“the price of storage kept going down, so it became just cheaper and cheaper to store all this massive amounts of data and no one wanted to throw it away.”
The core innovation of MapReduce was to move the code that processes data to the machines where the data sits, because data volumes were too large to transfer over the network without collapsing it.
“The big innovation of MapReduce, the big thing, if you go back to that time was, we have to move the computations close to the data.”
Once distributed compute became necessary, it became infeasible for each organization to run thousands of machines on-premises, so cloud providers offered computing as a utility, and anyone with a credit card could rent thousands of machines and start storing data cheaply.
“anyone could pretty much with a credit card start renting thousands of machines, this new computer in the cloud, and start storing things on it, very cheaply and get started.”
Networking advanced faster than CPUs, and research from UCSD on configuring data center networks let any two machines communicate at full speed, so by about 2009 to 2011 the network was virtualized and no longer forced computation to be moved to the data.
“they figured out how to configure the networks in these data centers in a way such that any two machines can completely at full speed communicate with each other. You can no longer collapse the network with these heavy computations.”
Hadoop with MapReduce became the first operating system for the distributed data center and enabled processing of any amount of data in parallel, but every computation had to be expressed through just two functions, map and reduce, which made programs cumbersome and limited writing to sophisticated programmers.
“all the computations would have to be described in terms of just two functions, one called map, and one called reduce. Which made it very cumbersome and complicated to write programs for this.”
Falling storage prices made it cheap to keep massive volumes of data, so organizations stopped deleting data and built large data lakes, even without a clear use for it.
“the price of storage kept going down, so it became just cheaper and cheaper to store all this massive amounts of data and no one wanted to throw it away”
Early big data processing ran mostly on disk because memory was expensive, so each computation step loaded data from disk, processed it, and wrote it back, which was far slower than in-memory processing, while CPUs were the component that stagnated.
“Back then, memory was expensive. Again, that's something that's changed a lot. Memory has gotten way cheaper, storage gotten way cheaper, the CPUs are the things that have stagnated”
The most exciting future development is the lakehouse paradigm, merging data lakes used for AI with data warehouses used for data management into one platform, which would give enterprises the end-to-end capabilities that Silicon Valley tech companies had in the early 2000s.
“the merger of two markets, two platforms that are going to merge technology for AI, and data science is going to merge with technology for data warehousing and data management.”
A company passes through distinct phases, product-market fit, scaling the machinery with experienced operators, and then optimization with processes for thousands of employees, and traits that help in the first phase, such as acting as an owner who personally cleans up, can hurt in the optimization phase, where the fix is process rather than personal intervention.
“sometimes the things that you have to be good at one phase actually hurts you now. So you have to transform at each phase.”
Berkeley's collaborative model, in which professor David Patterson gave up his own office to create an open shared space for systems and machine learning researchers, produced Spark because a statistician-machine-learning researcher sat next to systems researcher Matei Zaharia.
“So me and my fellow professors will just give up our own rooms, we don't need the room, give those up, and let's build an open space. Where we're all working together.”
The founder habit of personally cleaning up problems, like picking up trash in the kitchen, is right early on but wrong at scale, where the better response is to investigate root causes and install processes so problems never recur.
“If you see trash in the kitchen, even if you didn't put it there, you clean it up, it's your company, you're an owner. Actually this phase, I would say, don't pick it up”
Open-source projects like Spark are on-premises products with versions that users download, whereas Databricks is a cloud SaaS business that runs software for customers and upgrades it continuously, so customers never choose a version, the way Google Search has no version.
“Cloud software like Google Search doesn't have a version. When you go and start on Google, I don't ask you which version did you search on.”
Data management vendors are strong on governance, reliability, and speed but have no built-in AI, while AI and ML vendors lack governance, security, and data management capabilities, and the two sides also differ in languages, with Java favored by IT and Python by the line of business.
“there's data management companies that are awesome at data management... but they have zero AI built into that, to their technology.”
Databricks deliberately open-sourced every element of its platform where lock-in would occur, including Spark for data access, Delta for data quality and performance, MLflow for end-to-end machine learning operations, and Redash for visualizations, so customers choose it for value rather than for lock-in.
“pretty much every element of the Databricks platform where there would be a lock-in we've opened it up as an open-source project.”
Enterprises prefer open source because they have been burned since the 1980s by proprietary vendors who lock in data, become complacent, let original founders leave, and then raise prices because customers are too locked in to move.
“enterprises don't want to lock themselves into proprietary software, they've gotten burned since the '80s.”
Databricks' free Community Edition lets hundreds of thousands of data scientists sign up with a credit card without talking to sales, and the sales team engages only accounts with heavy usage, because strategic projects worth tens of millions require executive investment.
“our sales team start engaging with those that have a lot of usage on the platform. And that's when we start getting more strategic.”
A company must continually cannibalize its own revenue base, even when that hurts existing products, because a company that resists disruption of its own products will lose to innovators, which is why Databricks' database business is now a small share.
“Even if you face an innovator's dilemma, and it might hurt your existing revenue base, that's okay, cannibalize it and come up with the next thing.”
In the early years, the Spark team told incumbent Hadoop-era companies that Spark was 100 times faster, much easier to use, and supported machine learning and real-time computation, but industry ignored it as an academic project from 2009 to 2012.
“Please take this. This is 100 times faster. It's much, much, much easier to use. And it supports machine learning, it supports real-time computations.”
The big, well-known data company names have innovations that are roughly 10 years old and are being milked through go-to-market machines, so the genuinely interesting innovation in machine learning and data tooling is found in startups working on making AI more productive and real-time.
“They've had some innovation that's typically 10 years old, they're milking it with a go-to-market machine that's broadening, getting that tech out to bigger and bigger markets”
Companies that bring a big dataset to Databricks must first identify a concrete business use case with clear impact, because a vague wish to do AI with existing data is not enough and the most valuable projects are the ones tied to a specific business outcome.
“So understanding that business use case shouldn't be undervalued. If they're just showing up saying, 'Hey, we have a bunch of data and we want to do some cool stuff, some AI,' then that's usually when we say, 'Well, what does it really want to do?'”
Owning the full pipeline from data ingest through to production use cases is an advantage because it allows end-to-end optimizations that point solutions cannot make, although competitors could replicate the approach and most companies only cover a sliver of the pipeline.
“we actually own the whole pipeline from the data coming in all the way to the use cases, so it's vertically integrated that way.”
Databricks' main competitive advantage is building innovative DNA into the company so it keeps cannibalizing its own revenue, even under an innovator's dilemma, because it continuously reinvents what it sells, as Databricks did when Spark gave way to Delta.
“Even if you face an innovator's dilemma, and it might hurt your existing revenue base, that's okay, cannibalize it and come up with the next thing.”
Companies doing data and AI well consolidate responsibility under one leader or a center of excellence, because splitting IT from the line of business leads to politics between security and reliability goals and business-value goals, and the title such as chief data officer matters less than shared goals.
“The wrong way to do it is to completely separate the line of business and IT, IT is responsible for the data, and the line of business is responsible for these projects”
Enterprises that succeed with data and AI share five traits: a unified leader who can bring the organization along, open source to avoid lock-in, building AI into data collection from the start, betting on the cloud rather than on-premises data centers, and a multi-cloud strategy hedging across three or four large cloud vendors.
“Those are some of the ingredients we see really work well for the companies that are successful with data and AI.”
The biggest barrier in computing today is the split between IT-run data management, which is conservative and governed but has no AI, and line-of-business AI teams, which are close to use cases but lack governance and reliability, and these two groups often do not report to the same executive.
“The biggest wall is between data and AI. And you see it both organizationally inside enterprises, but you also see it in the tech and just the preferences people have.”
Organizations should design for AI from the outset rather than collecting large amounts of data first and planning AI projects later, because retrofitting AI onto data collected without use cases creates the same stalled outcomes.
“building with AI in mind from the beginning, don't just collect a lot of data and brag about how much data you've collected and figure out later”
From 2009 to 2012, Spark spread slowly, and enterprise companies using Hadoop dismissed it as an academic project, concerned that students might leave and leave them with unsupported software.
“No, this is just an academic project. What if these students quit to start their own company or something?”
Spark did not get industry adoption as a purely academic project at UC Berkeley, so by 2012 the team concluded it would need a company around it to gain traction, which led to forming Databricks.
“2012, we had it, and was like, 'If we want people to adopt this, we probably have to start a company around it because it's not going to work sitting here at UC Berkeley.'”
A project that can save a company $100 million is not something an individual engineer adopts with a credit card; it requires investment and resource commitment from company leadership before it becomes a paid enterprise relationship.
“A really strategic project that can help a company save $100 million, that's typically not a project that some engineer swipes a credit card”
Ghodsi forecasts that within 10 to 15 years every business unit in every company will use data and AI strategically, which makes the total addressable market for enabling the whole organization gigantic, so Databricks invests in democratizing AI across the organization rather than serving single research teams.
“eventually 10 years, 15 years from now, every company will have every BU, be using data and AI in a strategic way.”
Genetic testing revealed a marker predisposing Ghodsi's 12-month-old son to a particular cancer, and screening every three months allowed early detection, a capability that would not have been possible ten years earlier.
“we actually were able to detect this cancer by screening him every three months because we had done genetic testing on him and we had found the genetic marker that actually says that he's highly predisposed to this particular cancer.”
For Shell, predicting equipment breakdowns in advance lets it replace parts before failure, which saves hundreds of millions of dollars and benefits the environment and employees.
“So for a company like Shell, it's being able to predict its equipment breaking down in advance. If they can do that, then it can replace those parts in advance, that saves them hundreds of millions of dollars”
Computing a simple average of clicks across a petabyte spread over 100 machines would require moving all the data to one machine, which would crash the network; Berkeley's early Spark work did exactly this and took down parts of a network.
“we did that at UC Berkeley, we were developing the first versions of Spark, we crashed parts of that network. They called us up and said, 'What are you doing?'”
Building a machine learning model is highly iterative, typically requiring hundreds of cycles in which practitioners test a prediction, find accuracy lacking, augment with more or purchased data, and retry.
“It's a highly iterative task to come up with a machine learning model. Machine learning folks usually don't just sit there and then come up with a predictive model and they're done, they actually typically have to iterate on it hundreds of times.”
Before distributed computing, the way to solve large computation problems was to buy a very expensive supercomputer with enormous memory, as Twitter did in the very early days with one giant machine processing all tweets.
“Early 2000, you would buy a big supercomputer and solve a lot of the problems. In fact, I remember Berkeley, we were with Twitter in the very, very early days. They had one giant machine, would process all the tweets.”
The first use cases for distributed data processing were crawling the web to build indices and collecting click logs of user behavior, which enabled large-scale computations such as deciding which ads to show, that were impossible before.
“The first use cases of this were largely, it started with crawling the web and building indices for all the stuff that you crawled, all this large data account. Other use cases were click logs”
The original assumption of distributed data systems, that network bandwidth is the scarce resource and code must travel to data, has become obsolete because networks and cloud infrastructure now outpace data growth.
“that happened around 2009, 2010, 2011.”
Ten years earlier, successful tech companies organized data and application teams differently and used one technology stack for both, which is the model that today's enterprises are failing to replicate.
“this is not how it was done in the former tech companies 10 years ago. They were organized differently and they were using one tech stack for both.”
Memory is orders of magnitude faster than disk but is volatile, losing data on reboot, and a machine typically has much less memory capacity than disk storage.
“Memory, when you reboot, it goes away, but the difference is, it's much, much, much faster. So it's orders of magnitude faster”
The biggest lesson for a CEO is how important and how slow it is to build trust with leaders, because as the organization grows the CEO cannot do everything alone and needs strong leaders they can trust, which cannot be done overnight.
“For me, probably the biggest lesson is how important trust is and building trust with leaders, nothing you can do overnight.”
IT teams typically use Java and want production-ready systems, while line-of-business teams use Python, so the technology preferences of the two groups reinforce the organizational split between data management and AI.
“Typically, you use something like Java on the IT side, that's what they like and they want it to be production ready.”
Around 2005 to 2006, a few large internet companies with advertising businesses were already using collected data to optimize ad targeting, while most other enterprises did not know what to do with their data.
“the four tech companies knew, they had typically an ad business, they were collecting the data, they were optimizing how to show ads to people. The rest of the enterprises didn't really know what to do with it”
Comcast uses voice-enabled remote controls where users speak commands like asking for the weather, which is an example of AI-driven consumer prediction in production.
“For a company like Comcast, it's the use case where they have remote control and it has a voice button”
Transforming enterprises into data-native organizations is hard because they have 30, 40, or 50 years of history, and choosing the right leader who understands how to bring the organization along is critical to that change.
“because these enterprises have 30, 40, 50 years of history. So how do you transform them and make them adopt this new data native approach to things?”
The title of chief data officer is unimportant; what matters is picking a leader who can bring long-established enterprises with decades of history to a data-native approach.
“if you have a chief data officer, that's great, the title isn't important... picking the right leader”
Although Databricks runs Spark in the cloud for customers, Spark is only a small portion of what Databricks does today, because the company runs many other services on customers' behalf and automates them away.
“And Spark is a small portion of what we do today at Databricks, in the clouds, on behalf of our customers.”
Over the past year and a half, several other very large companies have begun endorsing and talking about the lakehouse pattern, which Databricks had championed for years.
“We've been saying it for many years, but in the last year or so, a year and a half, we've heard a lot of other really large companies also get behind this”
Enterprise leaders now recognize data and AI as essential because they fear a Silicon Valley startup will disrupt them the way Uber disrupted taxi medallions, Airbnb disrupted hotels, and Netflix disrupted Blockbuster.
“If we don't, just around the corner there might be a startup, probably out of Silicon Valley, that's completely tech-driven, it has thousands of engineers and they're just going to disrupt our business.”
Databricks is a seven-year-old company with about 1,700 employees that helps enterprises run machine learning, AI, and data SaaS on their massive datasets, because most enterprises lack the thousands of engineers that Silicon Valley tech companies used to build custom data platforms.
“We're a 7-year-old company, we have about 1,700 employees, and we help enterprises to take massive amounts of data and do machine learning, AI, and data SaaS with that data.”
Spark is now only a small portion of what Databricks does in the cloud, and the company's core products have moved to layers such as Delta on top of it.
“Spark is a small portion of what we do today at Databricks”
Regeneron used Databricks to find a particular gene responsible for chronic liver disease, which can then support drug testing and drug development to cure such diseases.
“For instance, they've found the gene that's responsible for chronic liver disease in Databricks. And then they can do drug testing and develop drugs to cure those diseases.”
A new category of data-driven applications lets users interact with data directly in the user interface and embed machine learning under the hood, which is much richer than classic BI dashboards built from histograms, dropdowns, and clickable graphs.
“Here you can actually leverage machine learning under the hood if you want it to. It's all interesting.”
Delta Lake, Databricks' most resource-intensive project, adds structure and quality to data lakes so they do not degenerate into a data swamp and can be accessed quickly, including in streaming real-time form.
“the biggest innovation and the project that Databricks is spending most of its resources on it's the Delta project... That's where you organize your data so you just don't have a data swamp”
Running on-premises data centers is now obviously too slow and inflexible for data work because it leaves companies stuck with equipment, and the COVID-19 pandemic made most people realize this.
“Running your own on-prem data centers at this point, it's probably obvious, it's going to be slow, you're going to be stuck with all the equipment. I think most people have realized that now with the pandemic.”
Early market objections that Spark needs too much memory, works only for some machine learning use cases, or handles only real-time computation were false, and educating the market on these misconceptions was the first two to three years' struggle for Databricks.
“Spark doesn't work if you don't have enough memory, which was not true. Spark is just good for some machine learning use cases, not others.”
The Databricks workflow runs from ingesting data with Spark into a lake, organizing it with Delta Lake for quality and speed, iterating on machine learning models with MLflow tracking and access control, and finally serving predictions to live applications.
“Typically, they have data sets, getting access to that data typically happens with Spark. ... The next step is to make sure that that data actually has high quality and a structure”
Databricks' first customer used AI and predictive models on real-time streaming data to decide how video traffic should be routed across the internet and to improve video quality.
“I think the first use case Databricks was a company that's trying to understand how video screens were being actually downloaded on the internet and how they could increase the quality of this.”
Patrick argues that as with Twilio, data teams are shifting from back-of-house to front-of-house strategic roles, and he asks whether chief data officers and data attention are rising in the same way.
“whether or not the data world is going through that same transition”
Companies that began deploying machine learning, such as Uber, use it for hundreds or thousands of internal use cases, including predicting prices, routes, food readiness, and matching riders for carpooling.
“A company like Uber predicts the price screen, it predicts the route for you. It tells you when the food is going to be ready, puts more people in the same route”
Cloudera received a 700 million dollar investment from Intel, one of the largest that year, which illustrated how much the market was rallying around Hadoop while Spark struggled for attention.
“I remember Cloudera had just got a $700 million investment from Intel. So it was one of the largest investments that year”
Dash is cited as an existing technology for building data-driven applications that interact with and visualize data, though the example adds little beyond the category definition.
“There are already technologies like OK, like Dash, where you can actually build data-driven applications”
Databricks' culture is highly teamwork-oriented, and its four cultural principles include 'teamwork makes the dream work,' inheriting the Berkeley collaborative model.
“One of our four cultural principles is teamwork makes the dream work. So it's a very highly teamwork-oriented culture.”
Ghodsi is most grateful to healthcare workers and to technology that enabled genetic detection of his son's cancer, so the kindest thing he has experienced came from medical care supported by technology.
“I'm thankful to all the health care workers, I'm also thankful to technology that actually enabled us to be able to actually spot this gene in advance”