323 | Jacob Barandes on Indivisible Stochastic Quantum Mechanics
What this covers
Jacob Barandes, a physicist and philosopher at Harvard, proposes a radical reformulation of quantum mechanics that discards the wave function entirely. Rather than invoking Many-Worlds or hidden variables à la David Bohm, Barandes argues that quantum theory can be reconstructed with only real configurations—particles or fields—evolving under stochastic laws that are non-Markovian. The central move is relaxing the Markov assumption, the assumption that future evolution depends only on present state, not history. Barandes contends this recovery restores empirical adequacy while shedding metaphysical excess baggage. The conversation with Sean Carroll traces both the historical roots of this idea—why the 1920s quantum revolution discarded classical ontology and probability together when perhaps only the Markov assumption needed rethinking—and the technical machinery that makes it work: indivisible stochastic processes generating conditional probabilities that reproduce quantum predictions, with decoherence reappearing as a mathematical phenomenon called a division event.
The discussion ranges widely over the landscape of interpretations and their failures. Bohmian mechanics, though elegant for non-relativistic systems of fixed particle number, becomes unwieldy and arguably unworkable in relativistic quantum field theory—a weakness that has persisted for roughly seventy years. Hugh Everett's Many-Worlds approach, which strips away Bohm's hidden particles, still faces problems: the derivation of probability grows increasingly baroque across generations (from Everett's one-page argument to Wallace's eighty-three pages), raising circularity risks, and the very assumption of a universal wave function governed by unitary evolution extrapolates wildly beyond what systems actually exhibit. Barandes sketches how his indivisible stochastic framework avoids these snares—it is local in space, satisfies Bell's theorem, and requires no speculative metaphysical furniture. The episode also covers deeper philosophical ground: Heisenberg's deliberate ontological pivot in 1925, the historical accident that Kolmogorov axiomatized probability after quantum textbooks already existed, and how physicists lost touch with the philosophical training that figures like Albert Einstein took for granted. Throughout, the conversation emphasizes how the 1920s generation may have narrowed their options unnecessarily by bundling three separable choices—kinematics, dynamics, and probability—into a single revolution.
Barandes argues that quantum mechanics can be reformulated with no wave function at all—only real configurations (particles or fields) evolving under indivisible, non-Markovian stochastic laws—and that abandoning the Markov assumption rather than classical ontology or probability recovers all quantum predictions while avoiding the metaphysical baggage of Many-Worlds and the practical failures of Bohmian mechanics.
- The historical 1920s shift discarded classical kinematics, dynamics, and probability together, but only the Markov assumption needed to be relaxed.
- Indivisible stochastic processes define equivalence classes of non-Markovian realizers whose simple conditional-probability laws reproduce quantum predictions, with decoherence reappearing as 'division events'.
- Both Bohmian mechanics (fails for relativistic field theory) and Everett (Stone Soup of speculative metaphysical hypotheses, probability/circularity problems) fall short on empirical adequacy and parsimony.
Non-Markovian stochastic dynamics with classical ontology recover quantum predictions without assuming fundamental wave functions or speculative metaphysics.
- In the double-slit experiment, if you assume the process is divisible at the walls (a Markov-like assumption) you get the classical no-interference pattern, but if you simply drop the divisibility assumption while keeping the particle always in one location, the generic prediction is an interference pattern—and adding an interacting qubit at the slits produces a division event that mathematically reproduces decoherence and restores the classical pattern.
“If you do not assume the process is divisible, if you simply drop that assumption, you will generically get an interference pattern on the other side. The particle is always in one location, it's never in both holes.”
- Quantization—replacing classical variables with operators on a Hilbert space—is not a fundamental physical process (nature does not start classical and decide to become quantum); it is a heuristic, trial-and-error method for guessing a more fundamental quantum theory from a classical description, and in the indivisible picture quantization instead means keeping classical ontology while choosing new non-Markovian indivisible dynamics.
“The quantization recipe is not a fundamental statement about what nature is doing, nature isn't classical, and then it just decides sometimes to quantize and become quantum mechanical.”
- In the indivisible stochastic theory, if particles are the modeled ontology, the electron is in one and only one definite location at every moment in time—forming a real trajectory—even though that trajectory is in general nowhere continuous and not smooth, so the particle can jump from one place to another with no path in between.
“If you ask, is the electron somewhere, some specific place at every moment in time, the answer is yes. And the totality of all those locations is a trajectory”
- Barandes's approach proposes that the 1920s physicists thought too narrowly about laws; one can retain classical-looking kinematics (particles or fields, real arrangements) and classical notions of probability, and recover an empirically adequate quantum theory simply by adopting a new, more general kind of law that was unavailable then—so there is no fundamental wave function, only real stuff that moves stochastically.
“maybe they could retain the classical kinematics, the classical-looking picture of reality of the stuff... and classical notions of probability, but they just needed a new set of laws, a new kind of laws that were unknown at the time.”
- While preparing a class in 2022, Barandes inadvertently recovered full quantum theory from a simple classical stochastic process with no gap and no gerrymandering, and on investigating discovered he had unknowingly given up the Markov assumption—and found essentially no prior serious effort to formulate quantum theory with classical probability and ontology but non-Markovian laws.
“remarkably what happened was I just got quantum theory. There was no gap... And after some effort to understand what had happened, I discovered that I'd inadvertently given up the Markov assumption.”
- Unlike the Everettian claim that the fundamental ontology of nature is quantum states in Hilbert space, Barandes adopts a philosophically modest stance: the ontology you plug into an indivisible stochastic model is theory-dependent—particles if modeling particles, fields if modeling fields—and we do not yet have the right to claim we know the true fundamental ontology of nature.
“There's not one particular ontology an uber ontology that is the ontology of everything 'cause we don't know what it is yet.”
- Given any inevitably finite amount of experimental data, there are always infinitely many theories that will fit it (the underdetermination of theory by data), so one must appeal to additional criteria such as simplicity or predictive power to select a theory; Barandes worried quantum theory might be a case where the theory itself is so intricate it overdetermines the interpretation, leaving no fully workable interpretation.
“given any inevitably finite amount of experimental data, there are always infinitely many theories that will fit it, this is called the underdetermination of theory by data.”
Bohmian and Many-Worlds interpretations each fail at least one of the four essential requirements an adequate quantum theory must satisfy.
- Barandes holds four non-negotiable requirements for an acceptable quantum theory: (1) empirical adequacy—it must get the experimental predictions right; (2) no ambiguous predictions in principle situations like Wigner's friend; (3) the ability to tell at least a schematic story of how the macroscopic world emerges from the microscopic; and (4) avoiding an endless Stone-Soup list of speculative metaphysical hypotheses—and his indivisible approach is designed to meet all four while Bohm fails (1) and Everett strains (1) and (4).
“The first was empirical adequacy... The second is that it shouldn't deliver ambiguous predictions in principle situations, and Wigner's friend is a good example... it should be able to let us tell a kind of picture of how the macroscopic world is emergent from a micro world... And then the final requirement is that we don't have an endless list of extra empirical, speculative metaphysical hypotheses”
- David Wallace argues that Bohmian/de Broglie-Bohm models not only are inelegant but fail at relatively elementary tasks such as explaining why the sky is blue (Rayleigh scattering, a relativistic light-charge interaction), and that the failure to accommodate such problems after roughly 70-100 years is a serious practical issue, not merely an aesthetic one.
“He wrote a paper and gave a bunch of talks called The Sky is Blue, why quantum theory... is not underdetermined in which he argues... that they don't succeed at important but relatively elementary tasks like explaining why the sky is blue. That comes from Rayleigh scattering”
- Pilot-wave/Bohmian hidden-variable theories work beautifully and elegantly for fixed numbers of non-relativistic spinless particles, but become extremely complicated and arguably unworkable when extended to interacting relativistic quantum field theories with fermions, devolving into ad hoc admixtures of pilot-wave and stochastic dynamics that are reverse-engineered to match known predictions.
“in the end, you got these sort of complicated admixtures of pilot wave dynamics and stochastic dynamics... the rules seemed now kind of gerrymandered, kind of ad hoc. Like we knew the predictions we needed, and now we were just gonna reverse engineer the rules”
- The Many-Worlds approach falls prey to what Barandes calls the Stone Soup problem: although advertised as axiomatically simple (just quantum state plus Schrodinger evolution), deriving the Born rule actually requires stacking many hard-to-justify speculative metaphysical hypotheses—richness axioms, rationality axioms, assumptions about valuing uncountable future copies of oneself, orthogonality of distinct possibilities—each of which lowers one's Bayesian credence in the theory.
“I began to develop the sense that it falls prey to what I now call the Stone Soup problem.”
- Everett's approach is essentially Bohm's approach with the localized particles dropped, in an attempt to simplify the axioms—keeping only the quantum state and Schrodinger evolution without the extra hidden-variable ingredients that Bohm's theory required.
“Essentially what Everett did was drop the particles, just drop them.”
- Decoherence by itself is not enough to resolve the measurement problem, as evidenced by the fact that Bohm—who discovered and clearly understood decoherence—did not think it sufficient and went on to introduce hidden variables, and because Bohm's own 1952 treatment of measurement still required decoherence to work.
“the argument I would make to anyone who says all we need to explain quantum theory and solve the measurement problems is decoherence, is just look at Bohm. Arguably Bohm understood decoherence... and he didn't think it was enough to resolve the measurement problem, and I think that's telling.”
- The attempts to derive the Born rule (probability) from the Many-Worlds picture grow longer and more technically intricate over time—Everett's argument was one page, Deutsch's 15 pages, Wallace's 83 pages—and this escalating complexity raises the risk of smuggling in the assumption you are trying to prove, threatening logical circularity.
“Hugh Everett, his argument for probability in many worlds is basically one page. David Deutsch's argument is 15 pages and the mathematics is much more intricate. And David Wallace's is 83 pages.”
The Markov assumption has dominated physics since Newton but is not fundamental; dropping it while keeping classical probability and kinematics yields quantum theory.
- The Markov assumption—that the probabilities for what a system does next depend only on its present state, not its history—has been baked into how physics formulated fundamental laws since Newton (F=ma), Maxwell, and even the Schrodinger equation and Born rule, and physicists have been spoiled by its success for centuries even though some non-fundamental systems are known to be non-Markovian.
“This is called the Markov assumption... it's baked into how we think about laws of physics... we've been very spoiled because the Markov assumption has served us incredibly well for centuries.”
- Earlier stochastic approaches to quantum theory—by Fritz Bopp, Imre Fenyes, and most famously Edward Nelson (Nelsonian stochastic mechanics)—all assumed the Markov property, requiring them to solve the Schrodinger equation and plug it into complicated forward- and backward-in-time stochastic differential equations that look reverse-engineered, which is why they felt as gerrymandered as Bohmian generalizations.
“All of them assume the Markov assumption. They all talked about the moving parts, the particles, moving around according to laws that just take the present state of the system”
- There is no compelling reason to believe the entire universe is described by a well-defined universal wave function evolving unitarily; systems get progressively further from unitary evolution as they grow larger (becoming open systems), and the assumption that this trend suddenly reverses at the scale of the whole universe—while also neglecting gravity/general relativity—is a wild and unjustified extrapolation that the Everett approach sensitively depends upon.
“it would require a pretty wild extrapolation of the Schrodinger equation from where we know that it works... we're also neglecting general relativity, we're neglecting gravity”
- The correspondence between quantum systems and a large class of non-Markovian systems can be read in reverse: a complicated real-world non-Markovian system with no h-bars can be represented using the tools of quantum theory, allowing one to apply a century of quantum machinery (renormalization group, effective field theory, stationary phase methods) to entirely new kinds of non-quantum systems.
“if I can represent it using the tools of quantum theory, then we can use all the mathematics we've developed over the past 100 plus years for quantum theory and apply them to these new kinds of systems.”
Division events in indivisible stochastic processes map onto decoherence mathematics and objectively ground the emergence of classical behavior from microscopic dynamics.
- As systems are built from more and more particles, the weird discontinuous particle behavior at small scales aggregates into apparently continuous motion at mesoscopic and macroscopic scales via collective degrees of freedom, and that motion becomes essentially Markovian and in the simplest cases deterministic like Newton's laws.
“as systems get bigger and bigger, in a sense of putting together more and more and more particles, the weird discontinuous behavior of the particles begins to look on larger scales like continuous motion.”
- A division event—a moment where the indivisible evolution can be divided because the system has revealed its configuration to another system via classical marginalization over an environment—maps directly onto the mathematics of decoherence in the Hilbert-space picture, though with different metaphysics (no branches), and these division events are as objectively well-defined as decoherent branches.
“You can show that what looks like decoherence in the usual Hilbert space picture is just the generation of a division event in this picture. But of course, the metaphysics is very different, we're not making branches”
- Unlike the Everett approach where branches themselves are fuzzy and depend on a system/environment split, in the indivisible approach the electron is always definitely in exactly one place—what is fuzzy is only the effective laws used to describe it, and having approximate effective laws is unremarkable since every effective field theory in physics has limited accuracy.
“In this picture, the fuzziness is only a fuzziness in the laws... what is not fuzzy is that the electron is always in one and only one place, that's not fuzzy. What's a little bit fuzzy is the laws”
- An indivisible stochastic process is defined by a few simple rules that do not pin down one fully-specified history-based non-Markovian process but rather an equivalence class of infinitely-complicated 'realizers'; the process is called indivisible because it lacks the property that time evolution from an initial to a final moment can always be subdivided through arbitrary intermediate times.
“It defines a collection of distinct such non-Markovian processes, each one of which to borrow language from my colleague Alex Meehan... calls realizers.”
Wave functions and Hilbert spaces are useful mathematical tools analogous to Hamilton-Jacobi functions, not descriptions of fundamental physical reality.
- The intuitive picture of a particle's Schrodinger wave going through both slits in physical 3D space is actually very misleading, because for N particles the wave lives in 3N-dimensional configuration space (six dimensions for two particles), so the familiar wave-through-holes story only works for a single particle and breaks down completely for multi-particle systems.
“the pattern you get actually looks like the projection down to 3D space of a wave pattern in six dimensional space... the waves live not in physical space, the waves live in configuration space”
- In the indivisible approach, wave functions and Hilbert spaces play a role analogous to the Hamilton-Jacobi function in classical mechanics—convenient mathematical tools for fast predictions that no one regards as physical objects—so one should still use standard quantum tools for complicated systems even though the fundamental ontology is the moving parts, not the wave function.
“we don't think that the Hamilton-Jacobi function is like a physical object... I'm saying that the wave functions and Hilbert space appurtenances... play a similar kind of a role. They're often very convenient to use”
- In the indivisible picture, the electrons in a hydrogen molecule are real point particles zipping around in dynamic equilibrium, whose individual motion is not empirically accessible but whose probability distribution over time is stationary; they do not radiate because you cannot apply classical electrodynamics to quantum particles—coupling them properly to the quantized electromagnetic field (QED) yields, by isomorphism, the same no-radiation prediction as standard quantum theory.
“The particles are zipping around and they're zipping around in dynamic equilibrium. And their dynamic equilibrium, as an equilibrium looks effectively stationary.”
The indivisible stochastic framework may ground microphysical causation in conditional probabilities, reopening whether causation is fundamental rather than emergent.
- Because the indivisible approach's laws are themselves conditional probabilities—the same ingredients used to cash out causal relationships in causal modeling—it may be possible to ground or develop a theory of microphysical causation in these laws, reopening Judea Pearl's question of whether causation is fundamental rather than merely an emergent artifact (as Pearl argued for non-directional laws like F=ma).
“the laws in this indivisible approach to quantum theory are conditional probabilities. So now you might ask... Could we try to ground or develop a theory of microphysical causation rooted in the laws of a theory like this?”
- Barandes suspects that general relativity's resistance to a fully probabilistic generalization—and by extension the difficulty of quantum gravity—is connected to non-Markovianity, because Einstein's field equation only behaves like a familiar predictive differential equation for nice (globally hyperbolic) spacetimes; developing a fully probabilistic 'probability gravity' first might be an important stepping stone toward quantum gravity.
“my suspicion is that this is connected with the difficulties in developing a theory of quantum gravity... The reason we haven't done that is because of this non-Markovianity”
Quantum mechanics requires rethinking what laws are rather than replacing classical ontology; 1920s physicists adopted too narrow a conception of dynamical law.
- A physical theory has three components: the kinematics (the stuff/moving parts, what philosophers call ontology), the dynamics (the laws governing how ingredients change, what philosophers call the nomology), and the probabilistic side (how probabilities are assigned), and the quantum revolution replaced the classical version of all three with quantum versions rather than discarding the categories themselves.
“you can think of a physical theory in very broad terms as having three components. One component is the stuff, the moving parts... The next component is the rules for those ingredients... And then the third component is how we assign probabilities to things in the theory”
- Classical probability theory is in a sense less classical than quantum mechanics, because Kolmogorov published his axiomatization of probability in 1933, after the foundational quantum textbooks by Dirac (1930) and von Neumann (1932) had already been written.
“Kolmogorov published his treatise Axiomatizing probability in 1933, which is after all the textbooks came out by Dirac and Von Neumann establishing. So interestingly probability theory is less classical than quantum mechanics in the sense that it was formalized after quantum.”
Philosophers trained alongside physicists historically produced deeper physical insight, and mid-twentieth-century foundations work remains under-credited by modern theory.
- Many of the physicists who gave us quantum theory and general relativity, including Einstein, were thoroughly trained in philosophy alongside physics—Einstein read Kant's three critiques by age 16 and ran a philosophy reading group—because mandatory philosophy coursework was standard at the time, and this dual training is visible in their dialogues.
“Einstein was thoroughly trained in philosophy, he read Kant's three critiques by the time he was 16, there was mandatory coursework at the university where he became a physicist, everybody had to learn philosophy.”
- David Bohm, in section 22.8 of his 1951 book Quantum Theory, laid out the destruction of interference in the measurement process—what we now call decoherence—showing branches emerge that stop interfering once a large measuring device is involved, yet he received no Nobel Prize and was exiled to Brazil, so contemporary physicists who rely on decoherence arguably owe a debt to the field of foundations.
“today we call this destruction of interference and this emergence of branches when a big enough system gets involved in the story, we call that decoherence.”
- Teaching is to doing research what stand-up is to being a comedian: you must reformulate your ideas and present them to new eyes repeatedly to stay fresh, connected, and creative, which is why many research ideas (including Barandes's own) have roots in classes the researcher was trying to teach.
“in my opinion teaching is to doing research what stand-up is to being a comedian. You really need to be reformulating your ideas and presenting them to new eyes over and over again in order to keep fresh”
Non-commutativity of observables and emergence of particle number via measurement reflect inherently non-classical properties that resist reduction to classical variables.
- For every quantum system there exist observables whose measurement order matters—measuring one disturbs the value of another so you cannot ask the questions in any order and get consistent answers—and this non-commutativity is an inherently non-classical property that classical commutative algebras of random variables cannot reproduce.
“for every quantum system we know of, there are always observables, things we might wanna know about it that fail to have this property, fail to have this ability to ask the questions in any order, and that non-commutativity is an inherently non-classical property.”
- The Bell-Kochen-Specker theorem (1960s) establishes that it cannot be the case that every observable feature you might measure is fundamentally there waiting to be revealed; at least some observables must emerge through the interaction between the measured system and the measuring device—so on a field ontology, particle number is an emergent measured quantity rather than a count of really-present particles.
“there's a theorem by Bell and Kochen and Specker in the 1960s... that says it cannot be the case that every observable feature you want to measure is fundamentally there waiting to be revealed.”
- Heisenberg's 1925 paper introducing what became matrix mechanics deliberately enacted a philosophical paradigm shift, explicitly stating that physics should abandon trying to think in classical physical pictures (ontology) and instead develop a mathematical formalism connecting observables to other observables, opening the door to modern quantum theory's abstract mathematics with murky or absent connection to ontology.
“the paper begins with a set of statements that any philosopher of science would regard as initiating a philosophical shift in paradigm... we should abandon trying to think of these pictures, and we should develop a mathematical formalism that just connects things we observe with other things we observe.”