Machine Learning in Investing
What this covers
This is a conversation between host Patrick and Jeremiah Lowin, who explains machine learning the way a practitioner would to investors. He begins with a taxonomy: a two-by-two grid of classification versus regression and supervised learning versus unsupervised, with linear and logistic regression, trees, clustering, and neural networks placed inside it. A minimal neural network, he says, is linear regression followed by a non-linear squashing function, and deep models are those simple blocks stacked.
The middle of the conversation is about what goes wrong. The test set should be used only once the model is finished, and hidden leakage can produce false eureka results. Time series should be split by time, not at random. Stock prices are non-stationary, so models work on returns instead. Overfitting is illustrated by a tank-image model that had really learned to detect night. Random forests are described as a hard-to-argue-against default, though trees struggle with email or images, and K-means clustering can serve as a pre-processing step.
Several positions are more contentious. Lowin calls interpretability overrated, since no one can inspect a billion parameters, though models can be understood in aggregate. He also calls the human the ultimate black box. In finance he stresses that defining the output label is itself a hard credit-assignment problem, and he warns of teams that reach The 42 Problem, getting answers without knowing the question. Over-engineering is his top pitfall, the Butter Knife vs Chainsaw mismatch. He favors models that can say "I don't know," and he is most interested in systems that replicate long-term investors in the mold of Warren Buffett.
Lowin argues machine learning is pattern-matching built on regression that is easy to assemble but hard to train, validate, and aim at the right question, so its real value in investing is enhancing experience-based human judgment rather than replacing it.
- AI systems only discover complex correlations in data, so they are powerful but narrow, like a puppy that performs a few things well
- Overfitting and data leakage are the main dangers, which is why strict train, validation, and test separation matters
- Most ML failures come from over-engineering and from not knowing what question the model is answering
Curated data, transformation, and model mechanics matter more than sophisticated model choice in investing.
- Data acquisition and metadata are the most valuable edge in ML investing, and a dataset everyone can buy for 250 dollars is unlikely to contain more than 250 dollars of value, so curated, transformed, and clean data matters far more than model choice.
“If you're buying the same dataset that everyone else can buy for 250 bucks, I mean, if there's more than 250 bucks of value in there, I'd be shocked.”
- Data that is logarithmically or exponentially distributed, such as GDP that scales with country size, should be transformed back into a linear space before building a model on it.
“that data, usually, you want to transform it back into a linear space before you build a model on top of it”
- To feed emails to a model, text is converted into a bag-of-words representation: a fixed-length vector of ones and zeros where each index marks whether a given word appears, so every email has the same shape regardless of length.
“Bag of words is basically a long list of ones and zeros, and each index in that list represents a word”
- Feature engineering in finance involves more than checking data quality; data must be transformed to fit the model, such as from log to linear scale, properly sanitized, and acquired with statistical best practice.
“Is it transformed from log to linear if necessary? Is it properly sanitized? Is it properly censored?”
- Classification asks whether something is A or B and outputs a discrete label such as spam or not spam, while regression explores a numerical relationship such as a stock's beta to the market.
“Classification means is this A or is this B? We're asking the system basically to output a one or a zero.”
- Stock prices are non-stationary because their level shifts over time, so a model trained at one price level will not transfer; the standard fix is to model returns, which look roughly like a bell curve with heavier tails and are closer to stationary.
“instead of dealing with a stock price, we deal with the stock returns”
- Unsupervised clustering such as K-means can be used as a pre-processing step on high-dimensional data, producing centroids that become new features for a downstream model when the useful structure is unknown.
“maybe as a pre-processing step we run it through a K-means algorithm, we get 10 different centroids out of that”
- A minimal neural network is linear regression followed by a non-linear squashing function, and stacking these simple building blocks into deep models produces powerful inference from an otherwise simple linear process.
“the minimal version of it, arguably, is regression followed by what scientists call a squashing function”
Honest out-of-sample testing requires strict separation of unseen test data from training and tuning.
- Time-series data should be split by time, training through some year and testing forward, because splitting across unrelated sectors or random dates lets the model learn from future correlations, which corrupts the test.
“you want to split up by time. So, you want to train through some year, you want to test from that year forward”
- Models are trained on a training set, tuned against a validation set, and only then assessed on a true out-of-sample test set that was never used for any decision, which is what makes the final performance estimate credible.
“only when you're absolutely done, when you're confident that you have a model that works, then you use your true out of sample test data set”
- The test set must be used only once the model is finished, because if it is used to guide tweaks it becomes part of the training feedback loop and can no longer provide an honest out-of-sample estimate.
“only when you're absolutely done, when you're confident that you have a model that works, then you use your true out of sample test data set”
- Hidden leakage, where a model accidentally caches an output that encodes future information, is a common source of false eureka results and can destroy the whole dataset's validity.
“the computer cached an output on one day that turned out to be important”
- Overfitting means the model learns something that looks correct on training data but is false in reality; a cautionary example is a model trained on tank images that actually learned whether a photo was taken at night.
“what they had really built is a model that knew if it was nighttime”
- Hyperparameter optimization changes the model's structure, such as number of factors in a beta model or neurons and layers in a network, rather than its fitted parameters, and doing it many times using a validation set contaminates that set.
“we're not optimizing the parameters of the model, ... we are now optimizing the model itself”
- Plain gradient descent can get stuck in local minima, a valley far above sea level; adding momentum, which carries parameters in their prior direction, helps the optimizer avoid such traps.
“we introduce a concept of momentum”
- Gradient descent trains models by measuring the error surface and repeatedly stepping parameters in the steepest downhill direction, like a marble rolling down a mountain until it reaches a low point.
“If you imagine a mountain, and you imagine standing on top of that mountain with a marble and dropping it”
- Unsupervised models lack true answers, but they can still be trained because the designer provides an error function that judges how well the model divides data, such as splitting it into two distinct groups, and the model updates itself based on that signal.
“I think you could bisect the universe two ways pretty cleanly as classifier or regression, unsupervised or supervised.”
- The test set must consist of examples the model has never seen during training, so the model cannot cache answers and the evaluation gives an honest estimate of performance.
“only when you're absolutely done, when you're confident that you have a model that works, then you use your true out of sample test data set”
ML should augment long-term investors' judgment rather than replace it or chase speed.
- The best entry point for ML in investing is a process you know well and that is relatively simple, so you can test whether a model's faster experience gathering improves answers, even marginally.
“Find something you know well, that works in a rather simple way or that you think has a relatively simple process”
- Lowin is most interested in machine systems that replicate the holding-period and judgment traits of great long-term investors, such as Buffett-style value investing, to enhance rather than replace investors and let them scale.
“I've become so interested in systems that now go the other way and look at long-term investing”
- The most successful ML deployments in investing use a model to detect one specific qualitative signal that a human already relies on, adding it as one more green or red flag in the decision process.
“you deploy a machine learning algorithm to just pick up that one thing and just become one more green or red flag that goes into a decision process”
- The most successful practitioners use their own behavior, trades, research, and errors as inputs to their models, creating a feedback loop that has proven effective in some cases.
“many of the most successful practitioners of this in the investment space are successful because they can use their own behavior, their own trades, their own research, their own errors”
- Optimizing how fast you descend a mountain matters less than being on the correct mountain, meaning a well-optimized model answering the wrong question is a serious problem.
“if you look around and realize you were on the wrong mountain in the first place, you've got a problem”
- Lowin is more interested in systems that emulate long-term value investing, such as holding periods and 'Warren Buffett in a box,' than in high-frequency trading, which he sees as a space where speed and algorithm sophistication dominate.
“I've become so interested in systems that now go the other way and look at long-term investing”
ML is narrow correlation-finding rather than thinking, posing a greater threat to experience-based expertise.
- Academic finance models were designed to explore parameters such as whether a beta is significant, while machine learning models are used purely for their output and pragmatic predictive value, so parameter-level interpretation is often sidelined.
“A machine learning model is used for its output. An academic model has become used for its output”
- AI systems are good at only one thing, discovering complex correlations in data, which humans call experience; they do not think, draw conclusions, or learn from the world, so they are better understood as narrow and dumb than as sophisticated.
“AI's are really good at only one thing, which is discovering complex correlations in data. In human terms, that's something we call experience”
- ML models are called black boxes, but we know with some certainty what happens inside them; the model takes in data, infers a correlation structure, and outputs a result.
“We tend to look at these things as if they're black boxes, and to some degree they are, but the truth is, we know what's happening in that black box with some certainty”
- The rush into high-frequency trading, where humans are not involved, creates a natural tendency to think whoever has the fastest and most sophisticated algorithms wins, which Lowin considers true but outside his own work.
“whoever can have the fastest and the most sophisticated and the most advanced algorithms, that person will win”
- The greater danger from machine learning is to people who believe their experience is why they are valuable, not only to people doing repetitive work, since ML is built to acquire and apply experience faster.
“the danger from machine learning is to people who believe that experience is why they're valuable”
ML fits experience-based tasks because AI is narrow rather than sophisticated.
- Training a neural network to predict future returns from historical returns alone is not viable in Lowin's view because it conflicts with his belief that markets are only occasionally predictable.
“if you're trying to tell me that you've trained a neural network to take historical returns and predict future returns, I don't even want to hear the rest of that story.”
- Because AI discovers correlations that humans call experience, the table stakes for an effective AI is that the task requires experience, so tasks lacking experience-based patterns are poor candidates.
“so, the table stakes for having an effective AI is doing something that requires experience”
- The popular conception of AI as sophisticated, exemplified by cars that drive themselves and Go champions, is at odds with the reality that these systems are narrow and controlled, closer to a puppy that performs a few tasks well enough to survive.
“AI's are very dumb, they're not thinking, they're not drawing conclusions, they're not learning from the world, they're just, they're like puppies.”
- In high-dimensional, long time series models, a parameter may matter only in rare conditions, such as when an email mentions Nigeria, so spending time deciding whether one parameter is influential may be wasted effort.
“That parameter may turn out to only even be useful when you see an email that mentions Nigeria.”
- If humans can detect a pattern in markets, then either machines will eventually detect the same pattern or humans are not actually detecting anything and are merely lucky, because Jeremiah rejects the view that human judgment is inherently beyond machines.
“my conclusion is that a machine will be able to detect the same patterns as a human”
Practitioners over-engineer ML and often fail to define the target question first.
- A frequent ML failure is not knowing what question the model was asked to answer, like the characters in Hitchhiker's Guide who got 42 without knowing the question, so teams deploy off-the-shelf deep nets without defining the target or what answers are possible.
“what did I ask this model to do? What answer was this model even capable of giving me?”
- The number one pitfall in applying ML to investing is over-engineering, driven by hype, external pressure, and the need to appear cutting-edge, when the insights in noisy financial data are often simple to find.
“that's, without a doubt, the number one thing is just over engineering, making things way more complex than they need to be”
- Building an AI model is easy because it is just layering regression models on top of each other with some finessing, but training it well is extremely hard and could take multiple careers of expertise.
“building an AI model is easy. It's just layering a bunch of regression models on top of each other with a little bit of finessing, it's training it that's really hard”
- It is very easy to end up with a chainsaw when all you needed was a butter knife, meaning practitioners adopt far more complex ML tools than the problem or their needs require, and often do not realize it.
“it's very, very easy to end up with a chainsaw when all you needed is a butter knife”
Defining a good trade outcome is a hard credit-assignment problem where humans still add value.
- In finance the output label is not obvious, so defining what counts as a good outcome is a massive cross-temporal credit assignment problem, and choosing a predictable proxy such as earnings or a cash flow forecast can be as important as feature engineering.
“It's this massive cross temporal credit assignment problem if you want a system to trade.”
- Humans are quite good at solving cross-temporal credit assignment in judging trades, though biases get in the way, so ML can enhance human pattern perception and decisions rather than replace them.
“humans, whether evolutionarily or not, are actually quite good at solving these sort of cross temporal credit assignment problems”
- Assessing trade quality explodes in complexity: holding a week longer, selling earlier, or choosing an alternative that lowers portfolio Sharpe ratio each produce different verdicts, so no simple profit-based label captures what a good trade is.
“So, the question of how to assess the quality of a trade explodes extremely quickly”
Models should admit uncertainty and trade only when markets are predictable.
- A model that can output an I-don't-know answer, or a probabilistic distribution rather than forced binary answers, is more trustworthy for investing than one forced to always produce a decision, such as a neural net that predicts future returns from past returns.
“it's the ability of a model to shrug.”
- Markets are not predictable on average but are occasionally quite predictable, so the winning ML approach must predict when the market can be predicted, and often the best move is to avoid trading in unpredictable regimes entirely.
“I believe that it is occasionally quite predictable. And, in some sense, if you want to win in an end to end machine learning approach to investing, you need to not only predict the market, you need to predict when the market can be predicted.”
- A parameter in an ML model may be useful only in rare contexts, such as emails mentioning Nigeria or princes, and sit dormant the rest of the time, which is hard to capture with standard average-based statistical confidence measures.
“That parameter may turn out to only even be useful when you see an email that mentions Nigeria.”
ML will become critical in investing within years as competitors adopt it first.
- Outside a few well-known firms, quantitative finance has no settled ML territory, and early excitement about end-to-end automatic trading has given way to skepticism in favor of decision-enhancement systems.
“I'm not sure I'd call anything settled in terms of quant finance.”
- A computer that could do what a fundamental stock picker does across 10,000 stocks instead of their 10 is extraordinarily disruptive, which is why many investors are resistant to and incentivized not to hear about ML.
“if your business is picking stocks through deep fundamental research and you own 10 at a time, and there's even a hint that a computer out there is going to do exactly what you do but can do it for 10,000 stocks”
- Within five to ten years, analytic ML will be critical in investing because competitors will adopt it first, and even a marginal gain such as seeing opportunities sooner or pricing assets faster is enough to matter since markets have disappeared for less.
“I think they're going to be critical. I think you won't be able to survive without some degree of analytic understanding, if only because the competition is going to get there first.”
- End-to-end automatic trading and models with no basis in reality, which Lowin once found exciting, are now seen as impossible and absurd.
“The things that were always most exciting to me, at first... are the things that I now believe are completely impossible and absurd. Which was end to end automatic trading”
- Outside a few well-known firms, quant finance has no settled applications of machine learning, so the frontier is still wide open.
“with the exception of a few firms that we all know, I'm not sure I'd call anything settled in terms of quant finance.”
Model failures are more traceable and interpretable than human judgment failures.
- Interpretability is overrated because individual parameters in a billion-parameter network cannot be meaningfully inspected, but models are interpretable in aggregate, for example by visualizing how each layer builds increasingly complex representations of the data.
“I think interpretability is overrated to some degree, I'm not sure that it is always useful.”
- Interactive browser visualizations built on TensorFlow show how a neural network builds increasingly complex representations at each layer, which undermines the claim that these models are total black boxes, though this qualitative insight is not statistical significance.
“it will actually show you interactively in your browser how a neural net is processing data at each layer of the net”
- The ultimate black box is a human whose judgment may be affected by something like last night's dinner, whereas when a model is wrong, the cause is traceable to specifiable errors in construction, learning, or data, making model failures more tractable than human ones.
“I have always found the ultimate black box to be a person who, if they had dinner last night that didn't agree with them, might make a bad call today.”
Models are classified by task type, learning mode, and hyperparameter choices.
- Decision trees work well when data can be cleanly split at each branch, as with categorical demographic variables for voting prediction, but they perform poorly on data like email or images where no clean bisection exists and where convolutional networks combine neighboring inputs instead.
“that demographic data should lend itself to being cleanly bisected at each branch of the decision tree”
- Models can be bisected into classification or regression tasks and into supervised models, which get labeled answers during training, or unsupervised models, which get no answers but a quality function to judge their output.
“I think you could bisect the universe two ways pretty cleanly as classifier or regression, unsupervised or supervised.”
- Tree models such as random forests are sometimes called set-it-and-forget-it because they have few hyperparameters and tend to perform well, making it hard to argue against using them by default.
“Sometimes they're called the set it and forget it model of machine learning, because they have very few hyper parameters”
- Beta, the relationship between a stock and the market or a risk factor, is a classic regression problem solved with linear regression, and CAPM is the basis for such models.
“in a finance context, beta is a great example of regression”
- Machine learning models can be organized in a two-by-two grid: classification versus regression on one axis and supervised versus unsupervised on the other, with linear and logistic regression, neural networks, clustering, and trees placed within it.
“I think you could bisect the universe two ways pretty cleanly as classifier or regression, unsupervised or supervised.”
- Hyperparameters, such as number of factors in a beta model or neurons in a network, are choices about the model itself, distinct from the parameters the model learns during training.
“we're not optimizing the parameters of the model... we are now optimizing the model itself”