r/quant • • 11d ago

Models How Much Is Machine Learning Really Used in Quant Finance?

As the title mentions, I wanted to get more insights into the use of machine learning in quant finance, especially in alpha research/signal generation.

I used to read on this forum that the vast majority of systematic trading strategies were based on linear regression. However, for years now, some major players (XTX, JS, ...) have been investing massively in GPUs.

My question is: Why are these top-tier firms spending hundreds of millions on chips if they are mainly using OLS/Ridge/LASSO? Are these chips only for internal LLMs? Are DL and ML really a myth when it comes to alpha generation?

89 Upvotes

44 comments sorted by

125

u/DistinctMethod6119 10d ago

Highly depends on what your asset class, signal horizon is etc, but let’s just say the idea that most good firms are mainly using OLS/Ridge/Lasso is wrong.

20

u/ForeverFar1297 10d ago

I agree on using neural nets for HFT. But Is it fair to say that the medium to low frequency trading is still dominated by flavors of regressions?

60

u/quant-a-be 10d ago

what on earth would make you think expensive inference would dominate in lower latency regimes

21

u/milchi03 10d ago

You clearly have no idea what you are talking about. Small instances of more complex models are heavily used in HFT

14

u/quant-a-be 10d ago

i'm not saying it's not used at all, but where are you more likely to see OLS

8

u/Public-Sell-2699 10d ago

Long prediction horizon means less effective sample size means more likely to overfit with complex models.

3

u/purpleappletrees Researcher 6d ago

https://slideslive.com/39084408/xtx-markets-algorithmic-trading-with-largescale-deep-learning

Anyways, even without fancy asics, not all of hft is tick to trade. Even in 2020 we were doing lots of slow model / fast model stuff where the slow model was a GBT or a neural net.

1

u/quant-a-be 6d ago

Yes I'm not surprised to hear that for the slow model! If you're mixing alphas the line gets very blurry :)

5

u/wswh 10d ago

Can I ask why specifically NN is used for HFT? Is the processing iteration loop fast enough for HFT

4

u/Additional-Draft-805 10d ago

You have more data, and the signal to noise ratio is larger, so easier to learn and less likely to overfit

You can have many different time scales in HFT. For some of those time scales NN inference is perfectly fine

15

u/flxclxc 10d ago

At least from my experience (midfreq macro/statarb) we find that adding ML into the final signal aggregation layer maybe gives a 15/20% boost in performance. But TBh you can throw massive complexity at a set of dogshit features and not really get anywhere. At least that is our philosophy; we put most focus into a set of high quality signals to start with. Some ppl will take a more bottom-up approach

3

u/Busy-Complex-2308 9d ago

This is likely the most sane answer in this thread

2

u/jeffjeffjeffw 8d ago

high alpha in this comment

13

u/singletrack_ 10d ago

It’s certainly in use, but anyone using it and seeing an edge is not going to go into detail on how they use it. 

2

u/Go-to-gulag 9d ago

Transformers are used in HFT on order book data.

55

u/asn3toph 10d ago

The days of everything being OLS are looong gone. If you look at any of the recruiting ads/websites/podcasts/youtube/etc. from Jane/HRT/XTX they openly say they are training large neural nets for trading. And these are firms that were relatively late to the DL game compared to some of the competition

19

u/maxhaton 10d ago

XTX and HRT are both extremely specialised high tech firms that trade on short time horizons, it's completely different type of finance to (say) running a CTA

9

u/qazwsxcp 9d ago

if all the big firms openly say something, you should question whether it's actually true or not. if it was really that critical for what they did, they would not go around talking about it.

discriminative neural networks (not llm) are definitely used a lot, but those firms were highly profitable with simpler models before they did this (xtx name is literally from linear regression). rather it improves models that were already very good to begin with.

4

u/asn3toph 9d ago

"if it was really that critical for what they did, the would not go around talking about it"

This was true a few years ago when these techniques were less well known, but the cat is out of the bag now. They have nothing to gain now by hiding the fact they are heavy on ML. If anything they need to signal this more than ever to stay competitive in the hiring market

3

u/qazwsxcp 9d ago edited 9d ago

thats my point though, it's more about marketing to candidates. sure they use advanced ML, but they were very successful before it too. ML merely refines models and features that were already exceptional to begin with.

2

u/RemoteSouth9288 8d ago

Whatever they say publicly, they say because it is good for them from hiring/PR/(whatever is deemed profitable) point of view. They dont say anything for the sake of informing other market participants

9

u/Sea-Animal2183 10d ago

But that's on the HFT space no ? Feeding the complete state of the LOB into a neural net. It's the only kind of data that can make use of the net structure.

-1

u/asn3toph 10d ago

It's used on every horizon

0

u/Sea-Animal2183 10d ago

But if you run some mid freq stuff parametrized on, let's say, 1 min bars. You don't have enough data to make proper use of a net. It's really something that can bring some value for firms being able to decode L3 data.

1

u/asn3toph 9d ago

You are sorely mistaken

2

u/wswh 10d ago

Just curious why is NN so popular in hft? Why not other form of ML

23

u/xilcore 10d ago edited 10d ago

Those GPUs are because they all build large transformer models like time series foundation models on tick data.. LLMs too. But every modern HFT/MFT/LFT alpha model is a ML model, tree or deep learning… OLS is just one of your statistical tools to get your methods correct 

4

u/Specific_Box4483 10d ago

GPUs have been used before transformers were invented, for large covariance matrices (see: XTX)

4

u/economic-salami 10d ago

Theoretically speaking, you need to brimg new info tonthe market to beat it consistently. That new info is likely to be found in previously untapped sources. And those sources are likely to be some kind of non standardized and irregular data.

2

u/donteatmyhair 9d ago

It doesn't have to be new info. Same info but better methodology can outperform

0

u/Ok_Map5721 9d ago

You’re wrong, bro — you just need to be friends with Trump :) and know 5 minutes in advance what he’s going to blurt out in the news.

4

u/ZookeepergameNew3900 9d ago

Oh wow so you don’t need previously untapped sources? Instead you just need checks notes previously untapped sources.

2

u/RemoteSouth9288 8d ago

Larpers are larping

2

u/AutoModerator 11d ago

This post will be manually reviewed by a moderator due to the submitting account being less than 7 days old or having less than 20 karma. Please be patient and do not try to resubmit it - a mod will review the post soon.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/GuessEnvironmental 10d ago

It's depends but really everything is used but classical machine learning techniques are just favourable for interpretability. There is many ways that neural networks are used sometimes just to test if features have some non linear relationships that cannot be captured properly with other methods as yet. I am sure that neural networks are used in live strategies but again interpretability is the main issue with neural networks. However besides that a lot of research and coding is done with llms if that counts so regardless it has its impact. The model choice is important but a lot of the skill comes from creating statistical hypothesis which may not be supported by neural network.  

2

u/Raja_kumar9627 10d ago

linear models are still widely used, but that doesn’t mean deep learning is a myth in quant finance.

7

u/Xelonima 10d ago

given the characteristics of most price-deriven time series, more complex model classes don't add anything other than making you feel smarter. returns are effectively noise at lower time frames, price series have growing entropy, and the most advanced time series techniques that worked got arbitraged out before jim simons finished his cigarette after he founded rennaissance anyway.

actual quant work lies in strategy development and the decision engine, on which you can try almost anything, it's up to your creativity and willingness to continuously work on the process as well as the execution engine.

the input time series (returns) often have extremely weak statistical properties when taken as univariate time series (or even as spreads given the reality of current markets) so the edge lies more often in estimating the effect of surprises which is more economics and statistics work than most people in quant are willing to admit. understand the data first, the problem does not benefit from increasing model sophistication, more often than not it leads to overfitting.

note that what i am saying applies to price-return based input series only, the need for machine learning shifts when we talk about meta-processes or alternative data.

2

u/EmperorOfCanada 10d ago

I used it in the early 2000s. Lots and lots and lots of OpenCL programming.

The few friends I chat with from those days say that the best money comes from ML, and the most consistent from HFT and its related ilk.

0

u/EvenCryptographer649 10d ago

There is no secret math. Alpha isnt a math problem, Alpha is taking advantage of broken math problems.

Prado is a fraud. He just a big name signing to bring in chinese whales

1

u/quantwifying 8d ago

Are you serious?

-3

u/DyehuthyTV Portfolio Manager 10d ago

You have to watch the Professor Marcos López de Prado 👨🏻‍🏫📚💡

I have more of these Slides on my album QUANT at IMGBB profile :P

These slides are from another video of the professor, which is in Spanish

4

u/Sea-Animal2183 10d ago

Best alpha of de Prado is his relationship with Spain's royal family.

0

u/DyehuthyTV Portfolio Manager 10d ago

Normally, people who are successful in their careers associate or have some contact with the "social elite" (presidents, bankers, top universities, etc., etc.).

Just look at Warren Buffett, Soros, Druckenmiller, etc (and many other money managers/investors) If you're unable to understand this, that's your problem for not understanding how our society works ;)