Statistical Arbitrage – An Easy Walkthrough | Hacker News Reader

Statistical Arbitrage – An Easy Walkthrough

dm13450.github.io

150 pointsxgdgsc48 comments

A few notes

We normally use ordinary least squares when we do the linear modelling. Maybe they are and I can’t read the Julia code.

Also they aren’t adjusting their closing prices do dividends as far as I can tell.

Nor are they taking dividends payments into account at all as far as I can tell.

They did get the shift from daily to hourly right when they went to their crypto example so good for them. They are ahead of most day traders there:)

Also no work in the hurst exponent to figure out the reversion time frame

Also no modelling on how each pair will fit into your portfolio composition and if you can get borrow, which you certainly can in this instance, and trading fees.

Also they skip regime detection, which is determining what regime the market in general is.

Lots of pairs only work in up or down or sideways markets and you’d want a different ratio for each. And you’d want Tom look at how your pairs works in each market regime. I find this to be the most important piece and. You never seen pairs tutorials take this into account.

Probably because you his is the hard piece that makes you money.

This is why pairs are thought to trade. This link is the star arg 101 tutorial that is in 50 different places in the internet but it doesn’t show any of the actual meaningful work you need to do to trade pairs.

Other missing pieces are leg sizing, how long you let a leg hang, how many legs do you out out, portfolio composition, dynamic hedge ratio calculation,market regime detection.

It shows the easy piece that you can code up in an afternoon.

And now I feel like I’ve shit all over this poor kids project. It’s well done and covers the 101 issues with a few pretty large missing pieces that I’ve pointed out

curious what exactly you're referring to by market regime detection / classification, anything I've seen on this (several models from bulge sellside) has been backward looking and fairly useless.

We’ll refine detection is detecting when the market refine changes. But I assume you know that.

Typically you’ll run one set of strategies in a bull market for a given Security and another in a sideways market and others in a bear market.

Keep in mind different securities can be in different regimes at the same time.

And the market in general can be in a different return from individual securities.

To detect regime change you typically model it as a hidden Markov process.

For back testing we typical pre-label the data by the regime that we know the market was in because we have the benefit of hindsight:

My understanding of this is that you want to classify what all of the other traders are doing, basically. That is, intro investment discussions build on intrinsics of what you are trading. As you trade more, you also want to trade on the behavior of everyone else that is trading.

Sadly, all market discussions I've seen are always "backward looking" and fairly useless for most folks. High frequency trading is basically cheating by do much smaller forecasts that can be acted on quickly and profitably. But if you can't react fast enough, the information is effectively useless.

Is like knowing tomorrow's rain forecast when you are trying to plant for the season. It is of little help. Even if it is far more accurate than the year's forecast.

Nope - you didn’t shit all over it. This is a fair disclaimer for anyone venturing on this journey.

There’s a steep learning curve - it’s many times harder than writing full stack code and solving business problems.

To be moderately successful, you’ll need to acquire practical knowledge on many different topics (economics, sociology, statistics, technical analysis, stock exchanges, brokers, margin, local trading/day trading laws, etc). And this is just for stocks/ETFs, leaving out Futures and Options (skip those in your first few years).

It’s a very raw, dog eats dog kind of world. Each person is out for themselves. You’ll experience a whirlwind of emotions. I find it best to schedule positions ahead of time, and avoid trades in real-time. It’s so important to manage your emotions and protect your capital.

This

The unintuitive thing about the stock market is that you are not trying to predict a system unrelated to yourself. But you are trying to predict other people's behavior. People who are just like you.

So when you notice a pattern - say "Stock S trades lower on Tuesdays and higher on Fridays", what do you do?

Do you buy on Tuesday to exploit the pattern?

Or do you buy on Friday because you are not a unique snowflake and others will notice the same historical pattern and therefore reverse it in the future?

There is no way to decide. Because even if you see long-running patterns in the history of the stock market, those patterns might have reverted just at that moment when people like you noticed them.

This holds for all patterns. Even if they are as complex as the ones described in the article.

ya so notice the pattern before anyone else and extract money by filling the inefficiency. this isn't a profound insight, the nature of alpha is that it's temporary and limited. doesn't mean it doesn't exist and you can't take advantage of it - as long as you're better than everyone else (faster, smarter, cheat-ier). tbh it's just like entrepreneurship - starting a business will attract competitors. but you can just outcompete them.

You don't know if "anyone else" already noticed the pattern. That does not reflect in the historical prices you can see. It will reflect in future prices which you can't see. And waiting for those does not help either. You are always at square one. Because others are sitting somewhere, watching the data and thinking about it just like you.

They don’t all notice a long time pattern at exactly the same time. Alpha decays over time as more people notice and exploit it, or the market slowly changes.

When alpha disappears overnight, it’s almost surely for reasons unrelated to other statistical event players - e.g. a tectonic shift caused by some bankruptcy, interest change, political decision, etc.

However, many traders do not know how to properly model and backtest. It works on backtest, fails in the real world, and they explain that “alpha is gone” when in reality it wasn’t there to begin with - it’s just that their backtest was bad - usually overfit or unrealistic assumptions.

You describe a world where you can watch the pattern slowly fade out and stop when it is gone.

But in reality, there is noise.

Say you start exploiting the pattern. You buy on Tuesday and sell on Friday. After 3 weeks of doing so, you lost money every time. Is the pattern gone? Or is this just statistical noise? Should you stop or plow through? You don't know.

Another way to look at it: We would have the exact same discussions if stocks prices were just random walk series.

To make a point in favor of pattern arbitrage, one would have to show that stocks differ from random walk series. Enough to be worth trading against this difference. As far as I know, nobody ever came up with a good argument in favor of this assumption.

The way to tell signal from noise is with enough statistics. That’s one benefit HFT has over “traditional” trading - it gives you thousands of data points per day. The other way to quickly get thousands of data points per day is to trade slowly but across many assets (which is a much harder game, granted).

Either way, if you know what you are doing, your backtest should also give you a good idea of the variance, and that should tell you if a 3 week loss is statistically probable or not. Personally, I guess I’d stay away from such strategies - I’d prefer a much lower alpha with much lower variance - so that I have effective feedback from the market.

Some firms, e.g. RenTech and Virtu , manage to have very consistent alpha. You haven’t seen a good argument because people who make money don’t care to convince you.

> You describe a world where you can watch the pattern slowly fade out and stop when it is gone.

Yes, this is a phenomenon that happens very often during alpha research. You discover something that is decaying already, and you join the wagon until there is no anomaly to correct anymore, at which point you should have found other wagons to join. It's an eternal race of finding new alphas while your previous ones decay.

The rest of your argument doesn't really make sense. You seem to be just against any form of statistical inference.

"Alpha" is called like that on purpose because it is _not_ noise anymore. If you regress it against your benchmark, you should definitely see a difference between alpha and epsilon, given you have enough points to reach statistical significance.

> one would have to show that stocks differ from random walk series

There is easily 40 years of litterature on the subject. You can convince yourself in 5m by running a PCA of stocks returns against beta, sector and country. Then you can run a second round of PCA of these residualized returns against momentum, size, value and quality. Quants funds find alpha against these latter residualized returns.

My favourite explanation is a contest to guess how many jelly beans are in the jar.

How can you win?

- Get lucky

- Have a data advantage (maybe you can measure the dimensions of the jar accurately from afar

- Have a modelling advantage (maybe you have a 3D model of a jar in a computer simulation)

> There is no way to decide.

Yep. Kind of like paper-rock-scissors, there is no dominant strategy.

Welcome to Iocane Powder, a dominant RPS strategy:

https://web.archive.org/web/20160819141717id_/http://www.ofb...

I'm familiar with Iocaine Powder. The fact that it won an internet Roshambo competition doesn't mean that it's the dominant strategy. If the opponent knows what move Iocaine Powder is going to play, the structure of Roshambo tells us that there's always a move that beats it. If it is not performing well, then it falls back to completely random. Completely random is a perfectly reasonable thing if you're in a Roshambo contest, not so much if you're trading money because transaction fees will eat your profits.

> Or do you buy on Friday because you are not a unique snowflake and others will notice the same historical pattern and therefore reverse it in the future?

Well, as you said, you're not a unique snowflake. Many people will make that extra thinking step too.

You can try and make another step, try to account for people aiming for the trend reversal. But you're not a unique snowflake - some people will think about it too, and account for those people the same way.

If this starts to look like infinite series, it is, and it converges to something, and I suppose there are some mathematical truths to be said about it. But the practical truth is, any pattern that becomes noticed will not end up reversed. It will be driven back down to random noise. This is the anti-inductive nature of phenomena like stock markets: once a pattern becomes known, it disappears.

What you describe is similar to what beagle3 described. A world where patterns exist and slowly fade out.

But that is not the reality of the stock market. See my reply to beagle3 for two ways to counter this view.

Specifically to your point that patterns exist but do not reverse: Then one could make money by just trading against the pattern. If no "punishment" comes down the line in form of a reversed pattern, there would be no risk.

The simplest example would be a hypothetical irrational bull market euphoria. More and more new investors come in under the assumption "OMG! Every day the prices are higher than yesterday!". If that pattern would not reverse, there could not be an "irrational euphoria" because prices would either rise forever or stay on the highest level. So it would be only rational to join the choir. Much to gain, nothing to lose.

In an arbitrage trade you might even want to put more money in if it goes against you - spread widens, so you can make more money as long as you are right.

Also you can decide by looking at what the market is pricing in relative to what the probabilities are (in your mind) - one might be a more scalable trade whereas it could be that the latter one was super cheap so you take a punt on it (e.g. let's say the implied volatility is far too low)

According to your theory, would you also explain Bitcoin's future as the halving event is approaching. Everybody knows the halving will happen in around april next year. Since everybody knows (everybody will notice the same historical pattern), do you think it will be reversed ?

In any liquid enough market one should assume significant known future events are priced in and the current price reflects them already. Are bitcoin markets 'liquid enough'? I don't know. And even when something is priced in, you can still expect prices to change a bit after events, because going from X% likelihood of happening to has happened is a difference.

Generally speaking though, the price of bitcoin will probably change much less because of the halving than someone might expect.

People have been talking about Bitcoin as an inflation hedge for a long time, despite the fact that Bitcoin has been printing money at a significant rate. Bitcoin would have been a lot lower valued if it didn't already have the scheduled halvings built into its price.

> So when you notice a pattern - say "Stock S trades lower on Tuesdays and higher on Fridays", what do you do? Do you buy on Tuesday to exploit the pattern? Or do you buy on Friday because you are not a unique snowflake and others will notice the same historical pattern and therefore reverse it in the future?

Why would you expect it to reverse?

If anything I’d expect the pattern to disappear once detects and arbed away.

I can’t see how you’d think the pattern would reverse itself.

That is the reason why most big quant hedge funds look for ghost patterns. Patterns that are not obviously at the first sight and which can be (hopefully) solely mined until the pattern disappears.

To catch onto obvious patterns you have to be not really fast but nano-microseconds fast, otherwise in a blink of an eye the price is already too high to jump in.

I imagine that if one has enough capital to fully arbitrage an opportunity, it is possible to

A) capture the entire available upside

B) estimate how large that opportunity might be, and not attempt to over-stoke it.

C) by capturing the entire opportunity, prevent the signal from being visible to the broader market. The only sign that a need is being met could be trade-volume rather than price movement.

That's why most quant hedge funds only have limited amount of aum. Most their strategies consist of a large amount of low volume trades. Low volume to prevent the ghost pattern being visible to others. This is especially true for non hft firms that hold positions in a much longer interval (minutes - hours - days). But this is untrue of course for lets say a big macro hedge fund that throws a hundred million dollars into a currency trade.

I totally agree on all three points and I even think that a certain amount of capital, opportunity related (not fees etc.), is not even needed to mine a pattern. Because according to the ones statistics and research the actually existing pattern should appear anyway.

> Do you buy on Tuesday to exploit the pattern? > Or do you buy on Friday because you are not a unique snowflake and others will notice the same historical pattern and therefore reverse it in the future?

Well you would typically do both, as 2 separate signals, or even more as time horizons widen.

This modelling is part of the "physics" side of quantitative finance, and often revolves around the use of oscillators and other dynamic systems.

Try to picture the pattern of voltage oscillations when you suddenly release a switch. Price often converge in exactly the same fashion. You can create a whole convincing analogy between voltage oscillations, capacitors, resistors, and actors of the stock market.

http://d2vlcm61l7u1fs.cloudfront.net/media/09d/09d606c6-6357...

The way to see it is that the price will most likely oscillate around the consensus price, until it more or less settles. The first part of the oscillation, right at the point of publication, will create a momentum from the current price, which will then overshoot to create a reversal, which will then overshoot to create an other momentum, etc. Each time a cycle of momentum / reversal switches, the amount of overshoot diminishes and the time horizon of correction increases.

To illustrate, let's imagine a news on ACME Corp, currently trading at $10, just got released saying that their earnings will be above expectations.

- In the first 30 seconds a strong momentum will drive the stock price from $10 to say $15.

- After the initial 30 seconds of momentum, the price will have most likely overshoot. Many factors will lead to this overshoot: abuse of market orders on news to get executed faster, imperfect execution of limit orders, genuine over-excitement from participants, delayed basket comparison computations, etc.

- From the 30th second to say the 15th minute, the overshoot will revert from $15 to say $13. People will compare ACME Corp to its sector, country and beta, they will realize it's overvalued and short it.

- After 15m the reversal will have overshoot to $13 and a new momentum will arrive until the end of the day, settling in on close at $14.

- After a week the original daily momentum will revert to $13.5 as slower mid-freqs catchup and agree on a more precise consensus price.

Etc

Overall the cycle of momentum / reversal did:

- 0m -> 30s : + $5

- 30s -> 15m : - $2

- 15m -> 1d : +$1

- 1d -> 1w : - $0.5

As someone who does kind of thing, I think while this is a good explainer, it doesn't explain the thing that actually makes it work.

It can be fascinating to read about and obviously it's an immensely deep subject.

But there's a tendency to focus on the pure intellectual "how do the prices move" models that doesn't take into account the business around how you actually get the model operating, both from an engineering point of view of "how do I code up the execution and housekeeping" and in the larger business sense of "how do I arrange the shop so that I am able to trade this model".

This exactly.

Working in the field too - I tend to think now that for quite a lot of this strategies, it's not really the genius idea behind that is worth money, it's all the accumulated knowledge about what could kill you that's getting incorporated into the system. Usually boring, tedious stuff create the best moat for the business. How do you make sure your ticker database is up to date, how do you make sure the dividends calendars are up to date, that you are trading the ticker you want to trade, and not the same name but on a different exchange and with a different fiscality, etc.

It's a bit like the saying about startups - the ideas are worthless about execution.

> how do I arrange the shop so that I am able to trade this model

Any chance you could explain this a bit more? Intuitively it makes sense that this would be as important if not more so than the modeling itself, but I have no context for the difficulties encountered when implementing it.

So for instance your model might need access to borrowing. In fact I know people whose entire trading strategy depends on being able to access shares to borrow in large size so that they can squeeze out a few bps of yield on certain situations. The guys who control the borrowing are picky about who they want to lend to.

Or your model might need to be really, really fast. Not everyone can just have access to a microwave link. The guys with the microwave are also picky.

Or your model needs certain kinds of data, which you will need to get certain vendors to sell you.

You might have a need to trade multiple models at once, which are not profitable on their own (uncorrelated, share costs). This is a juggling act done often at the business level.

For almost all trading you need some staff who know how to do things. It's not always the case that you just pay them a salary to do some job. Business decisions there as well.

A small disagreement with the article. Statistical arbitrage is a more general principle than "reliance on mean reversion" (though that is certainly a common example of statarb). It really encompasses any useful statistical prediction that can be used to trade with.

While I agree that the article is somewhat vague and imprecise, it is not accurate to encompass all statistical predictions as statarb either. That would more be what "quantitative" is.

For a strategy to be considered statistical arbitrage, it needs to be something _against_ something else. You can do purely directional statistical predictions, and that would not be statarb.

Very true! Though the dividing line between purely directional, and something against something else is unclear. Every price is against something else, in some sense.

After reading the whole thing, the main idea feels more like "statistical arbitrage, debunked!". He does the womp-womp when he doesn't find any alpha. He even mentions that the paper's author is often cited as an expert.

What is showcased here is a dummy pair trading example on single stock vs basket. Of course it does not work anymore.

There are a lot more evolved kinds of statarb though, that still work nowadays, and most likely new ones that we don't know just yet.

You can change the distance function, do it fully cross sectionally, etc.

Can you share some links or books that you would recommend for those that want to read further ?

Reading articles and publications is part of the job of quantitative research, and being a judge of their quality and reproducability even more so.

So, in all fair honesty with you, no I will not give you recommendations.

I have an analysis of cointegration between Bitcoin/Ethereum (in python) if folks are interested, https://andrewpwheeler.com/2022/02/09/cointegration-analysis...

This was over a year ago, so dynamics could of course have changed. But based on that analysis, BTC tended to lead ETH (Bitcoin goes up, ETH goes up). ETH up lead to BTC down though.

Not a trader, so don't take it as advice. Impressed by folks who beat the market, it always appears too volatile to me in my amateur stat modelling attempts to even kid myself.

"The good thing about it being a negative result means that we don’t have to start considering transaction costs or other annoying things like that."

This cracked me up. Haha. That is a good way to look as a "loosing strategy".

> We this that if JPM rises higher than XLF then it will soon revert and trade lower shortly. Likewise, if JPM falls lower than XLF then we think it will soon trade higher.

Why?

XLF is a financial sector ETF. JPM is a huge bank. If JPM does badly, XLF should probably do badly, and vice versa. Then it stands to reason that if JPM falls below XLF, it will soon go back up to where XLF is. This is relative, so the same situation with XLF rising above JPM, and JPM catching up/XLF falling back down, and similar situations.

Thanks! I know it may seem obvious to people clued up on this stuff but it's basically a dark art to me. I appreciate the comment

Imagine getting to know about this 20 years ago.

Statarb, the pair trading form, has been known since more than 20 years

Here's a link[0] to my (non-monetized, WIP) blog about Renaissance Technologies[1] who were one of the first ones to use statistical arbitrage. Others were D.E. Shaw[2] and Pete Mueller at Morgan Stanley (later PDT Partners[3].)

[0] https://sileret.com/blog/renaissance-technologies/ [1] https://en.wikipedia.org/wiki/Renaissance_Technologies [2] https://en.wikipedia.org/wiki/Pete_Muller_(businessman_and_s... [3] https://en.wikipedia.org/wiki/PDT_Partners

Thorpe has some interesting (unfortunately online the wayback machine has them) articles on stat-arb and his (modest) discovery of black-scholes from the 90s

Reply on news.ycombinator.com