Most trading strategies are not tested rigorously enough | Hacker News Reader

Most trading strategies are not tested rigorously enough

economist.com

86 pointsilamont37 comments

"Most trading strategies are not tested rigorously enough"

After having spent many-many years in the financial sector, I don't even know whether I should laugh or cry. :) The industry is not based on science, well, 99% of it isn't. Traders can be considered being the master of the universe just because pure luck. Well-researched, tested strategies are thrown out because they're not profitable enough to the senior management. If a model seems to bring in high profits then even the very makers of the model do not want to let it into production, the management will just use it because nobody cares of tail risk. And so on.

Great industry. :)

The other thing that strikes me as bizarre is the way the everything seems so coupled. Its like the exact opposite of good software engineering - we build the systems to be so interlinked that everything affects everything else. Like when the oil price came down, it was threatening to put the UK (a net importer) into recession.

My experience across industries is that people don't know basic statistics or the value of statistics. I even came across a manager of a data science team at a major company who did not know anything about statistical testing.

... what

Data science is a buzzword substitution for statistics.

How you could possibly have someone in a "data science" role without a statistics education is baffling.

It's really not. Statisticians usually receive only minimal training in statistical programming, data cleaning, data gathering and warehousing and so on – yet those are all essential to data science. On top of that, when dealing with bigger data sets, many common statistical tests become irrelevant, as their main purpose is to make it possible to reason about small sample sizes.

Of course, I don't mean to imply that a data scientist shouldn't have at least a basic statistical grounding: they do. But there's roles in data science for people with varying skills in programming, ops, statistics, ML, visualization and so on.

A number of years ago a friend and myself worked out what could best be described as a medium frequency trading strategy based on historical trading data. This was when high frequency trading was still in its infancy. The algorithm would find certain stocks with a high frequency of highs and lows. In simulation, it worked perfectly. We were making loads of "virtual money." This got us excited enough to unleash the algorithm on the world. Guess what the problem was? All the testing in the world can't account for the market dynamics your algorithm itself creates. That was our biggest downfall. Time and time again it worked in simulation but in the real world, its trading actions based on historical data was in fact impacting the market in unexpected ways that we were ultimately unable to account for.

This is exactly why I am on my way out. I have worked as a quant for traders for the past 5 years, and I am running out of patience. It's tough because Chicago is such a heavy financial hub. I have decided to move to a new city and try to exit trading all together.

And I'd say that this situation is an advance. 20 years ago when I was a naive techie working in trading, most of the trading strategies I saw were tested by some young, high-testosterone guy wading into a pit, shouting a lot at people, and feeling vindicated if he made some money. (Or, in a pinch, feeling vindicated by whatever positive signal he received before getting creamed.)

For those interested in this, I strongly recommend Nassim Nicholas Taleb's "Fooled by Randomness", which gives the reader a feel for how much supposedly mathematical and rational finance runs on intuition, survivorship bias, and plain bullshit.

A lot of the comments here seem to be from people with institutional experience or wannabe retail traders. As a retail trader (who used to phone my broker to make trades) that has progressed to an algorithmic retail trader (I do all my own development with some mentoring help from professionals) I can say that it is possible to make a good return on risk.

This did not happen overnight. I've spent thousands on my education and by that I mean I've been scammed, gone to useless seminars, read nearly 100 books on the subject and made terrible trading errors and trading losses.

I only started getting serious traction after a confluence of events that led to being tutored by an ex-JP Morgan quant and a software developer friend who has been developing trading software for the big Bank trading desks in London.

Moral of the story? Persistence.

How does this relate to the article? Persistance eventually overcomes a lack of testing to eventually lead to a robust testing methodology.

So yeah I do agree that most (retail) trading strategies are not tested rigorously but that's due to the difficulties around acquisition of inter-disciplinary knowledge and the balls to get real experience my putting money on the line.

I guess you wouldn't be interested in going into more detail?

I don't know anybody that uses anything as weak as 2-sigma results in HFT, at least. Most of our valid signals have 20-50+ sigma.

20+ sigma is mostly kidding yourself about assumptions.

That said, the 2-sigma tests are for client-facing algos, where the goal isn't to build wealth for the client, it's to generate fees.

Agreed, errors are non-normal so 20+ sigma is practically false. But it's not too far off -- I've seen strategies that never have a down day and have daily sharpes of 2.5-3.0, and those setups will generally have incredibly high sigma values even after accounting for non-normality (and other assumptions).

Remember, strategies that 'never have a down day' are only one day away from having a down day...

...and there are plenty of strategies that had only a single down day, unfortunately that one down day wiping out the entire company :)

Sure, but most strategies have such small positions that even if the market went to 0 instantly at your maximum long position, you'd still lose less than a year's worth of PNL in that strategy. The more relevant risk of blowout is something like Knight happening, statistical black swan events are not really an issue (in HFT).

I used to be very big into algorithmic and day trading, and one trader I was introduced to had Martingale'd himself into a record of something like 2 years without suffering a loss. Then he suffered a 300k loss and wiped out all of his earning plus more and then he lost a lot of his followers, at least for a while.

It doesn't look like the article is not about HFT in particular or even trading professionals in general. It's about academic articles, where the incentive is to publish a paper, not to make money. I recall (but cannot find) a paper that looked at the sort of strategies published by academics. Younger, non-tenured professors sometimes published results that were actually interesting, because they needed to get a job and built a reputation. Older professors don't do that. If they find an effect that's real, they take it to a hedge fund. There isn't too much incentive to publish a paper about something that will really make you money.

Since you seem pretty well acquainted with HFT, I'm curious about how much of a constraint capacity is? From what I understand the amount earned per trade is very small (this paper[1] suggests $1.45 per $10,000 traded). And since HFT is already a large fraction of the daily volume it seems that the natural way to increase profit (i.e. just trade more) isn't an option in most cases.

[1]: http://faculty.haas.berkeley.edu/hender/hft-pd.pdf

$1.45 per 10k traded seems really high. A good S&P futures strategy (one of the biggest products in the world) is typically going to make 60-80 cents per contract traded. Since each contract is approximately 100k, thats closer to .06-.08 cents per $10k traded? Even for equities (I don't touch US equities, so I'm not entirely sure how good the best strategies perform), making a full price tick per contract is still < $1earned/$10k traded... and good HFT strategies are more on the order of 5-10% of a tick per contract traded.

In any case, capacity is probably the first constraint you hit. Most firms have reasonably accurate simulations, so most HFT strategies are scaled up to as large as they can possibly trade (ie. quote the largest amount passively or aggress with the largest amount you possibly can) within a few days of being released -- once you can confirm that your live trading is at least mostly matching simulation, you usually try to simulate the maximum possible size it can trade and just start live trading that. Since you're typically scalping a tick at a time, your maximum size is typically some fraction of the zero level bid/offer -- relatively small. Typically the way you scale up is either have better execution (know when to size up/size down appropriately) or better prediction quality -- since you're adversely selected, your bad trades get filled at a much higher percentage than your good trades, so as your have better prediction quality, a smaller percentage of your volume is bad and you can start to fire larger and larger.

Thanks, that's really interesting. I hadn't even thought of the adverse selection problem. And you're right the $1.45 was high. I looked at the paper again and that number was before adding in costs. With trading costs the number was much lower, but still profitable (can't remember the exact number off the top of my head), and that paper was just US equities.

One of the largest quant funds out there... http://www.twosigma.com/

From Wikipedia: "Some people claim that by recognizing chart patterns they are able to predict future stock prices and profit by this prediction; other people respond by quoting 'past performance is no guarantee of future results' and argue that chart patterns are merely illusions created by people's subconscious."

In my experience, the patterns are not illusions, they are very much real, but they don't have predicative power.

http://www.bloomberg.com/news/articles/2015-02-20/high-frequ...

Predictive enough that when combined with money management strategies you can make money every single day of the year.

What Virtu does to make money (market making and latency arbitrage on a milli- and microsecond timescale) is very, very far away from recognizing chart patterns and trading on them!

All types of trading, no matter what system, is theoretical until it hits the market. The market is the only test.

To what extent does the industry use Bayesian methods? It seems to me that, for a given trading strategy, trading companies are really interested in the mean profit impact of the strategy and its distribution (e.g. "there is a 10% change we will lose more than $20M"). Bayesian methods will naturally produce such an answer.

Bayesian methods won't magically solve all problems (e.g. fitting to historical data) but could make the assumptions more clear.

I can't speak for other teams, obviously. But almost everything we do would fall under the heading of "Bayesian methods". It's such a broad term that it would be hard to write a trading algorithm that couldn't be described as Bayesian.

Past Performance is Not Necessarily Indicative of Future Results; ‎

When trading real money in a real market, predictions based on historical data go out the window.

Historical data will never be able to truly simulate manipulation or sympathetic, symbiotic or parasitic relationships. Ever back-test a trading system that simulates a Market Maker letting low block go under the bid or dialing down the sensitivity of the bid vs. the ask? Speaking from experience.

That's why I'm developing an algorithmic trading system based on sympathetic, symbiotic and parasitic hidden connections.

Also Ref: Contagious Speculation and a Cure for Cancer: A Non-Event that Made Stock Prices Soar - http://www0.gsb.columbia.edu/whoswho/getpub.cfm?pub=1555

> That's why I'm developing an algorithmic trading system based on sympathetic, symbiotic and parasitic hidden connections.

That sounds fascinating. Do keep us informed!

Should be done by tomorrow!

http://www.cymetica.com/recommend/app/hidden_connections?que...

"When trading real money in a real market, predictions based on historical data go out the window." No they don't. Depends on the style of course but most arb or stat arb strategies are fully derived from historical data.

"Ever back-test a trading system that simulates a Market Maker letting low block go under the bid or dialing down the sensitivity of the bid vs. the ask" Could you express this more clearly? Your language is sloppy. Yes I've back tested lots of market making strategies - all far more complex in behavior than traditional market making.

Show me how you back test for manipulation I'd love to incorporate that into my system!

Don't forget to read that paper above by Gur Huberman. Good starting point.

More slopppy+---adsf language for you...:::

How Brokers Can Avoid A Market-Maker's Tricks http://www.investopedia.com/articles/financialcareers/06/mma... lets incorporate this too.

I also agree that TA and back testing applied to long positioning workw great in bull market!

If you think there is any insight into current market structure in that linked article, you're on the wrong track. It manages to be 15 years out of date and confuse open outcry (pit trading) with the specialist/broker system employed for equities.

And there's nothing magic about simulating "manipulation" - you're not one of those deranged paranoid zerohedge balloonheads are you?

And it's clear you have no experience in this area if you say something as trite as your last sentence.

Good luck with your "system".

Yawn.

Reply on news.ycombinator.com