Can a Machine Learning Model Predict the SP500 by Looking at Candlesticks? | Hacker News Reader

Can a Machine Learning Model Predict the SP500 by Looking at Candlesticks?

mariofilho.com

28 pointsmariofilho33 comments

No, not at all. Move along. Everyone tries this initially, and (quickly) learns this lesson. I suppose it cost you a lot less to find out the answer via posting to HN though.

"The reason the stock market is hard to predict is because it is a prediction." --Andrew Critch

I wonder how well a literal candlestick model would work - as in literally acting like a diviner watching candles melt and trying to map something from it. Given how much is up to chance and comingled variables (how the market performs correlated to temperature for instance could make it technically correct if epistemologically utter stupid) it may wind up ironically better than random chance. Still not something to stake your life savings on.

12:40am : restate my assumptions

:) For anyone else who doesn't recognize this: https://www.imdb.com/title/tt0138704/

The book “A random walk down Wall Street” concluded “NO”

I mean, that book was looking at technical analysis and charting. Also, he was just looking at mutual funds (which can only go long and suffer from a host of other problems). Hedge funds weren't really invented/popular back then and quant investing was practically non-existent.

Economists love to talk about EMH, and there's a great joke that illustrates the difference between economists and traders: The economist is asked what he would do if he saw a $20 dollar on the street. He replies "well it wouldn't ever happen, because someone would have already picked it up!"

The trader is the one picking up the $20, and the economist is the one who never believes it can exist.

Back to the point, it seems clear to me that there are few quant hedge funds that can have consistently outperformed the market that disproves the null hypothesis (no out performance) with a P < 0.05. Names like RenTech, Bridgewater, AQR, 2 Sigma.

> I mean, that book was looking at technical analysis and charting

Trying to predict the future by looking at candlesticks is pretty much the definition of charting.

Yeah that’s a good point. But I think the author of a random walk on Wall St was trying to make a stronger point: that all out performance on Wall St is just a random walk.

>it seems clear to me that there are few quant hedge funds that can have consistently outperformed the market that disproves the null hypothesis (no out performance) with a P < 0.05

The meaning p < 0.05 is that by random chance you expect to find 5% of companies doing this well relative to the rest with no actual underlying cause. The existence of a few companies that manage this is proof of exactly nothing about those companies.

There's also a strong hint that the most prominent and successful Funds with long records of market over-performance were actually just insider trading all along and their continued success is an indictment of our ability to effectively police white collar criminals:

https://www.newyorker.com/magazine/2017/01/16/when-the-feds-...

Comparing SAC/Point 72 with RenTech or Bridgewater is a joke.

SAC is/was run by stupid goons (I know a couple of them, they’re dumb). Bridgewater and RenTech employ top down systematic quantitative strategies (I also know a couple of them, they’re quite smart) that consistently generate alpha.

It’s easy to discredit people/industries that you are not familiar with. But I work in it, and awhile a lot of hedge funds are full of shit, there are a few that are the real deal.

You can systematically outperform the market year over year, and just because you might be more familiar with SV than Wall St doesn’t mean we are a bunch of crooks.

I work day in, day out, trying to make money for our funds investors. And honestly, I find it extremely insulting that you are calling all of us a bunch of criminals.

Edit: I’m sorry I sounded s little hostile, but I take pride in my work and believe I am helping the world. I think a lot of HNers are unfamiliar with the tech people on Wall St. And that’s okay, but we are just like you: trying to make money for the firm using technology and quantitative reasoning. The same kind of work you might do at FAANG, we do on Wall St.

Notice how many stock-picking darlings suddenly started losing money once regulators started finally enforcing insider trading rules. The likelihood is that crazy stock picking performance pre-2008 was mostly on insider knowledge, not on charting prowess.

The thing is, there are large extinction events with hedge funds. The chance that RenTech (run by Jim Simons, former NSA cryptologist), beats all competitors, year over year, for over 35 years against a different set of hedge funds each year is much less than 5%. Also, to think that RenTech is insider trading is borderline ludicrous. They don’t even hire finance people. Most of their original employees come from the IBM linguistics lab.

I work in the industry, and to be honest, many/most hedge funds are full of shit. They have high fees and enrich themselves at the expense of their clients. But let me tell you, RenTech is the real fucking deal. No one can match their returns. In fact, they make so much money that their main money maker (the Medallion Fund), isn’t even open to outside investors (only employees get to contribute). Look at their historical return stream, it’s ridiculous. According to Wikipedia, from 1994 to 2014 it averaged an annual return of over 70%.

I don’t know what the P value is of that off the top of my head but it must be under .00005.

https://en.m.wikipedia.org/wiki/Renaissance_Technologies

"A random walk down Wall Street" is quite the contradiction of a book, though. It simultaneously claims that markets are so efficient that a lay-person cannot profit from them while acknowledging the existence of asset bubbles. The whole "efficient market hypothesis" to me seems much more like a religious belief among neoclassical finance types than anything actually scientific.

Furthermore, there's a fair amount of research that suggests that Brownian motion/random walk does not at all explain the movements of a stock's price [1][2][3]

[1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=346975

[2] http://assets.press.princeton.edu/chapters/s6558.pdf

[3] https://www.jstor.org/stable/4538722?seq=1#page_scan_tab_con...

> The whole "efficient market hypothesis" to me seems much more like a religious belief among neoclassical finance types than anything actually scientific.

Let's ask the creator of the efficient market hypothesis [1]:

What is the efficient-markets hypothesis and how good a working model is it?

Gene Fama: It’s a very simple statement: prices reflect all available information. Testing that turns out to be more difficult, but it’s a simple hypothesis.

Richard Thaler: I like to distinguish two aspects of it. One is whether you can beat the market. The other is whether prices are correct.

Gene Fama: It’s a model, so it’s not completely true. No models are completely true. They are approximations to the world. The question is: “For what purposes are they good approximations?” As far as I’m concerned, they’re good approximations for almost every purpose. I don’t know any investors who shouldn’t act as if markets are efficient. There are all kinds of tests, with respect to the response of prices to specific kinds of information, in which the hypothesis that prices adjust quickly to information looks very good. It’s a model—it’s not entirely always true, but it’s a good working model for most practical uses.

[blah blah blah]

The point is not that markets are efficient. They’re not. It’s just a model. The question is, “How inefficient are they?” I tend to give more weight to systematic things like failure to adjust completely to earnings announcements, or momentum, than to anecdotes, which are curiosity items rather than evidence.

------

This does not sound like "religious belief" to me.

[1] http://review.chicagobooth.edu/economics/2016/video/are-mark...

If a well trained ai at scale to read the sticks and only the sticks fails, why would it be considered possible for humans to do it?

They can't. Charting is garbage, everyone knows this.

https://www.investopedia.com/university/charts/

seems some people do. Charting is probably bunk and it might be impossible to get reliable trades. if that could be scientifically proven people wouldn’t lose their money tryng to use it.

>> seems some people do.

Yes, people do charting. No, they dont actually make money on it. There is a vast market to manage money very profitably for anyone who can demonstrate consistent performance (mutual funds, etfs, hedge funds, etc.) There are also many sites now that will audit your performance and prove you are performing well by tracing outcomes. If indeed charting was profitable, there would be proof of it and people trying to profit off it by managing money using it.

How to set yourself to failure by starting with "one of the most widely known techniques". You can't outperform the average by using the same tools as everybody else.

I was always curious if you trained a model on a literal visual representation (pixels/image) of the charts or candlesticks, would the model be able to “see” something that we can’t.

Of course not. It's the same data but presented in a harder to process format.

It may be the similar but as they say, a picture speaks a thousand words, the visual features that a CNN might pick up could be something completely different than the features someone could think of. It is all about data representation. Hypothetically, the data representation shouldn't matter, but I think it is like viewing the optimization surface from a different angle, it is possible to get something different out of it.

This is something I got curious about too. It's very likely the answer is no, but I would like to test it at some point.

Using RMSE doesn't make sense to me, your losses are linear to how wrong you are. Shouldn't the error just be how far off you were?

Isn't that what quantitative trading is?

Quantitative trading isn't trying to predict the S&P500. It's trying to predict the distribution of returns, which is a (slightly) easier problem. Quants are just trying to make money, not predict the future. And judging by the returns of quant funds like Bridgewater or Renaissance, the answer is that while they might not be able to predict the S&P, they can generate alpha year over year.

I think most quants trade on

1) volatility (ie mis-pricing of the expected distribution of returns) 2) front running large orders 3) rapid news analysis

I don't think there is a successful model for market directional prediction based on previous price action (ie charting).

No. Hedge funds make money on asset allocation and asymmetric information, i.e. exclusive market information (see Quandl). They don't make alpha with complex trading rules.

No. Saved you a click.

Yes, no, maybe, I don't know

Can you repeat the question?

Remind me what the ROC curve looks like for `can` again.

Reply on news.ycombinator.com