Show HN: Optimized trading algorithms using IPython parallel and ec2 | Hacker News Reader

Show HN: Optimized trading algorithms using IPython parallel and ec2

twiecki.github.com

102 pointstwiecki28 comments

Cool technical presentation, but trying to optimize moving average trading strategies is a fool's game - you might find a set of parameters that work in all out-sample/cross-validation tests, but there's still a good chance it'll lose money in actual trading going forward - market paradigms can shift, unforeseen world events, t-cost/slippage higher than what the model accounted for (especially with more frequent rebalancing), etc.

If you actually want to trade moving average strategies, your best best is just to diversify and run several different strategies across a variety of parameter sets and across various sectors/asset classes, without trying to overly optimize a single strategy

Certainly that's always a risk. Zipline does however support simulating transaction-costs and slippage so those will be accounted for (as best as possible).

As for market-paradigm shifts, I totally agree. One way to deal with that is to constantly re-optimize the parameters based on recent data. This is also known as walk-forward optimization on which there is a slide at the very end (see http://blog.quantopian.com/parameter-optimization/ for a more information).

Finally, as to running multiple strategies with different parameter settings. The OLMAR paper that describes the algorithm (http://arxiv.org/abs/1206.4626) has a variation of this where they use a range of different-length moving averages, rather than just one.

Also, this is not your standard dual moving average cross-over strategy. Rather, it rebalances its portfolio (optimally) under the assumption that the stocks will revert to their moving average. The eps parameter, which I explored in this talk, influences how aggressively the algo rebalances its portfolio -- smaller values will result in a more stable, conservative strategy.

++ You're right on all counts. You've obviously done this before...

To anyone newly thinking about this topic, it is certainly the most challenging project you can take on. There are no laws of the market like there are laws of physics. You can get your code to work perfectly, but all that means is that your results will be the same as your model's going forward. As for a model, it's likely to need to change fairly frequently, even becoming the opposite of a previously great model for example, and you can never truly account for the effect of politics on markets. One tip is that a sudden catastrophic crash is far more likely than a gigantic surprise to the upside so if you're playing with real money, you need to use options to insure against such an event. Luckily (or not luckily in the case of trading itself) markets are highly correlated, so index options will probably fill this need.

I noticed you got close prices from Yahoo that you are using for your backtest. If you get the close prices in real life and optimize, when are you going to trade?

You have to deal with intraday prices now. Got those? Well they are not split or dividend adjusted. Good luck.

Don't forget mergers. I spoke about this problem recently at Matt Turck's Big Data Meetup (http://vimeo.com/60598560). Quantopian provides zipline powered backtesting over fully adjusted intra-day data (minute bars) for free.

Unfortunately I wasn't able to attend February's Big Data Meetup, so I missed your talk. EDIT: (I did just view it online, my comment takes what you said into account.)

My point about intraday data with corporate actions wasn't really about the difficulty with corporate actions, but rather the leaky abstraction of trading present in the slides. To give you credit, I don't know what issues Quantopian addresses. On the other hand, has any one tried running Zipline-backtested strategies in real life? Does any one know what issues aren't addressed by Quantopian? Corp actions was one part, T-cost model was hinted at in the slides, but was any though given to borrow costs and availability? There leaks everywhere. It is not that you don't seem like smart guys who made this cool thing freely generously available to everyone, but that you seem like you spent too little time downtown NYC.

PS. Fully adjusted bars are nice, but they have an epoch to be adjusted to. Unfortunately having an epoch that is not today() means you can't add today's data to it. If you can't add today's data to it, you can use this system to generate real trades to trade. Now you need two sets of data and two sets of code to work with. Good luck.

Thanks for taking a look, and for the feedback. Zipline does model transaction costs, both commissions and price impact of your own trading (slippage). The commissions and slippage models are pluggable, so you can use what is there or roll your own.

Quantopian does not have data for stock borrowing costs or availability, and Zipline's slippage/cost model does not account for them either. We'll find a way to get that data and plug the hole. The challenge has been finding a clean way to get it from the brokers, or finding an aggregator with a reasonable price (any advice?). In the meantime, we've been open about this limitation, and the zipline code is opensource, so I think/hope anyone who cares to know does probably know.

Quantopian is building our live trading environment now, so we don't yet have comparisons between the backtest results and real trading.

Regarding your point about the epoch, I'm not sure I entirely follow you. Part of the point of zipline's design is to allow easy swapping of datasources, mainly to allow the transition from backtesting to paper trading and then to real trading to be seamless. One algo code can run either historically or live. Adjustments from splits and mergers are back-projected, so that current day prices need no adjustment. Dividends are dealt with as announce, ex, and pay events, meaning we do not smooth out the over-night drops, instead we increment/decrement cash.

I'm in NYC regularly to host the NYC Algorithmic Trading meetup - it would be awesome to talk to you about these issues in person, please consider coming: http://www.meetup.com/NYC-Algorithmic-Trading/

I would think it's relatively easy to swap in refined data after the fact. The attraction here is the process to consume the data.

Yes, yahoo finance prices can be quite messy and sometimes even wrong for the reasons you mention. I wanted the code to be portable so that's why I used them.

We have the same strategy on Quantopian though (https://www.quantopian.com/posts/olmar-implementation-fixed-...) where we have 10 years of minutely data, adjusted for splits and dividends. It seems to do quite well.

presentation is unreadable on Linux Chrome btw

An interesting course on using software algorithms for stock analysis and trading can be found at Coursera: https://www.coursera.org/course/compinvesting1

That course also uses Python as programming language in the examples. Short description from that URL: "Find out how modern electronic markets work, why stock prices change in the ways they do, and how computation can help our understanding of them. Build algorithms and visualizations to inform investing practice."

Fascinating from a technical perspective, but how does it add value to the world beyond the search for increasingly complex ways to make money?

Investing is one way people plan for the future. Helping people plan for the future is a good thing. Algorithmic trading has mostly focused on (ultra) short timescales, without much regard for future planning. I think the world would benefit tremendously if more of that effort went toward investing on longer timescales. Investing is still almost fully manual today, and packed with inefficiencies and costs that could be automated away.

Very cool! My question is, how do you use something like Zipline to actually execute trades.

We are building just that at https://www.quantopian.com. It runs on zipline, and we're hooking it up to brokers.

That's currently not implemented, but definitely something we would like include. What I'd really like to see is a client for a bitcoin exchange. See our contribution request page for more details: https://github.com/quantopian/zipline/wiki/Contribution-Requ...

Finally, live trading is something which we are working on to enable on Quantopian (which uses zipline for backtesting your algorithms). The syntax is very similar so you could just convert your zipline strategy to Quantopian (which is free to use).

For those with an account, E*TRADE offers a REST based API @ https://developer.etrade.com . Onboarding is manual but once you get your key you can execute trades for your account via the API. Equities / ETFs only right now, they were working on support for other asset types but not sure when they are going to launch it.

Full disclosure - I used to run their developer platform.

How do commissions work? per trade? Does the api provide any data, or just execution?

Commissions are the same as your E*TRADE account; list price is $9.99 for equities although if you are doing higher volumes you can negotiate discounts ($7.99 if you do 150 trades / quarter, and call them up if you are doing a lot of trades for lower pricing).

Data: https://developer.etrade.com/ctnt/dev-portal/getDetail?conte...

There's also a call for data on option chains.

If you're looking to do API trading as a retail customer, Interactive Brokers is the best I've found. 1/2 cent per share per trade with a $1 minimum commission per trade.

A downside is that they are between you and the exchanges at least when you use their SMART routing, so I get the impression your fills will be optimized for their benefit (via their Timber Hill subsidiary), not yours. Years ago, you could get filled substantially better than your limit order especially at the open for example, but no more.

I can't scroll down the code on slide 8.

Yeah, the code is cut-off unfortunately. It's the new IPython nbconverter that still has some kinks. The idea of the algo is expressed on the next slide.

You can, however, view the full IPython NB here: http://nbviewer.ipython.org/urls/raw.github.com/twiecki/zipl...

yeah, for now you can not scroll down... I will add this request to the list of issues ;-) thanks for the feedback!

Beautiful presentation; really loving all the stuff built on top nbviewer.

As an aside, I think an enterprise-quality zipline solution could be a worth a pretty fat chunk of $$ especially considering that the budget for KX and its support is generally > 1M per year per firm. (Yes, there's complex logistics and legal issues)

I believe seeing on github some requests for h5 support -- this + support could potentially be a big inroad...

Finally got it to load. Don't know what the problem was. I kept getting a blank page (wouldn't even let me right-click). Cracked open the dev tools, and the DOM was there, and all of the resources were loading. No clue what the problem was.

[ Running: Chromium Version 26.0.1386.0 Ubuntu 12.10 (177362) ]

The third or forth slide causes my IPad's instance of Safari to crash. (IOS 6.1.3)

This was a great talk, find it on pyvideo.org in a few days

Reply on news.ycombinator.com