Python For Finance: Algorithmic Trading
medium.com
medium.com
So despite the fact that the subject is interesting, I'd consider it a waste of time to try and gain anything but a basic understanding of the industry and how algo trading works.
unless its one of the many things in your life that don't
That does not necessarily follow; markets are not static and players are free to enter and leave.
Its fair to say that participants can leave the market, but in practice that isn't what we see. In practice there are firms doing this trading, and they are staying in business, and obviously making money.
Are they fleecing the little guy then? This explanation falls flat for me, i.e. for the amount of money they seem to be making, it would take a lot of small time participants losing everything every day. Most people I know aren't even active traders.
So why is it an accepted truism that the market is zero sum?
Yes, more or less (although that's just one way they make money, and probably not the most lucrative) And fortunately for them there are plenty of "little guys" ready to enter the market on a regular basis.
There is a lot of information asymmetry, and an individual is capable of capturing that without requiring a PhD or the resources of a large firm. That's not to say it's easy per se, but it's not hopeless. It requires special expertise or an unconventional approach.
Otherwise I agree that most people probably shouldn't attempt it (for risk tolerance reasons).
Why?
Yes, but you aren't going to have any such data. If the information is available to you, it's available to other participants of the market as well and you're not the only one trading on it.
Yes, agreed: the longer the data is available, the more likely it is that others catch and start using it. That's a constant battle of course - continually coming up with new sources of data and new ways of getting it before most of the market.
> what makes you think only you have thought about trading on that data which must necessarily be public for you to have it legally?
I don't think I'm the only one who has thought about trading on it; on the contrary, I know there are many parties interested in trading on material information that is only technically public, but very obscure. However, after consistently profiting on this data, I can be fairly well assured that there are relatively few parties with the same informational edge, because the market consensus clearly doesn't price it in until it becomes explicitly widely known.
As I said, I don't need to have certainty that I am literally the only party with the information, so long as it is known by so few parties that the market consensus doesn't price in the sentiment. All material data has a half-life, which decays with time and market awareness. In some cases, the nature of the data or the methodology for sourcing it is so novel that it is conceivable to be the sole party with the information for a short and meaningful time period (where information means the exact same dataset acquired in a categorically similar way, not the downstream sentiment derived from the dataset). But that is a much higher bar, and it's completely unnecessary for profit. It's nice for pride though.
You'll have to forgive me for not going into any greater detail than that, both because any more than this is strictly a trade secret, and because I'd rather not have a competitor for my personal trading operations (or have someone turn around and sell this data, ruining my alpha in the process). The best I can tell you is to look into the analysis of satellite imagery, but with greater human ingenuity and a more specialized approach. That's not going to get you all the way there, but at least that much of the methodology is publicly known thanks to the old article about Walmart parking lots. You can't collect actual information about a company, but there is a lot of room for human creativity in finding strong proxies for that data that the rest of the market is mostly ignoring.
Sure you may not be as profitable as top quant companies, but do you really mind that much?
That sounds too much like wishful thinking to me. Furthermore, if someone is really smart enough to beat the market consistently why on earth would they trade just on their own personal account? Work for a hedge fund and use other people's money to leverage your bets.
Alternatively, if you have a strategy that empirically works, and a strategy for identifying such strategies, why take on investors to share the risk? You can scale up your own investment and leverage yourself with less regulatory oversight.
Why on earth do you think the top quant companies will leave alpha on the table for you to snatch up?
It's not so much that they leave alpha on the table, rather that there is so much alpha available, and the capacity constraints and reward profiles are so different for many of them, that individuals can prosper outside of a firm. They just normally don't, because they lack the same training (and because it's very competitive).
Big firms leave tons of (for them) small opportunities on the table, because it's not worth their time.
Similarly, there are people who can regularly beat the market that don't have much to offer large funds because their methods don't scale.
As a simple example, a farmer hedges his wheat crop selling wheat futures. That allows him to reduce some of his risk, thereby allowing him to plant more (i.e. growing the economy). The market marker who bought the wheat futures may be trading multiple commodities. This trade diversifies his risk, allowing him to trade more of other commodities, allowing other farmers to offset more of their risks (i.e. growing the economy). Investment companies or hedge funds enter the market purchasing futures, to diversify their risks, allowing them to invest more in equity markets, which fund companies (i.e. growing the economy).
If my point isn't already clear, markets and trading facilitate the diversification and allocation of risk capital to market participants thereby growing the economy. It's the conduit for capital which does make something.
The same thing applies to getting a job for instance. Take job X. If job X was available and worth doing at wage $, someone would be doing it. So why bother applying ?
So what it really means is that the semi-strong and strong form of the efficient market hypothesis is bullshit: there are plenty of opportunities in the market, you're just not seeing most of them.
I would argue that nearly everyone doesn't even try to see opportunities.
My point is that models will never accurately predict the future.
"OK, how many times have you seen a $20 bill on the sidewalk???"
(I actually heard this response with a $100 bill as the example, which would make it even less common).
MIT does not teach "HFT", those "MIT PHD Quants" are just as untrained in finance as you are. The rest is math, something that requires time to study but not necessarily a certificate. And creativity. I've only seen a single HFT-algo that made me say "wow, that's creative"; all others were just straight implementations of finance "wisdom".
2.) The less you know the better
You've forgotten you can introduce new units of whatever is being traded and quite often is & there are additional complications such as dividends, stock options, etc which makes your oversimplification lacking substance.
Adjusted to average growth of stock market (and inflation), it is a zero sum game. You aren't creating value by predicting the future. Just like you don't create value when you predict which lottery ticket will win.
1) Reduce spreads.
2) Stabilise prices.
Now, whether it makes any sense to put to work all those bright minds for these purposes, is a question the society should have asked some time ago...
On the other hand, this zero-sum game helps companies raise capital to make real investments with real returns, and that part isn't zero-sum.
Where I think HFT is zero sum is your competing against other HFT people to more accurately and quickly predict the future in the very near term, then buy & sell in a very short period. And since the market changes all the time, these algos must change to stay profitable over time as the competition improves their algos to beat you. Algo trading might work in markets where HFT isn't big yet, but most people won't know which markets that is or how to actually trade there, and if you get good enough to make successful models, you're basically in the industry, it's probably way more than a hobby at that point.
An ultra-HFT liquidity provider being profitable has little bearing on the potential profitability of quantitative trading on significantly larger timescales. There isn't sufficient volatility in most products for these actors to eat the lunch of actors who have alpha on trades that can hold for significant periods.
I think the other goal being that if you create a profitable strategy, you can enter it into competitions, trade with other people's money, and make a profit. I think the platform is a value add, I have an account and use the research notebooks. Some day when I more spare time I'll make some trading algorithms.
I don't know what you verified, but it directly contradicts my own experience and the FAQ: https://www.quantopian.com/faq#data-sources
To your second point, as another commenter said Quantopian doesn't see your algorithm unless you give them permission, as outlined in the terms of service (which are very readable, by the way).
NB: I am not unaffiliated with Quantopian and don't use the platform for my own trading, but I have tried it out before.
Oh, they can? Well that's funny, because I just went onto Quantopian and checked for myself.
Gee, would you look at that...they are there.
This took me all of five minutes. Do you have anything else I can easily disprove while I'm at it?
I'm not searching for a needle in a haystack for you. Make a falsifiable claim, and make the entire claim, with the specific dates you're talking about, or stop this astroturfing against Quantopian and QuantConnect that you're doing.
It's like a slot machine tournament. No one has an edge but some percentage of people will do well enough over a finite period to convince them to put real money behind their lucky algorithm.
You should understand the following concepts at a minimum:
- Markowitz portfolio optimization (mean-variance analysis)
- Beta-neutral portfolios (i.e. using MSCI BARRA, sector ETFs or PCA factors, etc)
- Alpha decay
- Time series analysis (autocorrelation, GARCH, ARMA processes)
- Basic price-based signals (momentum, volatility, value, etc) - etc, etc.
My takeaway is: it's not about implementing a couple of trading strategies. It's about implementing a pipeline that rapidly allows you to test what-if scenarios.
I might have like 10 ideas a day for strategies. How many of those can I rigorously validate per week? What about variations? I.e. tuning various hyper-parameters? Combinations?
How quickly can I recombine data from various sources into exactly the layout I need to train and test this or that strat?
Me:
1) Build some simple forecasting NNs.
2) Realize I need to be able to generate and test ideas waaay faster. Start working on infrastructure.
3) Get impatient -- get some "bright" ideas, do some manual trading.
4) Make some money on the first day, lose double that on the next two days.
5) Decide to ban myself from manual trading as obviously I'm an addict. Resolve to only do algorithmic trading.
6) Back to coding infrastructure. Get bored ... do some manual trades.
7) After losing around $20k, stop the trading madness, and go back to buying and holding great tech companies.
8) Enjoy my 25% returns.
> 7) After losing around $20k
QuantConnect & LEAN gives you ability to do tick->daily resolutions; for equity, morning-star, future, option, forex and cfd trading - all with a fully open source project which includes samples of data to get you started.
The grunt work is still done in C# so its faster than other full python based backtesting engines. Edit: I'm the Founder @ QC.
His bio and pronoun usage said as much.
>also on QuantConnect one cannot actually see the data, so there is no way to verify how good the data is...
Can you not load up the data and test it however you wish? They even say where they get their data from.
Second, yes you absolutely can verify QuantConnect's data. You can use it as much as you want within the context of their platform, you just can't download the data en masse from their platform and use it on your own. But if you have tick data (equities) or minute data (options) yourself, you can certainly verify it (which of course, you don't, because you think ebay is a good source for financial data).
I am going to continue griefing you in threads like this where you spread blatant misinformation, because at this point I'm convinced you have an ulterior motive or are in fact selling this data you keep talking about on ebay.
Pick one. You don't get both. I'm happy to share where you can get real data from.
http://www.nanex.net/nxcore.html
No affiliation with any of these.
We provide FX/CFD data for free download; the other data is restricted by the exchanges sadly so we can't make it available. Instead we put the tools we used to make it into LEAN format into LEAN (/Toolbox) so you can purchase it and convert it yourself.
1. Any chance of a Robinhood integration à la Quantopian?
2. How are the architectural revisions[0] coming along?
Also some totally unsolicited feedback:
If I'm being completely honest, I found it difficult to get going with QC. The documentation is decent, but there's not enough to avoid having to review LEAN source right off the bat. The examples also tend to mix helper classes with lower-level functionality, and that can create confusion.
The framework itself feels a bit over-reliant on OOP. Some aspects feel too tightly coupled, others too little. Obviously LEAN has been around for many years now, so architectural baggage is perfectly understandable.
A total rewrite I'm sure isn't feasible, though I'd suggest the following ethos in any case:
a) Design primitive user-accessible data structures with virtually no inbuilt functionality.
b) Build low-level components that operate using those data structures.
c) Build high-level components that compose low-level components.
d) Allow users to author their own components, and to compose components of any type however they see fit.
Pretty sure you're already on that track in a sense, so it's good to see things headed in the right direction. What keeps me from writing a custom framework is the data, the fact QC does a ton of grunt work, and that it's well-tested.
tl;dr Please break apart the monolithic QCAlgorithm class as much as you can! :)
[0] https://www.quantconnect.com/forum/discussion/1816/qc-algori...
Please do not try to trade actively unless that's your full-time job. Passive investing using index funds is definitely not sexy, but it gets the job done.
Only if you have a significant amount of capital to play with. People looking into active trading are doing it because they don't have the necessary capital to make passive investing meaningful; they're looking for much much larger returns that you can get from an index fund, which necessarily comes with more risk. Traders are gamblers.
Active trading, by definition, incurs higher transaction costs than passive investing. Since transaction costs are generally priced per trade, active trading is more costly on a percentage basis for individuals with less capital.
It wasn't investment advice, nor did I say it was a good idea; I'm merely telling you why people choose active investing: they believe they can do better and there's enough stories out there of the "lucky" person who did better to keep that hope alive. In all likelihood they're going to lose their money, but you'll never convince people trying to win a lottery that they're better off accepting they can't. I literally said traders are gamblers, obviously that means I think active trading is gambling, that's not investment advice.
My biggest thing with the Python for Finance books - I know Python, I want to learn finance. All these books are the inverse of that, for people who know finance and want to learn Python. There is a good site for quantitative economics [1] that has tutorials in Python and Julia. I would love a mathematics of finance book that had the examples in Python.
Read Options, Futures and Other Derivatives and Algorithmic Trading and DMA and you basically know everything you'll get from public sources that could be meaningful for trading.
If he knows how to code and is looking for "mathematics of finance" he should start with those, not the CFA.
http://news.efinancialcareers.com/us-en/150266/why-the-cfa-w...
On another note, even if what you were saying were true (and it isn't), it wouldn't demonstrate that it's useful to have the CFA. Certifications are usually helpful for HR or for regulation, not for actual knowledge. And everything you need to know about finance can be learned in far less than 1000 hours for trading if you're in a research or development role.
Sort by who manages real money and who doesnt. Anyone managing $1bn or more all has it. That is not very much money to manage.
The vast majority of the CFA curriculum covers investing fundamentals, including accounting (100% necessary for building models), quant (obviously necessary), econ (pretty important).
https://en.wikipedia.org/wiki/Walk_forward_optimization
See also: "The Evaluation and Optimization of Trading Strategies" by Robert Pardo
Nothing can be predicted because there's way too many confounding factors.
Astrology is story-telling with no scientific grounding.
Technical analysis is merely another name for hindsight bias. Those patterns only look like they repeat in hindsight because you're ignoring all the failed patterns that setup right but failed to play out and thus don't look like the pattern in hindsight.
Technical analysis is exactly like astrology and is in no way grounded in science. Quantitative analysis is based in the scientific process, technical analysis is superstition created by visual traders who think they see patterns but fail to understand biases. You are confusing the two, technical analysis is a pattern library of nonsense like "head and shoulders" or "dead cat bounce" based purely on visual patterns in price data that only "appear" to repeat due to hindsight bias. It is in no way remotely scientific.
I've spend thousands of hours testing trading patterns in Forex, they're all nonsense that lose as much as they win over any decent sample size which equates to random.
Edit: to the guy below, You should ask for sources that back up the claims that it does work. You won't find any, it's trader superstition, not science. You don't have to prove things don't work; you have to prove they do. At least if you want to call it based in science. However, the field of quantitative analysis itself is proof technical analysis doesn't work; it's what happened with people who actually knew math got into trading and discovered the technical analysis superstition that visual traders who lack math skills made up.
I'm not busting your balls, I actually agree with you but I haven't seen it really proven.
Not the original guy but I just want to point out that Warren Buffett, Peter Lynch do not believe in technical analysis.
It's gambling, they win randomly and falsely attribute the wins to tech analysis and dismiss all the losses as them "reading" it wrong. The random wins attribute to an addiction and superstition but they'd win just as much flipping a coin to decide whether to buy or sell. These people are very bad at math, if they were any good they'd be doing quantitative analysis and looking for actual patterns in the data that stand up to real scientific inquiry. Technical analysis is to quantitative analysis what alternative medicine is to actual medicine.
Algorithmic trading is very difficult, but it is empirical. Information asymmetry exists in the market, and if you can capture it you profit. I don't know how capacity constrained my strategy is (I think it would be difficult to work this strategy profitably with over $10M depending on how much effort you put into execution; my execution is unsophisticated and I'm working with two orders of magnitude less than that), but it's working.
Am I reading that right, you lose one trade you lose 70% of your capital? Sounds like a very very dangerous strategy with a near 100% certainty of blowing up the account. The optimal trade size for equity growth (which is itself hugely risky) is vastly smaller than that.
There is a separate risk management metric in play that automatically closes positions and pauses the strategy after a specific number of losses in a row, or after the strategy's capital allocation drops below a certain percentage (whichever comes first).
The other metrics are as stated.
Oh... HUGE difference. What percentage of the portfolio is the position? How long have you run this strategy; I ask because those are huge returns that indicate to me you're taking huge risk which can work short term but nearly always wipes you out over a sufficient amount of time.
> It earns between 3 - 15% in options trading every few days with a win rate of around 70% and an average holding time of a few hours.
If those returns could be held for any length of time you'd be wealthy in no time.
No single position can be more than 10% of the portfolio, there is no limit on concurrent positions, and the strategy cannot use more than 20% of the account's capital. Each position has a profit target, normally a 5% increase on the cost basis, and when it's opened a GTC order is immediately submitted to flip it. But this isn't a market-making algorithm or something that requires serious latency considerations - the target holding period for each position is less than five days, after which if the position is still open it's flagged. The strategy has been running continuously for only eight months; prior to that it was backtested with historical data before launching live. It used to be that the mean holding time was around 2 days, but lately it's been flipping positions in fairly short intraday scales (an hour or less), which means I'll be adding more risk accountability to it.
> If those returns could be held for any length of time you'd be wealthy in no time.
Yes...however this strategy has generated less than $30,000 in profits since its launch (on an initial outlay of $10,000), and I don't reinvest profits.
This is an options trading algorithm, and I don't think there is enough liquidity available in the targets to significantly ramp up capital from here without adding on more leverage (and risk). The other difficulty is that a confluence of factors needs to happen somewhat simultaneously in order for a candidate equity to become a target.
Due to this, it's more accurate to say that the strategy generates a few hundred dollars weekly to biweekly (on average). I may be overcautious, but this is primarily a research project for me. I would not actually be trading with real capital were it not for the fact that I'd like to demonstrate the results in the future.
When filing for its IPO in March 2014, it was disclosed that during five years Virtu Financial made profit 1,277 out of 1,278 days, losing money just one day.
They are your competitor and who will prevent a disgruntle employee or a hacker to steel your successful trading strategy?
Just buy some data from eBay, you can get 20 years of historical stock market data for less than $100 and you can test any trading strategy or idea imaginable, including trend following, buy and hold ETFs, etc...
The barrier of entry is pretty low and you can develop a great lifestyle business with no customers, employees and investors around that...
All that said, here are the 2 most important advises I wish someone told me when I started years ago:
Have an iron clad risk management strategy for your portfolio, e.g. if you are willing to risk 1% of your capital on any given day and tomorrow happens to be the Black Monday and all stock go down 50% you still have to be sure you’ll loose only 1% if you have to sell everything... This is where the back data comes into play – the goal is to back test not how much money you are going to make, but how much money you are going to loose in the worst case scenario if you have to liquidate everything and move to all cash. If you have this working 100%, you just need a trading strategy with a minimal edge and you’ll make money in the long run in the market and the compounding effect will really make you a rich person (rich person to me is someone who makes more money then he spends, WITHOUT working more than 1 hour a day)
And the second advice is never ever trust and buy a trading strategy from someone, the fact he is selling or "teaching" a trading strategy means it is not working and he is trying to make some money by scamming people and selling the dream, all those trading educators cannot make money trading and that’s why they are coming with up with those fake trading courses... just check this website to see the magnitude of this scam and how people are loosing their life savings... I just don’t know why FBI, SEC and the government is not doing something to stop those guys...
https://www.tradingschools.org/
Good luck and as I said before, just treat is as a hobby at first and spend the time testing and you’ll find something that is working for you...
EDIT: some spelling errors fixed.
you should pick a more scalable strategy, the kind of strategy that becomes more profitable the more you talk your own book.
I'm going to reiterate this right now: you are not going to find data valuable enough for trading insights on ebay. Go look at the prices of historical data for various time resolutions at TickData or CBOE Livevol and then ask yourself if cheap, competitive data on ebay is too good to be true.
If the data is competitive, it's expensive. If it's not competitive, you shouldn't be paying for it, especially not from ebay. The first time I read about him doing this I was puzzled, now I'm getting increasingly frustrated because it's blatantly false information.
if you know a cheaper or even better a free place please advice...
It would have taken me months-years to get to that point. Shortly after they also included integrations to IB and Robinhood so you could very easily take your ideas and actually live trade them. I immediately stopped development on my trading platform and just started using Q's.
Q could potentially take my algorithm and start using it themselves, though they actually put money into algorithms they think are promising and share the profits with you. That's a small risk IMHO. Hackers and the like... I did this stuff for 10+ years. People WAY over estimate the value of this code- you have an idea, you backtest, hope it works, and then you put it in the wild. I have never seen an algo that is a real guaranteed money maker (aside from arbitrage/hft stuff). What are these hackers going to do when they take a 10% loss one month and don't understand why? Some "signals" can be used in-line fairly generically with other algorithms to enhance their performance, but I will still bet on the guy that knows/understands the algorithms to not lose his shirt, then some hacker dude that likely has no capital to begin with and may not even understand the basics of financial markets.
One thing you seem to overlook is that it is one thing to go buy a bunch of historical data off ebay, but you need to constantly update that if you are going to live trade- Q handles that part as well.
If I was looking to make a business out of this, I would build my own platform. But using it as a research platform to invest my own money, Quantopian is awesome.