Money Machines – An Interview with an Anonymous Algorithmic Trader
logicmag.io
logicmag.io
A key takeaway is:
"One of the fallacies that people have is the assumption that because the people who are working at certain firms are smart, they must be successful."
The number of times a new hotshot hedge fund moved in to office building where I rent, only to be gone without a trace in a month is staggering. Usually, they'll raise money from an unsuspecting family office or two, employ a techy and an admin person to do the heavy lifting, whilst the smooth talking CEO alt-tabs between Bloomberg and reddit all day. Most have terrible risk management strategies, and the majority are simply trend following (sorry, machine learning/AI). Some of these firms are also my clients, and I have often needed to write off money they owe because they've literally gone bankrupt overnight.
I'm right now working on optimizing my long term investment strategy, but the hardest thing is that there's nobody to talk to it about it (unlike when I have other coding problems), and I have no idea when I'm doing something totally stupid in my code base. Getting the right metrics to optimize is really hard, as mostly I'm just looking at numbers and graphs.
I worked for a large investment bank, was paid well, but was also rather unhappy. I'm not a fan of office politics and also hated the fact that what we were doing was actually quite unsophisticated, yet we dressed it up like the most complicated thing ever to clients.
We were using wavelets for analysing market micro-structure and using any generated signals to influence our trades. Wavelets were the topic of my thesis, so I was a good fit for the team. I know it sounds complicated, but if you looked at the signals we generated, they looked awfully similar to a rather basic trend-following style regression model. I believe we found a very complicated way to compute signals that a simple trend follower would compute in far fewer CPU cycles.
Anyway, I got a fairly decent bonus one year (because markets were trending nicely), and decided I want to set up my own tech company. I was always a nerdy guy, and I thought I could contribute more in the tech space. With hindsight it was a good move, and financially I believe I am now better off than most of my trading colleagues.
My advice to new guys joining our desk (maybe it helps you a bit): focus on your exit, more than your entry, and if you can help it, be really, really, really lucky :P
Doesn't this describe nearly all of the financial industry? Do you happen to know of any exceptions?
The market is, in theory, not a zero sum game. Algorithmic systems, in practice, are. In theory the only way you can genuinely beat the market is when you find an inefficiency in it. And inefficiencies are generally based on a lack of knowledge. If everybody knows what you know, that inefficiency will no longer exist. And now pair this with the reality that most (and it's probably safe to say all) algorithmic systems are not really especially complex. That doesn't mean they're easy to make though. The 'secret sauce' is in exactly what data is used, how the pieces you have interoperate, the way inputs and outputs are massaged, and so on. But this drives a huge need for secrecy. Nobody who is successful is going to want to talk about how their system is designed largely because most would not be especially difficult to replicate if they provided more than very abstract details.
And given some metric like trading volume how much does that affect that number?
I've heard of some hedge funds raising way less than they could have, maybe because their strategies would be killed much faster with 500mm than with 50mm ?
You know those books that show you how to get rich doing X? Those writers make far more money from the book than they do from X.
No doubt you can find corners of the investing internet that will discuss these things, but you'll probably find an awful lot of chaff in your wheat, because the successful ones won't be chatting about it on forums.
Now on a more constructive note, optimising your long term investment strategy is probably best achieved by getting a pension with some cheap trackers, and a savings account. If you're already doing that then great because that's 95% of your optimisations there.
I wont recommend any web sites, because the details will depend on the country where you live.
How long did you do it? Is it possible to make a living as an algo trader?
I was officially trading (i.e. coding the strategy) for just under 3 years, then got offered a senior non-trading role (but my pay was heavily linked to the desk's performance).
Can you make a living? Yes, if you're doing it as a salaried employee at a reputable institution. If you're going to try and algo trade with your 50k of savings, I would say "unlikely", and you might cause yourself a lot of anxiety.
When you say yes to making a living but as an employee is that because of at an institution you are trading with much larger sum of money and hence there is enough return to pay for your salary bonus and then some or something else like getting a regular paycheck in a other wise very volatile return world? Essentially is the amount capital the key thing or something else.
Also how do traders typically come up with a strategy? Original insight or mostly copying the herd with their own variations?
Regarding strategy generation, it's a mix. Quants like me almost always take the systematic, research like approach - analyse data, see if you can ascertain any predictable patterns and then paper trade them in the live market. If it makes money on paper, it might make money in the real market. Then just iterate and refine. Some guys do trade with their gut, but are rarely successful.
I work at a big quant/algo prop trading firm. The reason you can't compete is that most successful algorithmic trading relies on economy of scale.
Yes capital to trade with is one aspect, but more importantly is the amount of knowledge needed to carry this off. Building a safe, robust strategy is hard even if you have a working idea.
It requires probably top 1% knowledge in distributed systems, Linux admin, networking, database administration, security and more. The chance that any individual (or even team of less than 5 people) has those skills covered in addition to the quantitative know how to actually come up with a strategy is quite small.
There is a reason why algo trading is dominated by large firms. Working at somebody like Citadel or Optiver, if you say "oh damn, there's this weird effect on our server. When x happens I get a period of noticeable slowdown" they will turn around and say "Ok, well John here is a core dev on the Linux kernel, he'll come to your desk and help you figure it out".
Imagine that but with every problem pretty much. Running trading strategies you will hit problems. The level of work to overcome all of them satisfactorily whilst not degrading your edge is so much work for an individual I suspect you would burn out pretty sharply.
You have to code:
- Feed handlers
- Back office
- Research platform
- Strategy execution
- GUI
You need to have:
- PB agreements
- Network access (telco the world over suck as doing this in a reasonable time)
- Exchange memberships / ISDAs (depends what you're up to)
I guess you could have spun out from a larger group with some of this stuff, but then you'd probably not crash and burn that fast.
I do agree with the sentiment that fast talkers get more money than they can handle.
Also a lot of the well known quant firms that hire loads of phds don't actually do anything particularly interesting or profitable. They are simply benefitting from institutional momentum: there are still pension funds and family offices with a mandate to stick money in trend followers, so the big trend following names hold on to their assets.
Not sure where to go from here honestly. Should the code evaluate the data in real-time for each trade, or wait to the end/start of the trading day and do some evaluations then make a decision?
Obviously, I am not going to be able to compete with the big boys using really sophisticated code (written in a low level language like C), executing in colocation right next to the exchanges, running Apache Hadoop and Spark. Still, I am interested in just tinkering with simple algorithms with the data.
Created it after not being able to do livetrading on Quantopian.
Here's the associated python library and documentation: https://docs.kloudtrader.com/ https://github.com/KloudTrader/libkloudtrader
There's a lot I can talk about but here are some points:
- Most of the time you don't trade instruments in isolation. They are all correlated (even if inversely) one way or another , so movements in one will affect others. Having a myopic view of just one instrument will have too much unexplainable randomness.
- The other way to scale is having more data inform your pricing - i.e. don't just look at the prices of one or more instruments. Look at twitter, look at weather reports, news etc. It kind of suggested that in the article. The meaningful movements in the market are most of the time due to truths outside of the price itself.
- You have to decide whether you want to trade directionally and long term (like a hedge fund) and take on positions over time, or trade in and out of positions quickly (like high frequency trading firms) to minimize risk. You can be anywhere along that spectrum. The faster you are, the more myopic you can be and just react to current trading activity of other participants. The more long-term you are, the more you have to look at the big picture, more data, more instruments, making sure you have the right balance in your portfolio (whatever that means to you).
I'd be happy to expand more :)
I'd say starting a "pure" HFT firm is incredibly hard nowadays. There's a big upfront investment and huge operating costs, and there's a lot of consolidation happening in the industry.
In general, most hardware and software is tailored to high-throughput, and a huge part of the effort of HFT is fighting all that to prioritize latency. If you want to compete with the very best firms, that means renting rackspace in a colocation with the exchanges, overclocked CPUs, expensive network cards, FPGAs, paying out the nose for the most accurate market-data feeds, and many order-entry connections. Then you have to tweak the BIOS, kernel settings, isolate your processes on cores, write your own userspace networking stack etc. etc. etc. On top of that if you want to trade between different colocations, you might need a network of microwave towers because fiber is too slow (light travels about 2/3 slower in fiber because it effectively "bounces" along the fiber). At the end of the day though, if your algorithm or strategy doesn't work, you won't make any money - speed is only a part of it. At the previous firm, we had a market making desk that was very profitable, but was quite "slow" by HFT standards (triple digit microseconds vs sub-microsecond response times). You still need to be smart about it.
Having said all that, I think HFT gets a bad rap, about being a waste of resources/talent, and somehow "stealing" from the common man. The truth is, they are only "stealing" from each other - they compete to remove the tiniest inefficiencies in pricing. They ensure the cost of trading is actually as low as it can possibly be. By themselves they don't have enough capital to move the market in any meaningful way. In fact, most flash crashes occur when HFT firms step out of the way! They move out of the way of huge market movements because they can't cushion the blow without taking an unnecessary risk as a business. It's like asking your plumber to fix your sink while your house is on fire. Market orders, stop orders and panic are most of the problem when it comes to flash crashes. PSA: Please use limit orders :)
I might ruffle some feathers with the comments, but I think overall HFTs aren't the bogeyman most people think them to be. Sure there are some bad apples, but that's true of any industry, and it comes down to the people - not the system itself.
Are you saying, it was guaranteed to make money? I.E. the algorithm never failed, or never made losing trades? That seems insane, to always be able to make money even if it's fractions of a percent.
Recently, when the market got way overdone and oversold, I'd even call some of it panic selling with the bottoming happening Christmas Eve. I am curious how I can be more analytical, and data driven to make decisions during these events, rather than just gut feeling of panicking, negative news, and oversold conditions. Well I did add to my Apple position in this late December, I should have pulled the trigger more. I was somewhat conservative, missing the huge runup last month in January of most all stocks.
I think you're on the right track regarding value investment, if you're thinking about it long-term, and it's for personal investments. Passive index funds like SPY are the best bet for most people. Finding companies or following trends you in particular have some insight on can also help. Things like "having worked in industry X, and having read their whitepapers, this new company is clearly overbought and is all marketing hype".
If you want to get more analytical, look at portfolio theory, different hedging strategies, and try and find "alpha" - roughly meaning the extra factors explaining the price of an instrument that gives you an edge over others. Check out Quantopian - they have some nice articles too. The key is coming up with a model that accurately captures the risk of your assets which would allow you to properly allocate your money amongst them.
If you start expanding beyond a small set of specific companies, don't forget to find data for delisted companies to avoid survivorship bias.
E(log(wealth)) - c * std(log(wealth))
or
E(wealth) - c * std(wealth)
The first one is better for building up wealth over a long time, but when I looked at Modern Portfolio Theory, it is suggesting to use the second formula.
I'm also planning to spend about 5% of my wealth every year, and the second number gives me lower risk in the short time frame, so I think something in the middle would be a better criterion.
I try not to worry about stdev and focus on the soundness of my process, but in practice I have money split into higher-risk where I try to be clever, and low-risk failsafe investments to raise the floor of the worst case scenario.
Try this strategy and see if you can automate it. It's called the wheel strategy. What you want to do is setup a $200K account, and sell puts with certain deltas, and then if assigned, sell calls with certain deltas, until you get called. This will get you a relative return of up to 20% per year if done well. You can use your skills to figure out the best delta position to trade, you can look for high volatility to take advantage of. Once you get this completed, you can look for other trends. I always recommend starting with this because it's the easiest way to make some money in the market. It's a lot of work to do it manually, but automated, it would let you earn income while you are not doing much.
You will also want to read and understand what calls and puts are, and what selling/buy calls/puts entails. Probably want to do this with stocks like Disney, GE, and so forth, something with large market caps, small movements, and something you are willing to hold long term and which will not go bankrupt/drop significantly in the near future, disclaimer: I do not like GE at all, however, you also want to try this with 1 contract to start, because 1 contract is 100 shares. You can probably work with a smaller starting capital. If you get this implemented and working, it can generate passive income for you in a sense, or you can blow an account.
Edit: You want to settle/close out around earnings because the volatility is insane at that time.
This is a fairly easy strategy to automate and you can use a small account to do so.
Generally speaking, if someone offers a simple strategy for getting well-above-market returns, they're either lying or failing to mention that this strategy occasionally bankrupts you. I think this is the latter.
I don't utilize it because my strategy is much better.
The risk with the wheel is getting assigned stock which you hold forever, but you sell calls against it and over time it ends up being profitable.
> You’ve already got robo-advisors, which use algorithms to manage assets for retail investors. We’re also probably only a few years away from you being able to log into a brokerage account and run a sophisticated institutional-grade algorithm yourself.
> People tend to assume that the diffusion of these technologies is a good thing. I’m more ambivalent. I think it could be a big mistake to have the population at large play around with algorithms. Some people who are very good at it might benefit from having access to this broadened toolset. But most people will just end up paying too much or make bad decisions because they’re being given access to a technology that they aren’t equipped to do anything useful with. They can lose money with it, however.
Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges.
Those folks with fiber lines into the exchanges are definitely not running hadoop.
algorithmic trading is market distorting. its not actually helping "us" with real value for our futures, funds, company value. Its not informed trade on real conditions, its trade on the mechanistic moments in trading message flows.
its a really bad feedback of second-order differential.
Surely waiting a day to act on the information isn't helpful. How about a minute? A few hundred ms, about the speed of a conscious awareness? Getting down to 10us seems like the logical conclusion of all this.
I think the more nuanced view was talked 3/4 of the way down: with algo trading, prices are based on the models not the fundamentals. And then you add in models of what the other guys are doing. And pretty soon you don't care where the price should be, you just care about computing in 10us where everyone else thinks it's going to be in 15us.
> Another fallacy in the lead-up to the financial crisis was the assumption that financial markets were so efficient that participants didn’t need to do the underlying work to figure out what the securities were actually worth. Because you could rely on the market to efficiently incorporate all available information about the bond. All you need to think about is the price that someone else is willing to buy it from you at or sell it to you at.... if they assume that the price of an instrument already reflects all of the information and analysis that you could possibly do—then they are vulnerable to that assumption being false.
That's now flash-crash territory.
If it were inefficient, unless the algo traders were putting arbitrarily large sums of money behind their positions, they would get eaten alive. Since the successful ones don't, it seems they are just pricing market reactions faster
All trades would be looked at by an actual human, making sure that everything made sense and lined up.
I have my own doubts about a purely algorithmic approach (for now at least). Computers are great for many tasks, but for something as inherently irrational as the markets I think that should be at least partially moderated by a person with a deep understanding of the markets.
> The level of human oversight varies. Among sophisticated quantitative investors, the process is fairly automatic. The models are being researched and refined almost constantly, but you would rarely intervene in the trading decisions of a live model. A number of hedge funds, mutual funds, and exchange-traded funds (ETFs) run on auto-pilot.
https://qoppac.blogspot.com/ https://twitter.com/investingidiocy https://www.systematicmoney.org/
Also has a few videos scattered around that are worth a look too.
I would imagine it may be easy for an unsophisticated/hobby "algo-trader" to make this type of mistake but with over a million in capital, you should probably be a bit more prudent with risk management.
Basically every time I bring up jobs, ai and society someone comes out of the woodwork to insult my character or sarcastically dismiss me. Never any substantive counter-arguments. I’m really looking forward to what you guys will come up with this time. Always such a pleasure.
Jobs have been eliminated in America because of poor industrial and trade policy. Nothing to do with "AI." I get extremely irate when people make the assertion that "oh, well, the jobs are going away soon anyway" -using this complete and utter falsehood as an excuse to continue the looting of the country.
The guy interviewed here is a real mixed bag. Some of what he says is nonsense, some of it is accurate. FIRE has basically been a vampire squid on the economy without adding much value. Most of it could be "automated" away by using a magic 8-ball; it doesn't actually provide any value. Except the politicians rice bowls depend on the status quo.
What gives, you my ask. I sit in one corner of a 6 desks arrangement and the rest of these desks plus another arrangement of 6 desks are occupied by a company in the RPA bussiness.
The CEO of the company sits right next to me (When I started working in this space almost a year ago all these desks were empty, so obviously I took the best position) so we have a good couple of chats the days he shows up (They have a "formal" office in the city).
They have a couple of whiteboards in the wall next to my right, where they use actual, physical pseudo kanban boards with postits and shit that get reorganized probably weekly, because they can't cope with the amount of new work they are getting.
They have doubled the number of programmers since May, in a country (Argentina) where things are not looking that bright for nobody.
From my desk, I can _feel_ the pain of all the jobs they are killing.
I can't tell you _exactly_ what "large group" of jobs they have killed because the chats are not that intimate, but I see the list of companies they are working with, and all the big ones from the country are there. all of them.
They _are_ killing jobs, a plenty, without discrimination. And this is barely starting.
Some VC funded startup allegedly working with companies does not equal killing jobs with "AI." If this were really happening it would be real easy to document! The X industry is gone now. Like buggy whips. But it's not; it's science fiction. I know, I know, people _really_believe_ -that doesn't make it real. Numbers, people: name names, no anecdotes about the stories from people next to you in the WeWork or whatever the journalistic dipshits are releasing from a PR agency.
I'm happy to place a long bet at whatever odds you like for whatever specific prediction you care to make if you want to put your money where your mouth is. Auger, "long now" whatever -I'm good for it.
If banning me is such a horrible burden for you, you are free to email me so that we can discuss a solution.
When a comment is both low-information and personally negative, it becomes trollish almost regardless of topic. I think I read "So you can’t reason", etc., as personal, though looking at it now I see that you meant "you" in the sense of "one". Sorry! Nonetheless, you've posted such comments in the past, including recently (e.g. https://news.ycombinator.com/item?id=19099603, https://news.ycombinator.com/item?id=19110921), so the main request stands.
I had a couple of chats with the CEO about this, and he mentioned briefly that this affected jobs with a high level of repetitive tasks (E.g., invoice processing and stuff).
There was one process we kind of discussed at length enough as to answer this: In my country, it's normal for supermarkets to have 3rd party employees to take care of certain aisles that have only products from the 3rd party.
The thing is, putting products in order there is a time consuming process, and obviously prone to error.
The change would be, the employee would take a picture of the aisle once finished, send the picture to a service that would process it automatically and alert the employee before she left the supermarket if there was a problem or anything (Like, a product needing replacement or similar).
> All you have documented here is some company doubling the number of programmers since May
Yep, that's right. the downside of this, is that there are a lot of companies looking to automate processes that are time consuming, and that affects jobs directly.
hat the company decides to do in the end regarding the people in those jobs is up to the complany obviously. Sorry about the late reply, thought.
E.g. secretaries, call center employees, accountants, etc.
Anyway parent poster said "a lot of people will lose their jobs soon" because of "AI." I deny this has or will happen, and people's misunderstanding of it is downright criminal.
Information systems have created the ease of the gig economy. It allows people to provide more granular, on demand jobs.
The net effect is more jobs. This may be a bad thinf
[1] https://www.bostonglobe.com/opinion/2014/05/09/the-lawyer-bu...
To me, AI and driverless cars are hyped too much. Computer companies like to overpromise and then under deliver. When I first started in my career, CASE tools were going to kill off the programmer, fast forward 20+ years, and developers are still hanging around coding stuff.
Are jobs going to change? Sure.