AI That Picks Stocks Better Than the Pros
technologyreview.com
technologyreview.com
If long-term investors were the only ones buying and selling shares it would be much more difficult to buy and sell shares when you wanted to. Net result: shares are a less attractive investment, IPOs raise less, economy suffers.
And what you are proposing is that we should only allow Investing and restrict/remove trading. The problem with this approach is that you lose your ability to find 'market value' for your investment on an everyday basis. Let's say we force everyone to hold stock for at least a month. What if I had bought BP stock just before the gulf oil spill news came out? I will be stuck and would have to incur a huge loss by the time I am allowed to sell my stock. Also, there are thousands of investors and they may need to liquidate their position at any time based on their own circumstances.
And finally, stock market does achieve their goal of allowing company to raise capital. Theoretically, it doesn't matter what happens to the stock price after company has issued their common stock. They got their capital and can do what they want with it. On practical basis, company do care about their stock price for numerous reasons which I won't get into right now.
Yes, and No. It's backwards. There's no `company' which can care or not care about its stock price. The company, as a legal person, can't really care about anything at all.
And after the IPO the shareholders are the new owners of the company, so they do (or should) care about what happens to the company.
It's not a question of whether life will go on. It's a question of whether it would go on better or worse than it does now.
The question is cost/benefit of prohibiting short-term transactions. So far I'm sure there's a cost, unconvinced there's a benefit, and suspecting that a lot of the opposition to it is based on the natural human urge to be jealous when other people make money by doing no apparent work.
That's a small part of automated trading.
Honestly, if you are buying stocks long-term, you probably aren't focusing on short-term pricing changes.
Large institutional investors can and do trade off-market entirely via Posit, block desks, other dark pools, etc.
But yes, you do not make large trades transparently in the marketplace, because people will take advantage of that. There is no way to trade without leaking information as well.
(FWIW, I worked on an automated execution desk.)
FWIW, I work at a risk analytics company. And if you worked on an automated execution desk, please excuse me for suggesting that you may be more inclined to see and emphasize the positives of what you do every day. I fully admit that I'm probably more inclined to see the negatives, but if you're really suggesting they aren't there, you're kidding yourself.
That's a small part of automated trading.
It's a small part of automated trading, but reading the news to pounce on something and holding it for only 20 minutes clearly falls in that small part. We're obviously not talking about filling stop loss orders and the like.
Honestly, if you are buying stocks long-term, you probably aren't focusing on short-term pricing changes.
That's just wrong. Funds are compelled to make transactions for a wide variety of reasons all the time. They are very interested in what happens to prices when they start to move to fulfill those obligations.
Large institutional investors can and do trade off-market entirely via Posit, block desks, other dark pools, etc.
"Can and do" does not mean "always and only".
But yes, you do not make large trades transparently in the marketplace, because people will take advantage of that. There is no way to trade without leaking information as well.
But the point is that automated trading systems are far more sensitive to that information, and can act on it much more quickly, meaning the methods to minimize the transparency of a large trade have to become more and more sophisticated, meaning more investment in those methods == tax.
I think that if you are holding for a long time, you probably don't care as much about the difference of a few pennies. The alpha due to long term trades needs be much larger than short term volatility to be able to make a profit. Whereas a lot of high speed trading folks seem to be trading around the volatility and short-term moves.
I'm not really speaking about things about positives vs negatives.
There's really no way to balance all of the desires you want. I think the fairest thing to do would be to have a once-a-day crossing for the stocks. But this would take away continuous trading, which has other advantages.
(More specifically, I worked in a group that did automated trading for other parties. Probably mostly like the big folks you were talking about. However, I worked mostly in proprietary trading strategies directly. A good chunk of it involved risk model improvements.)
A few pennies? What do you mean by that? High speed trading only nets fractions of a cent per trade, yes, but by the time a big order actually gets filled, you could be talking a few pennies per share, or more. Another part of the tax that I'm referring to is the cost of keeping pace in the arms race (or hiring people to do it for you). That certainly makes a difference to the kind of low-margin funds that might be trading because they have to more than because they want to, which also happen to be the kind of funds that most people who don't realize they're invested in the stock market are invested in (which is where the positives and negatives come in, imo).
<< Automated trading is a tax on transactions, not a lubricant.
My next question then, is whether this flexibility to sell instantaneously is worth the risk of near instantaneous drops of 700+ points in the dow, as happened a couple of weeks ago. My gut tells me that letting computers have a free run of it is not good. But I'm definitely open to being convinced otherwise. And I'm definitely open to a better solution than you have to hold on to stocks for a month or more.
So I'm inclined to pick the free-market devil I know over the government-controlled devil I don't.
The "devil" of returning to that level of regulation hardly seems unknown.
playing devils advocate, imagine how much more we would have grown had the financial system been unregulated.
The problem is that there does not (currently) exist any entity or human being smart enough to price stocks correctly. The effect appears so dramatic simply because markets move quickly, and the low transaction costs do little to damp the oscillations taking place.
The "pain" caused by such sudden oscillations is certainly real, and can be thought of as shear stress as the financial system pulls on a slow-moving economy. But rather than arguing for damping the financial system, as you do, I would prefer to lower transaction costs in the rest of the economy. At some point, this becomes difficult: the bottleneck is human beings' capability to respond at high speed (that the financial sector lets computers do most of the trading nowadays is clear admission of this weakness).
The only solution, then, is to let computers make all the other decisions in the economy as well. Hence, I propose a tax on all financial firms, to be deposited directly to the accounts of the Singularity Institute.
(You may decide for yourself how much of the above you consider to be serious..)
There was an article on this a few weeks back where a guy in the know was explaining how such systems are a bad thing. Say you bid a price of $2 per share for a share that is currently at $1.92. Since the advent of HFTs, there's a much higher probability that the price you eventually pay will be a lot closer to $2 than to the current $1.92, (perhaps $1.98 or $1.99) whereas before HFTs you could get it closer to the current price. This matters particularly when you buy a lot of shares, and when you hold them for a long time, as the couple of cents difference compounds a lot over time.
I'm not an expert on the subject, that was my takeaway from it.
cf.: http://www.businessinsider.com/john-carney-matt-taibbi-falls...
You have to specify more exactly what you mean when you say the current price is $1.92. Is it the current best (highest) bid, best (lowest) offer, or last traded price (this is generally the displayed price, but there is no guarantee that you can buy or sell at that price, since that depends on current outstanding offers and bids.
Suppose the last traded price is $1.92, and the current best bid (highest price anyone is willing to pay) is $1.90 and the best offer (lowest price anyone is willing to sell) is $1.95. You can either decide to buy from the best offer or bid higher than the current best bid and hope to find a seller.
Where HFT comes in is in that the increased amount of money and number of participants in the market tends to cause the difference between the best bid and best offer prices ("the spread") to go down, making it cheaper for everyone to enter and exit the market as they wish. This is the market making effect hugh3 referred to above.
Please note I am not an expert and may turn out to be wrong on some of this.
Say you want to buy your 3000 shares, and you don't want to spend more than $20/share, so you enter your bid with a $20 limit. Your limit is supposed to be invisible to the market, only you and your dealer know it. But the HFT algorithms have a way to probe your order and uncover your limit.
The stock is currently trading around $15, so your order hits the exchange, and in the first few milliseconds, a HFT computer makes a micro-offer for 10 shares at $16, then 10 shares at $17, then 10 shares $20, then 10 shares at $21.
When the last offer is rejected b/c it's over your limit, the computer then guesses your limit is $20 and offers to sell you 2960 shares at your limit of $20.
All this happens in just a few milliseconds, and it just cost you roughly (2960x20 + $20 + $17 + $16 + $15) - (3000 x $15) = 59,268 - 45,000 = $14,268. That's 30% more than you would have paid had you been able to execute at $15.
It's basically cheating, and should be made illegal. Just one more example of how our markets are rigged.
Remember, the price is a two-way street. It's what you are willing to pay, and it's what the seller is willing to sell for. If you say, "I would like to buy at $20" and your partner says "$20 is fine with me too", and the trade happens at $20, then the market is working perfectly. You could have gotten a better deal, but it's not what you asked for.
However, he basically has the ability to read your mind and see that he can negotiate you up to $20, while you lack the same ability to reciprocate.
It's an unfair advantage enabled by large firms having the money to buy better access to market mechanisms and powerful enough computers to exploit them.
EDIT: Before you go on to say that football and investing are two very different things, let me state that I am fully aware of that. I've had that conversation before when this topic comes up. Common people are able to participate in the investment process while only a very few get to play in the NFL. The people that are upset are the ones that try to game the market themselves but are under-capitalized. From prior conversations, long-term investors simply don't care about these things. The people all up in arms are the one's that have lost money because they are trying to do something they are not in a position to do. I'm only talking about trading stocks and futures here; the more advanced investment products, like CDOs and CDSs, and the issues in the news regarding them, are a totally different conversation.
Participants getting exploited withdraw and find other ways of getting what they need, most recently through dark pools.
Crucial liquidity diminishes, and eventually all that's left are the crooks. The suckers are broke and honest non-suckers are gone.
If you want functioning capital markets, you have to regulate and crack down on stuff like this.
The flash crash was exacerbated by HFT, not caused by the absence of it.
Congress doesn't have to outlaw HFT, but it does have to outlaw HFT order limit probing. Though it would probably not hurt to completely unplug HFT anyway.
There is zero net benefit to the markets from it, and there will be plenty of trading and liquidity without it (just as there was before it). Arguably even more, as funds that now resort to dark pools return to the open markets.
Volatility spikes would still exist (always have always will), but HFT exacerbates them, lack of HFT mitigates them.
The problem with HFT is that it is not a productive enterprise. It does not take raw materials like copper and sand, and apply labor, capital, and innvotation to turn them into products worth more than the sum of their inputs, creating real wealth. It doesn't mine or farm those raw materials. It doesn't even provide a specialized service that saves people time/money.
All it does is skim off the top of the global capital flow, increasing the cost of that capital to everyone else, particularly to the productive enterprises mentioned above. It is parasitic, and gives nothing back, despite the 'story' Wall Street has created to justify it.
It doesn't even provide a specialized service that saves people time/money.
You are so far from reality here. Do you understand what the futures markets are for? Do you understand that there are a whole class of speculators, among them are HFT firms, that stand ready to do business with a farmer growing corn, a gold miner mining metal, a cattle producer with cattle to go to the market, etc? By providing liquidity, the farmer can focus on farming, the miner can focus on mining, and the cattle rancher can focus on ranching (whatever they call it) instead of worrying about what they may sell their raw goods at in the market four, five, or six months from now. So, no, I am not out in the fields digging up copper and sand, but I, and others in the industry, make it possible for those that do those things to focus on doing those things. If I figure out a way to make some extra money by using mathematics and computer science, then so be it.
We should cease all activities deemed "non productive" by the state and get back to work in the mines.
The market is supposed to be fair and a level playing field for all those who can afford to buy stock, but the HFTs tilt the odds in favor of the select few.
When the odds are tilted in favour of a select few, it becomes a problem for all. The bond markets are far less regulated, thereby allowing Wall Street to profit massively from confusion and a lack of regulation - Wall Street makes most of it's profits on the bond market. The result is that insane toxic assets get sold, all because the industry can supposedly self-regulate. Result: global catastrophe. (See "The Big Short" by Michael Lewis for a great read on how Wall Street caused the financial crash of 2008).
Meanwhile, some even smarter guys at an investment bank just down the road know all these tricks, too, and have come up with a way to algorithmically spot them happening more accurately or more quickly than anyone else. So they start jumping in front of your trades, not because they want the stock, but because they know you want it and are probably willing to chase a rising bid for a while. Then all the other algorithms start to notice, and they start doing the same thing.
You, the friendly investment manager, are now financing both sides of what really amounts to one of the coolest deathmatches going, a bunch of math and computer geeks playing on your quarter and keeping a penny for the honor of letting you watch them do it. But you actually don't care, because it wasn't your quarter in the first place, and now that your fund is back in order, you're not going to get sued, and your customers, who had no idea that any of this was going on, and who may not even realize they're your customers, are back to paying 2% fees for (statistically likely) below-market returns.
In the past, these kinds of large trades would usually end up splitting the difference, with 'real' sellers earning a bit more than what they would have otherwise settled for and 'real' buyers paying a bit less than their actual ceiling. But now 'real' buyers end up paying close to their high bid, and 'real' sellers end up earning their low ask, with the bots fighting it out for the very thinly sliced middle territory.
In one sense, I do agree that this is an example of a market working perfectly. Algorithmic traders are exploiting an arbitrage opportunity between mainstream industry practice and cutting edge technical ability, and just like theory would suggest, they're making bank from it.
I think this is all there is to be said. Your trading has a pattern that makes them money. The algorithmic traders are going to use that information advantage to make money. That's the idea of the market; trying to extract as much value from securities as possible.
Even if you impose arbitrary trading limits, the big banks are still going to find a way to make money off of you.
Under normal market operation, sometimes the market will trade upwards during the time it takes to fill your order and you'll pay in a range from whatever the market started at up to your limit.
Sometimes the market will trade down during the time it takes to fill your order and will never hit your limit, especially if you make the order while there are more sellers than buyers (say BP after the accident).
So over the course of many trades you may be able to expect average execution prices around where the market was initially trading when you placed the order, since sometimes you'll pay nearly exactly that, sometimes you'll pay more, and sometimes less.
But as these HFT limit probing scams become more widespread, execution averages will skew toward being more expensive for the party on the other side of the trade from the HFT, which is typically big mutual and pension funds that deal in millions of shares, or Joe Six-Pack buying through some less-sophisticated Etrade or whatever.
If anyone out there had a sell order at less than $20, their orders are immediately matched to yours. Lets say someone wanted to sell 1000 shares at $15.1, someone else 500 shares at $15.2, and nothing above that. So you just bought 1500 shares at $15.1 and $15.2. Now there are no shares available to buy below $20, but your order is only half filled.
What happens next is that an HFT system will rush in with a sell order of 1500 shares at $20.0. Your order is filled within a second, and at the price you asked for. Without the HFT system, you'd need to wait a few seconds for a human to come along and fill your order at $20.0.
Now, there are certainly means of hiding one's intention. There are also means of finding out where hidden liquidity exists. I will admit to that. Look up icebergs and strategies to detect icebergs. Just because people attempt to hide their intentions while some try to discover those intentions, that doesn't mean there is anything suspect taking place. Information, or the lack of information, is important.
On the day in question, the algorithms' "oh crap" mode kicked in, the bots no longer felt they had a grasp on fair value, and decided to withdraw all their bids and offers. Now whoever was left trying to sell (like all the people with STOP orders) had to do so in an environment with very few counterparties.
The stock exchanges give these market maker bots a few special privileges in return for them providing liquidity regularly. It varies by exchange, but one example is a market maker must quote a spread of no greater than 10 cents, and provide both a bid and offer at least 99% of the time. In return, the market maker gets a special spot in line when orders come through (think of it as first dibbs on certain proposed orders).
So it sounds like we should either blame the guys who put in market stop orders, or we should blame the exchanges for not requiring more strict rules for their market makers (like 100% uptime). You can't blame the algorithms for wisely using the 1% of the time they are allowed to hide and wait for more opportune times.
The algorithms were not turned off because they didn't work. They algorithms were turned off because everyone knew the exchanges were going to break trades.
If your smart algorithm finds the bottom at 10, buys, sells at 20, and the market goes up to 30, you just made $10, right ($20-$10)? If the exchanges break your trade at 10, you are now short the stock at $20, and the current price is $30! Unfortunately, it's impossible to predict whether the trade will be broken; it's a human making that decision.
Incidentally, most high frequency traders get no special privileges. There are a few that do (the ones you describe), but they are special cases rather than the general circumstance.
Now it's as if you sold but the buy never happened (i.e., you sold stocks you didn't own). That's a short.
It seems to me the only reason such speed is necessary is to beat everyone else, and that's only necessary because everyone else does it.
Um, sure. It would be a bad idea for me if I'm not allowed to sell the stock I want to sell as soon as I want it, and it would be a bad idea for somebody willing to buy from me, if they are not allowed to.
Oh, you're asking why it's a bad idea for you? I don't know, maybe it's not. Do you think I should be prohibited from the trades I like because you think it's a bad idea?
The stock market is therefore a perfect balance of supply and demand and any discrepancies are immediately rectified.
By taking out some of the profit of knee-jerk reactions to news, the AI encourages people to be more restrained in their reactions, to consider longer-term effects, which makes for a more stable market.
This high speed, in conjunction with its distributed "cloud" nature, makes the market the most efficient means of allocating resources conceivable. In regulated industries -- say, healthcare (viz Medicare/Medicaid), media (viz FCC), etc. -- decisions take months or even years, and are based as much on lobbying and pandering as concrete knowledge. The markets, on the other hand, produce an efficient allocation in fleetingly small timeslices, and do so with complete objectivity. At these levels, greed even crowds out corruption.
I believe that those who object to the kind of automated trading described in the OP are dreaming that they could be the ones to score a windfall on some specialized knowledge, rather than have to play on the same playing field (scoured of arbitrage opportunities) that everyone else does.
Given that some of the movements in the stock market have a lot in common with a random walk, you have to excuse my skepticism in wondering how much is signal and how much is noise.
Please note that I'm not taking the absurd position that there's no information to be found. Certainly, there are real events (e.g. SCOX going into bankruptcy) which drive stocks down. Rather, I'm of the opinion that most of the "information" in market movements is noise, not signal. And given things like that "flash crash" we can see that there are times when the noise can completely swamp the signal to a degree I think everyone should be able to recognize.
Now imagine 10,000 people (and robots) all doing this simultaneously. This is how markets work at the micro level. It may sound like a self-fulfilling prophecy but it happens so fast that the pattern never really exists to begin with -- it's just a way to explain why there are no patterns.
At least theoretically. Most traders (by head count, not dollars) haven't the tiniest idea what they're doing. Luckily, these "noise traders" all tend to counteract each other, and when they do manage to push prices off-course then someone who does know what's going on steps in and pushes things back in the right direction, making a profit in the process. Sometimes there are catastrophic failures, but while they might scare investors they don't affect the ability of the market to predict pricing long-term. The broad markets recovered extremely quickly during the flash crash and without any regulatory intervention.
As for whether the random walk means the pricing is "noise": the market price is an estimation of the fundamental value of the company, which is an imaginary number that is impossible to calculate but is occasionally pinned down when someone gets bought out. Because it is an estimation, and because the fundamental value is also ever-changing due to uncountable external factors (interest rates, consumer confidence, phase of the moon), the result looks like a random walk. It's a noisy estimation of an imaginary number. Neat, huh?
</wall-of-text>
The only time money goes from investors to companies is at public offerings (initial or otherwise). All other transactions in shares (be it by long term investors or by high frequency hedge funds) do not benefit the companies directly. But they do provide an indirect benefit: without the existence of these investors (i.e. the secondary market) people would be very reluctant to invest in an IPO as they would be no exit strategy. It is important for people buying in an IPO that the stock will be in demand in the future. The holding period of the people who will buy is irrelevant, the size and level of aggregate demand is not.
This being said, I am not arguing that have it is a good thing that the majority of volume is high frequency in nature.
Markets exist because people want to be able to buy and sell things efficiently, in one place, and make money. They are not out to serve a charitable goal. Buy-and-hold investment is not somehow "better" or more moral than fast trading, both are just choices made by individuals, companies, or computer algorithms operating on their behalf.
Think of it this way -- there is a buyer or seller on each side of every transaction this algorithm does, who is willingly (even urgently) entering into that transaction.
The stock market does not generally act as a mechanism to allow investors to funnel capital to companies. The stock market, as most people think about it, is really a secondary market for fractional ownership of companies.
The process by which that fractional ownership came to be owned by other people is the process by which companies are funded (companies go public and sell shares to the public. this is the major process by which the stock market helps companies raise capital (in exchange for equity)). So all the trading that happens intraday, or even holding some stock for years and years generally does not actually serve to invest in the company.
The question of whether this is a good thing is entirely orthogonal to the point of the article, which seems to be that reading news fast gives you information about markets.
While marketmaking is indeed a valuable market service as described by hugh3, the AI in question doesn't fill that role in any meaningful way (the AI stock picker seems to be a market taker -- it actually removes liquidity when it sees opportunities)
A defensible argument for algorithms like this is that they help maintain market efficiency/ensuring that risk in the marketplace is accurately priced (if such a thing is possible). The AI basically is incorporating news that the world knows and acting on it. If the AI didn't do it, then the markets would, but slower. The fact that its making money on 20 second timescales just means that its getting news 20 seconds faster than the rest of the market, which isn't particularly shocking because you might imagine that the rest of the world is responding without the benefit of an AI that reads/interprets articles in bulk.
Doesn't this completely invalidate the results of this test?
Compare with: I've developed a system for testing whether or not any given number is prime. Using data from the numbers 5, 7, 11, and 17, ...
Actually my first thought was: In the other consecutive weeks, the system lost its shirt and jumped out the window.... twice.
This is silly. It's quite easy to generate an algorithm that would perform similarly. I wonder how it would have performed had it been run across the entirety of 2005 (or any year for that matter), and not just during cherry picked weeks.
Also, if I came up with a method like this, and I actually had evidence that it worked, I'd be patenting it and selling it to an investment bank for very large sums of money before I published the details.
http://www.studyfinance.com/jfsd/pdffiles/v8n1/liang.pdf
Apparently the random stocks outperform the "pros" picks after 6 months (the pros have the edge in the short term)
Actual quote from the study: "Results suggest that the pros selection statistically outperforms the random selection only in the one-week period. Over a six-month holding period, the random stocks perform better than the pros recommendations."
I think you mean darning.
import random
stocks = (ibm, apple, ...)
stock_to_buy = random.choice(stocks)
(My theory, as someone involved in the industry, is that emotion dictates 90% of whether or not someone is going to make a trade. This is why nobody noticed how bad the subprime bonds / CDOs were -- they bought them, so they must be good. I feel the same way about my telephone and keyboard ;)
academics though, typically don't care as much about simulating for transaction costs because they're more interested in perceived mispricing in the marketplace as an interesting economic/intellectual phenomenae. Frequently academic papers will assume a 1-price world where the 1-price is midmarket and build the model off that.
If the market were indeed entirely efficient, then the described trade wouldn't be better than chance even with no transaction costs or bid-ask.
Sorry to say, but I'm completely unsurprised by this - statistical analysis is the most sensible approach to the problem. Gamblers tend to be less sensible and calling a gambler a stock market trading pro doesn't stop him being a gambler...
I'd also be wary of labeling pros in a system which will naturally produce "lucky" and "unlucky" individuals regardless of skill... at least some proportion of successful traders are just lucky. I can't measure it, but it seems very heavily implied...
> (In his work, 'verb' is a technical term, and does not exactly correspond with the conventional definition of the word.)