Hedge funds use satellite images to beat Wall Street
newsroom.haas.berkeley.edu
newsroom.haas.berkeley.edu
We should not aspire to drag hedge-fund research down to the Main Street level. If there is a place for government in this context, it is to fund the education necessary to help retail investors learn a) how to take beneficial approaches to investing and risk, b) to appreciate the realities of trading against professional investors, and c) to understand how to conceive, test, and implement their own hypotheses in the real world.
In general, an informed market is a healthy one.
What about companies that find out about your retail trade before it's even executed, and are able to act on that information to profit from your very intent to trade?
Front-running is very common in bond trading.
They are market-making, which is something else entirely than front-running.
It's often claimed that HFT help make markets more liquid and efficient, "which helps everyone".
It's much less than clear whether the presence of HFT helps make an entire market more efficient, or whether just the HFTs are able to benefit.
Are you sure it's illegal if it's a hedge fund with access to order flow from a third party?
"Front running is one of the easiest ways to make money. It's essentially insider trading, except the inside information isn't about corporate activity; the information is about client order flow. In this case, since the index investors are not their clients, it is legal for hedge funds and any independent traders to front run them. One could argue that the hedge fund managers are doing nothing wrong; it's the investors' fault for acting irresponsibly. The problem with that argument is that many of these investors don't have a clue about what is happening to them." [0]
http://www.dark-bid.com/hedge-funds-front-running-investors....
Now about the Russel indexes - the article has a point there. They are notoriously shitty indexes. Nobody should be putting money into those. But still - nothing illegal(nor should it be). This is yes, clearly a case of "investors' fault for acting irresponsibly". Nobody is forcing them to use these indexes and this information isn't hidden anywhere. Last thing we need is 100 more regulations.
HF traders provide liquidity and market-making when it suits them. Doesn't that stop when it doesn't?
edit: found the article quoted last time I discussed this.
https://blogs.cfainstitute.org/investor/2013/04/24/what-to-d...
Larry Harris, Professor of Finance as USC;LA says
"A few high-frequency traders front-run buy-side traders who are working orders, thereby making the latter’s trades more expensive. Such activities are legal if the high-frequency traders do not improperly obtain information about the orders they front-run."
It's only profitable if you dont get caught. SEC will make a very expensive and public example if they catch you.
Source: +15 years in the industry, a number of which have been spent working in/with compliance.
Firms don't buy order flow from Robinhood in order to trade in front of those orders, they buy order flow from Robinhood to execute those orders for those customers which is a service that those customers pay for as purchasers of liquidity.
Retail orders are small and generally aren't capable of moving prices. Your order for 100 shares of whatever isn't going to move the price so you can't really make money ahead of it.
You should stop trading now.
This is better of course, but nothing you can't do in some other way with a little work.
"Some of Mr. Lynch's best picks came from watching his three teen-age daughters shop at a nearby mall,..."
https://www.baltimoresun.com/news/bs-xpm-1993-03-09-19930680...
All the satellites do is automate the work and I'm pretty sure hedge funds have been using satellite pics for well over a decade now. (I recall reading about the parking lot use case pre-2010) So really all that's changed is that they're now using ML to automate most of the work that analysts used to do manually.
I worked at one of the companies doing machine learning on top of large quantities of satellite imagery and selling the results.
It's more than a little work; it's pretty challenging, in a variety of ways. Just the training aspect is surprisingly difficult to get right.
Remember the Matt Levine test: insider trading is about theft, not fairness.
When a company insider uses private company information to trade (or colludes with an outside party to do so) they are stealing material non-public information from the company for their own benefit. Nothing of the kind is happening when a hedge fund (or anyone else) acquires and analyzes satellite photos.
There is no expectation of fairness in the stock market. All sorts of people work very hard and spend a lot of money to develop proprietary information in the pursuit of stronger returns. There is nothing at all wrong wit this, especially because it means that in the end we all get more accurately priced securities.
Retail has no business in the securities market. thats where it ends up. stop playing. retail investors flocking to the stock market is a 40 year old meme that is built upon non-objective thought and gullibility. It is built on laziness. Retail doesn't meme commodities because there is almost no lazy investing approach to it. But the reality is that there are other capital markets. Retail isn't helping the stock market, they aren't helping the companies they like by buying stocks they "recognize", they are MAYBE lowering borrowing costs for the company, its executives and employees, but retail's trading has nothing to do with anything.
It's built on desperation.
There were some solid ways to develop retirement strategies with lifelong careers, pensions, and other older retirement vehicles. However, pensions are largely irrelevant outside the public sector now, the older retirement vehicles have stagnated/become poor investments (like annuities, which have become a landmine for terrible fees), and what was left was 401ks and individual retail stocks.
Mutual funds are a similar landmine, where transparency is lost and they're often built for the tax particularities of retirement accounts.
That leaves retail and... ETFs, which we can only wait for someone to find an angle to extract more value at the expense of small investors' returns.
It is pretty amazing and ingenious.
No, it doesn't undermine the observation that retail shouldn't be playing.
that's just not true and is a gross over simplification. for one, not beating the market is NOOOOOT the same as losing money. For another, even if they do not beat the market, there are a lot of other benefits.
People gain an understanding of how things like bonds and equities work. I cant tell you how many people (masters degree included) think you can loose more money than you invest by buying a stock. I seen someone literally not participate in 401k matching because of that thought.
there is something to be said about having greater control over one's money. When someone says lets eat out at this expensive 150$ plate restaurant, I think gee I can buy X shares of Y stock instead.
I personally have already maxed out my 401k, and if I want to retire I need to contribute more. currently I am beating the market, and I have little care if 'the average investor' isn't. it gives me something to do, empowers me, and is not 'at best' gambling.
I personally invest primarily in well diversified, extremely low cost ETFs and I buy 1 - 2 positions a year outside of that portfolio (right now the main one is bitcoin for me). But, I'm always looking around for something that I think can be a 10X return.
But... I rely on my primary diversified portfolio for my retirement and the single 1 or 2 investments I make outside of that are for fun, knowledge and I fully acknowledge gambling.
Nothing wrong with being investment savvy and all that jazz but obsessing over it doesn't sound healthy.
People can enjoy a fancy plate or 2.
Personally I find the above work boring - it ends up being a full time job, so once I subtract my salary I'm not really better off financially than an index fund and I'd hate my job.
As for why don't the big guys do this: it isn't worth their time to chase a small gain (say $20,000) so there are many things the small guy can do that are more than wroth it that is beneath the notice of the big guy. They know about these things, they need to make more money to pay for their overhead.
I'm not even sure there is a valid distinction between the two groups. The retirees are shareholders.
It is quite literally dumb and fucked up to tie comfortable retirement to things like releasing carbon into the atmosphere, or defense industry profits, or Wal-Mart's quarterly profits, or to the stability of Jeff Bezos' marriage.
But couldn’t you argue in exactly the same way for disenfranchisement? Do you think retired people should be allowed to vote?
I believe that everyone has a human right to vote. However, retired people have generally already established themselves; therefore, the changes that government makes, particularly on social issues, has minimal effect on them. Therefore, I think its unfair for the generation who is still establishing themselves not to be held hostage to the previous generation's vision of society.
I think you have one more negation than you intended in that sentence.
The upshot is that for a year in which market indexes are up 9.1% and the smartest investors do much better, Pat Q. Public can still earn maybe 8% as a retail investor in single stocks if she/he doesn't churn it all away in overly rapid trading. That's not a bad outcome.
Stragglers will stay in the game for a long time in such a scenario.
Alpha (the skill/edge component) is the excess return on a risk-adjusted basis.
Roughly, alpha is total-return - risk-free-rate - porfolio-beta * (market-return - risk-free-rate).
You can have a total return of 20%, a risk free rate of 3%, a Beta of 2, and a market return of 13% and you've actually underperformed the market on a risk-adjusted basis. Your alpha would be 20% - 3% - 2 * (13% - 3%) = -3%
The point of the stock market is not to let small or large investors become rich. It's to provide funding to companies.
If there is a distribution of skill in investors, using mutual funds might prevent you from being in the lower percentiles of under-performing investors. However if you really have a strategy that involves taking the market price that manages to consistently under-perform, then you could make money off of that strategy too.
If they're not the smarter analyst then yeah they're going to lose i.e. not outperform the market. Of course that doesn't mean that they'll lose money.
I must be dumb, but I don't understand this statement... How is that a "theft"? You knowing that I eat 2 eggs in the morning when I don't want people to know is not you "stealing" information from me.
From an outsider perspective like me (I don't own stocks except through my 401k), Wall Street seems like a game of information asymmetry anyway where the only winners have an information edge at some moment in time. Now the question is about how "fairly" was the information obtained which seems pretty blurry to define to me.
Suppose you are giving a prize for someone who guesses how many eggs you eat in the morning. If I watch you checkout at the grocery store and based on the number of eggs you buy, guess that you eat 2 eggs a day, and win the prize, that is not theft.
If however your chef who prepares you breakfast every morning “guesses” you eat 2 eggs, then that would be problematic.
If the chef working in your restaurant sells a cake he made there and puts the money in his pocket instead of yours, that's theft. It's legal to sell cakes the chef made, it's just the the resulting value belongs to the employer and not to the chef personally.
If the chef working in your restaurant sells some of the cooking ingredients and puts the money in his pocket instead of yours, that's theft. It's legal to sell spare cooking ingredients from a kitchen, it's just that the resulting value belongs to the employer.
And in the exact same manner, if the chef working in your restaurant sells the customer order statistics (e.g. how many eggs were eaten) and puts the money in his pocket instead of yours, that's theft. It's legal to sell customer order stats, it's just that as far as those stats have value, that value belongs to the employer and not to the chef personally.
When I take information entrusted to me by my company and use that information to profit in the market, I have taken information, used it in a manner not authorized, and profited.
That's different from someone else unaffiliated with the company independently deriving the information, using it in the market, and profiting.
As an example, the researchers who discovered VW's cheating on diesel emissions tests (assuming they were independent) would be perfectly within their rights to short or buy puts on Volkswagen. VW executives who were responding to the EPA inquiry or otherwise discovered the cheating would not be within their rights. In both cases, there was a moment where that information was material to the valuation of VW and not public, yet one group could legally trade on it and another could not.
Correct, you don't necessarily have to "steal" that information to find it out.
If you hire me to cook to you breakfast with the understanding that your breakfast habits are confidential and I give that information away, then that is "stealing" information because I have a duty to keep that information private.
If I spy on you while you are eating and figure it out and give the information away, then I did not "steal" the information because I don't have a duty to keep that information secret.
While appreciating the usefulness of this quote, one should also understand that usefulness is limited by the actual enforcement of insider trading laws and insider trading doctrine generally, which does not square with the analogy. There is certainly an element of "fairness" that is being protected implicitly and/or explicitly in enforcement actions, with the purported interest of keeping the markets attractive to retail investors. On the other hand, "fairness" is undermined in cases like Enron where prohibitions on insider trading likely artificially reduced downward pressure on the price of the stock (costing investors who bought later larger losses) by disallowing folks who understood what was going on from selling the stock.
I believe the counter argument is only people with enough wealth to afford an account at one of these hedge funds gains an immediate advantage here. So it's yet another way "the rich get richer."
Doesn't bother me, but I can see why other people would have a problem with it. And I severely doubt more than 5% of the planet cares about whether "in the end we all get more accurately priced securities."
Accurately priced securities are like a responsive public health system or redundancy in the electricity grid. Almost nobody cares about these things directly.
Yet virtually everybody enjoys the enormous gains in material living standards that are facilitated by having these things in place.
Better data yields more efficient markets which is what we actually want.
“Technology was supposed to level the playing field, but what I see is the fence separating sophisticated and unsophisticated investors growing higher,” says Patatoukas
As if that's a bad thing or a surprise to anyone...
By being correct about a stock in a trade you end up being rewarded for decreasing the cost of capital of companies that do the most with it.
And just a minor quibble, a more accurate stock price doesn't always lower the cost to a particular company of raising money. It makes the market more efficient over all, for sure, but often companies would prefer a less accurate (ie, higher) stock price for the purposes of raising money.
Needless to say, I'm not convinced that this kind of volume is necessary to get most of the benefit of the stock market as a price discovery mechanism. I'm not saying anything needs to change, I'm just not sure it's such a huge public good.
I wonder if the author is thinking of groups that suspected the 2008 crash but either didn't raise the alarm and made bets on the outcome or weren't heard in the commotion. Any proprietary information they might have had could have alerted the public or something but there's always noise or incorrect predictions about the market and hindsight is 20/20. If their worldview is that private groups controlling information for private gain is bad, well that's already going on so this is just an extension of that.
I'm sure most funds have groups looking into data collection on top of modeling market results based on various inputs the data suggests. I'm sure most funds have some amount of proprietary data or algorithms to run against it, even if the info overlaps or they sourced from the same information like publicly available government released satellite photos. I'm sure that the market (mostly computer programatically trading) reacts fairly quickly to big players making less than subtle moves. There may be some method to deciding how far to go with certain information to not tip your hand or reveal information by inference. It's how the whole thing works already, and dropping into a mutual fund seems the best way to take advantage with minimal risk.
There is no market regulatory philosophy that says "thou shalt not profit from working harder or smarter." Rather, working harder/smarter is encouraged, because when information is discovered, the new owner of that information acts on it in such a way that it creates price pressure in the market to push that security to be more accurately priced.
This is an unalloyed good. Yes, the people who get the information may profit from it, in a similar way any capitalist profits by reacting to price signals. There is no guarantee that all information should be available to everyone. If your information gathering is better and more determined, and your analysis superior, you may profit more than those who do not put in the effort.
One can get API access just like that? If so then I'm interested, might be fun to dabble with--can you name some of these providers?
Also, there are armies of vendors selling all kinds of data to hedge funds, the satellite data providers are just good at getting splashy articles written.
I am genuinely curious how that benefits me (or the average joe).
the article is exactly right to present the professors’ opinion as-is.
I think it's incumbent on good journalists to put the views of the people they interview in the proper context. If the subject of an article says something wrong, and the article fails to point that out, it is a lie of omission.
And given how difficult it is to define insider trading that seems like a pretty accurate assessment to me. Alternative data firms are the new expert network firms in the wake of Rajaratnam.
You are making the same mistake the article does when you state this line is blurring.
I didn’t realize it was so cut and dry. How silly of me! I guess all of the work I’ve done providing legal advice to hedge funds on what is and isn’t insider trading since Preet Bahara first came to town was for nothing. Thanks for clearing that up.
I fail to see the parallel with Rajaratnam, since Rajaratnam was accused and convicted of insider trading on the basis of leaking from corporate insider executives(!) That's precisely the distinction that is being drawn in the comments here, between theft of insider information and the acquisition of difficult-to-intuit public information.
To answer your question, these parts:
> But technology is increasingly blurring the boundaries between public and private information, creating data opportunities that are legal, but are expensive and often require special expertise to access. “Technology was supposed to level the playing field, but what I see is the fence separating sophisticated and unsophisticated investors growing higher,” says Patatoukas, who is passionate about teaching his students to analyze public sources of financial information and finds the trend troubling. “That’s the dark side of big data. Our evidence suggests that unequal access to alternative data leaves individual investors outside the information loop.”
> In the aftermath of the financial crisis, there has been increased regulatory interest in the role of informed trading and disclosure requirements to protect the fairness and integrity of capital markets. With this in mind, Patatoukas hopes that the paper will get the attention of the regulators. “In a market setting where the line separating public from material non-public information is getting blurrier, the question that regulators need to answer is: What is their role in terms of leveling the playing field for individual investors?”
...until people are forced to play the game by external conditions, e.g. being employed and therefore having a pension that is based on funds.
Hedge funds also fly cesnas over refinery fields to look at the levels of those big tanks you see off the highway... most of them have roofs that move up or down with capacity, and those that don't you look at with an infrared camera to look at relative temperature differences.
1. They are just trying to ride the ML hype train and they sell effectively polished turds to board member buddies. This is the vast majority of the alt data and analytics firms. They have flashy dashboards and allow you to drag&drop together models and data. The whole thing looks very fancy during the sales pitch. It's all animated and shows charts that impress CEOs. Then when your quants get hold of the data they discover that it's real world predictive power is zero.
2. They actually discover valuable data or build models that genuinely produce an edge on the market. These either turn into hedge funds or are bought by an existing hedge fund to run the model in house. You rarely hear of these because their offering is public only briefly or not at all.
https://www.economist.com/finance-and-economics/2019/02/28/n...
Remote sensing like satellite is excellent for other similar types financial modeling applications, but I wouldn't use it for modeling retail as it is much too coarse.
This is GOOD for the market(the practice, not the article). Price manipulation is a real thing, and fact-based research such as this means prices are more likely to reflect reality. This type of analysis is what we need more of! It's not opinion-based drivel from a talk show or blog, it's empirical data, analyzed impartially.
edit: typo
From a technical perspective it's actually an interesting problem because there are so many things to consider. Really makes you notice how disadvantaged retail traders are, though. I think traditionally a lot of hedge fund strategies are kept pretty secret, but this one is so out of reach and technically difficult for most people that they don't really care if it gets out.
Really, what they are likely comparing is the relative changes in parking lot occupancy over time. So it’s not necessary to have complete coverage of all parking lots. You just need enough coverage to make the prediction with a certain level of confidence.
Unless your thesis is people who park underground shop differently. That is a valid idea and you cannot answer it via this study. However you can study it by other means if you believe this is important.
Or you might be saying having alternative methods to count traffic like cameras or other ground based counters.
The game here is essentially to get noisy signals about companies earlier than the official ones provided by the company. Take satellite pictures of parking lots, do phone surveys of consumers, count trucks leaving supplier companies, whatever. If it's a good predictor, you know upcoming price moves early, and can likely make significant profits off of it.
Something I'm wondering about though: The article makes it sound like this is some new unfair advantage of the big traders. But isn't this an old game? If you can afford to acquire and process the data you're likely going to beat the other participants. That was always true. I don't see what's qualitatively different here. Is there?
A world in which nobody bothered to check whether a company was actually producing anything would turn the stock market into a collective guessing game.
This is kind of a thing in China.
Finding someone successful at stock picking and asking him what he did to be successful is like finding that 1 out of 1024 person who flipped a coin heads 10 times in a row and asking him what makes him such a great coin flipper.
The hard part is not regressing to the mean, and consistently beating the market, which sure, nobody can do. But if I learn today that a stock is going to rise tomorrow, and I buy as much as I can, the stock will rise a bit today. So the market has become a little bit more efficient.
That I happen to then base all my subsequent trades on overconfidence and tossing darts doesn't retroactively invalidate that trade. You can even make a market more efficient on losing trades- you short a stock, it goes down a bit, but you spend more maintaining the short than the stock goes down. Then, the stock crashes after your short expires. Oops! But your prediction was still partly right, and that information was integrated into the price a bit earlier than it would have been without you.
That's not too surprising. Isn't that the entire motivation?
Eamon Javers' book Broker, Trader, Lawyer, Spy covers some cases of this sort of thing. One I can easily remember is a hedge fund flew private planes around taking areal photos near power plants. Coal fire power plants store their fuel out in the open and they buy it by the train-load, so they were able to time the market based upon when power plants were going to make large coal purchases, as they do it when their piles of coal reach a certain size. I don't remember the numbers quoted in the book but they made money doing this, it was a couple percent.
It's only unfair if you think your 401k or personal trading account is/can/will grow anything like what hedge funds do. This coal market manipulating fund, they put money in, flying planes around and taking pictures isn't free, then they did analysis on the pictures which wasn't free, then they put real money in to the market to bet on it moving a certain way... which they are really only allowed to do because there is a counter-party that is willing to take that bet on. If it were really egregious, you could argue that the folks buying energy at incrementally higher rates are getting the shaft and that's not really fair but their energy provider could be smarter about when it orders up fuel. Once it goes public, the players and market tend to react to it
The alternative is a Soviet-style disconnection from reality. A market planner in a far-away office planned for a farm to produce some amount of output, so, that's how much the farm officially put out, nevermind that there was a drought and part of the output was diverted to the black market etc. But how is the planner supposed to know? Without feedback, without the truth, it becomes impossible to come up with accurate numbers for anything. Without the truth, the economy is a house of cards.
Moral ethics aside, this looks legal?!, even if the shorts tank shares based on images of car parking lots..
That's always been the argument for index funds for regular investors, you're never going to compete with hedge funds, why waste your time?
To get this data you wouldn't need to buy the cars. So it sounds like they did something that sounded like that as opposed to what you remember. Even w/o social engineering you could quite easily get some dealer to give you that type of information (or someone working there or otherwise).
Seems they were after the shipping identifier, as when we changed the shipping identifier to not leak shipment volume information, the orders soon stopped.
There's market opportunity here. Off the top of my head:
1) Drone photos instead of satellite photos to get at the data cheaper
2) ML counting of cars
3) Some way to legally sense the number of cellphones within a given radius (can possibly be combined with a drone)
4) Simply aiming a network cam at the entrance/exit from across the street and counting the number of people that enter and exit via ML
All of these could get you at public data that highly correlates with retail sales
The issue, if there is one, is that it is not cheap to get this data on a large enough scale to be useful.
I suppose it's possible a sampling of just a few parking lots would be representative enough to be useful? In which case you don't need drones, you can just go to the parking lots. Which is apparently what Sam Walton did? I dunno. If that were good enough, again I'm not sure why the hedge funds would be paying for sattelite photos instead.
>“What we found is that it’s a gain for large sophisticated investors who can afford the substantial costs of acquiring and processing big alternative data”
This is yet another example of wealth accumulating, of rich actors getting richer faster than poor ones. Without some controls on the behaviours that allow wealth to accumulate, it will further concentrate among the wealthy. I would argue that is a bad outcome.
Any game where you get more of an advantage the more points you accumulate from a fixed pool of points is going to rapidly result in an unequal share of the points going to the first person to acquire the advantage.
Monopoly was created as a game to warn of the pitfalls of capitalism (it's a simplified form of the winner takes if not all a disproportionate share scenario that is life)[1]
http://www.bbc.com/capital/story/20170728-monopoly-was-inven... [1]
Of course that I think this puts me at odds with many people, I like capitalism, I like the social pluses of it but I believe that capitalism should be sub-ordinate to society directly (more what you see in the north of europe) and less the other way around (pretty much most other places and arguably the US).
"A company called Planet Labs Inc. has launched a small constellation of what it calls “cubesats” that can deliver much more frequent imagery of economically sensitive spots than traditional satellites. Those spots include retailers’ parking lots, oil-storage tanks or farmland"
https://www.wsj.com/articles/satellites-hedge-funds-eye-in-t...
Is becoming a popular choice for Hedge Funds seeking this kind of data.
Anyone who ever considers this sort of thing to be "unfair" or "cheating" should also consider how comparatively easy it is to make a nonsense of.
Who wants to go halfs with me on a large-format printer?
This type of data is ultimately used to predict revenue for consumer companies. Cellphone data and wifi data is much better for that type of prédiction.
1. It's from hedge funds, and you can invest in a hedge fund today and reap the benefits of their strategies yourself
2. There exists a gap right now, where a startup or data analysis firm could provide this kind of information to smaller investors through some kind of subscription model.
3. Ultimately the government itself could "level the playing field" by either gathering this information through other channels and releasing it (BLS or SEC style) or by replicating the strategies and releasing it.
I think #3 could be viable. We already require businesses to share very important and actionable info with the government, and since everyone does it, the risks of one firm having data leaked and another not are mitigated. (SEC requires public reports, BLS takes private data and aggregates it for higher level views) If the government required businesses to report data to them on a weekly or monthly basis regarding things that are being satellite tracked, they could release this information on a set schedule to basically demolish the ability of institutional investors to gain secret information to bet against the public positions
I see it as just a quantified and scaled up version of, say, noticing more Tesla cars on the street, and choosing to invest. I think making something like that illegal would only be harmful to the stock market and investors.
[0] https://www.bloomberg.com/opinion/articles/2015-01-23/capita...
[1] https://news.ycombinator.com/item?id=8966817
[2] https://www.reuters.com/article/us-sec-capital-one-fin-insid...
If it's "public" it's fair game for the stock market. My favorite example was the company Quandl[1] which was monitoring charted flight patterns of companies to determine if a company in the area was going to be purchased.
[1] https://blog.quandl.com/corporate-aviation-intelligence/airc...
Especially as it related to big-box retailers.
HF compensation is different can of worms.
This is definitely a case of art imitating life though.
What are you thinking about?
Indeed, it's negative. While the big funds steal money from small investors the waste of energy and the pollution caused by launching satellites remains.