The Quants Run Wall Street Now
wsj.com
wsj.com
Mirror, mirror, on the Wall [Street,] who's the [quantiest] of them all?
I sense many money managers can't differentiate themselves with current ops and now want "quant headcount" as a differentiator in pitch decks.
I bet we will start to see title inflation as a result as well- that is to say, it is not that more highly skilled quants will be hired, but rather run-of-the-mill employees will be branded and marketed as "quants"
I've noticed this drift everywhere:
-Sales people are now "product evangelists" or "product specialists"
-Operations employees who do accounts payable or receivable are now "Finance Analysts" (seriously)
-HR are now "Human Capital Analysts"
-And virtually everyone is now "Senior" or "Lead" or even "Manager" (of some process, not of necessarily employees)
So, I guess we shouldn't be surprised when someone fresh out of college who more or less guesses on stock picks with the firm's money and some fancy charts are now hailed as "HIGHLY SKILLED QUANTITATIVE ANALYSTS"
In the United States I'd expect this to be covered by Feist Publications, Inc., v. Rural Telephone Service Co., 499 U.S. 340 (1991).
[1] https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
[2] https://www.law.cornell.edu/copyright/cases/499_US_340.htm
https://www.unc.edu/courses/2006spring/law/357c/001/projects...
A compilation whose selection criteria is, for example, "everything traded on NASDAQ" and whose organization is "order by date" seems unlikely to me to have sufficient creativity to qualify.
1. Almost all buyers are institutional funds or individuals with the means to trade as well-informed investors. To put it succinctly, they're in it as serious business, because it's incredibly expensive. They have no incentive to make the edge they just purchased for five - six figures public.
2. These vendors go to various lengths to protect the data, including steganographic "trap streets" to identify the account the data belongs to.
3. There are significant legal liabilities inherent to releasing the data without a redistribution license. You agree to these explicitly by buying the data.
My layman's understanding is the more actors that participate on the exchanges, the more volume, the more the exchanges profit. If this is true, then making the historical trading data would encourage more entrants onto the exchanges, and the exchanges would earn far more than selling the data to a few large institutions.
I recall an example given in a class I took. (I may be misremembering the details, though).
Some people were trying to apply machine learning to currency trading. They had a bunch of data. They normalized the data (a common step in machine learning), and divided it into training and test and validation sets, and trained their model. Everything looked great, and they were getting excellent results on the test set.
When they went live with real money instead of making the nice profit predicted, they lost a lot of money.
Their mistake? They normalized the whole data set up front. What they should have done is split it into the training, test, and validation sets, and normalized each of those individually. Normalizing before splitting compromises the independence of the sets, biasing the learning.
(I must admit I never did quite understand this. Normalizing is optional. As far as I understand if one does an arbitrary transformation on one's data as a whole that should not actually make learning go bad, at least as long as the same transformation is applied to all the input when you go live. So when they did a normalizing step on the whole data set, why wasn't that just like doing any other arbitrary transform? There is serious dark magic here...)
You cannot normalize the entire dataset at once; the normalization contains some parity effectively allowing the algorithm to cheat and see the future... but in real life we can't see the future.
The simple example that eliminates some of the "black magic" is that normalizing against the entire set lets the algorithm know what the highest and lowest points across the entire data set are - and knowing the lowest and highest lets the algorithm buy low/sell high for all of the known data.
In real life, that's sort of like having access to the answers when you're given the questions. Being trained in that environment won't do much good.
I saw that the CBOE also offers 3:45pm data before the spreads widen up (sometimes drastically) when positions get unwound.
In case anyone is wondering where they should actually get data to be competitive, I can personally speak for these two:
* Nanex NxCore
* CBOE Livevol
I haven't purchased data from here, but I have heard it well recommended from people I trust:
* TickData
* QuantQuote
There are likely others, but bear in mind that the higher the data quality, the higher the price. For most strategies you'll likely want vetted intraday data, preferably at minute resolution or lower. Tick data is better, but it's going to be huge (my drives are near 100TB). You can reformat it into custom bar sizes if you have data on the actual trades/quotes.
If you insist on data at this price point, why not just scrape Yahoo Finance? There are literally GitHub projects that obviate this for you.
What I am trying to demonstrate here is that your strategy can perform significantly differently if it is using data on different time resolutions. If you find an inefficiency that can be traded by analyzing snapshot data at 1 minute bars, that might turn out to be nothing at all when you look at the tick data. If your strategy is profitable but you don't have real insight into the trades versus just the quotes, you might have unrealistic expectations for fills, or hold times, etc...and if your strategy is profitable, your data is the first way in which you can try to prove it's not correlated with the broader market's regime.
I'm not interested in your actual strategy, I just want to understand what you're doing categorically to see if there's any way it can possibly work instead of being attributable to a bull market. You should be able to talk about the type of strategy this is without discussing the actual signal(s) key to the strategy. This is just becoming kafkaesque for me. I'm legitimately shocked no one else has asked you about this just out of basic curiosity about how you arrived at your current methodology. You mentioned Flash Boys and tick data...but HFT is not the only paradigm for which you'd use tick data.
I'm trying to give you the benefit of the doubt here but from my experience none of what you're doing makes sense. You have to understand that the reason I'm giving you so much flak for this is because you're repeatedly recommending people buy financial data from ebay, you're being cagey about why your recommendation is sound, and your recommendation is utterly alien to the way in which people professionally trade using historical data.
If your data from ebay is actually, somehow reliable, then fine; but if it isn't, recommending it for people who are looking for actionable data is irresponsible.
I mean, I'm not sure I accept the premise that there's enough information there to find significant alpha - but if you assume that he's using a different data source for that, he can test the trading performance on close prices alone.
I know there are certain types of smart whizzes who see working for Google/Facebook/Amazon as some sort of intellectual step down, or where in finance the value to society is some handwavy "market-making" argument (depending on how philosophical you get), but presumably these types going into this are also tenure-track, research producing scientists. You could spend your time disproving the Higgs at Cern, or you could optimize pennies (albeit in ever more challenging ways).
Have these Renaissance/TwoSigma type firms really ever delivered social value that isn't a new trading model with a limited shelf life? Is there new tech? Network, hardware, software innovation that spills over into the real world?
It's hard to make the same kind of argument about adtech.
Also when you develop something like physics, once proved - it always works with very good precision(reason we can launch satellites accurately). Or let's say a software or database that is supposed to work a certain way. What's being done at quant funds - I can't think of something on those lines that could be useful. Lot of these things don't always work because you're still speculating with large inaccuracy but a slight edge statistically.
Some interesting links:
http://fortune.com/2016/05/11/warren-buffett-hedge-fund-bet/
http://www.marketwatch.com/story/why-99-of-trading-is-pointl...
All I see Wall St and Silicon Valley doing is using new found speed and scale for their own benefit while concocting self deluding narratives about how this benefits everyone. Nothing extraordinary about that.
His point is that, contrary to the narrative of the "little guy" getting screwed by Wall St, they are actually helped by the liquidity HFT market makers contribute.
Everyone is unsophisticated at something. It's not an insult.
I def. don't hold Facebook etc in a much higher regard here, but if we're talking about brain drain from socially useful fields, I don't think anyone can credibly argue that finance (esp. HFT et al) provides any meaningful social value.
In fact I'll go one further: by sucking up the best and brightest, high finance is doing a net harm to more socially useful sectors.
I'm not being prescriptive wrt what choices people should make, btw, just speaking in terms of social good.
How's about: an act that is socially valuable has benefits, immediate or otherwise, realized by parties that aren't directly involved in the transaction.
Eg, a blackjack dealer's labor probably only matters to her players and her boss, but a cancer researcher impacts well beyond that. Both can be lucrative, but one is clearly more socially valuable.
HFT firms are make money by taking a spread, which is effectively the cost of making a single trade. As they compete and get better and better at there jobs, this spread well be reduced, the amount of money available to pay for the best and brightest programmers will come down, and they'll suck fewer of them away from other parts of the economy.
The point being, the further they go past this 'point of diminishing returns', the less money they can possibly make. To the extent that HFT is a zero sum game, there will always be a limit on how much money they can spend.
(I believe but cannot prove that this limiting effect has already started, HFT firms are consolidating and making thinner margins than they used to)
Nearly all hedge funds still take 2/20 for example, and their returns have been questionable for a decade, so idk we have reason to have faith in high finance's ability to self correct.
The result can be altruistic, I guess, but my point is that judging intentions with regards to open source on the part of large companies is a fool's errand in my opinion.
You can read more about it here, too: http://beakernotebook.com
Jane Street is also pretty awesome - they open sourced their in-house standard library for OCaml, called Base: https://github.com/janestreet/base (they use a lot of OCaml).
Speaking as a physicist who left the field, the vast majority of research physicists are not going to make a significant impact on advancing human knowledge. They're all too busy teaching bored undergrads, competing for underfunded grants, and politicking to get their name onto more papers.
I'd rather do math and make lots of money. Wouldn't you?
> the vast majority of research physicists are....all too busy teaching bored undergrads,
?
Most of them view it as a mercenary job, nothing more. They don't give a shit about finance itself, but they enjoy the money, they enjoy solving problems - especially ones that are seen as impossible - and obviously they also enjoy the deep technical aspects of the job as well. It scratches a lot of different itches and it's a very high prestige gig ("Wall Street", "Wolf of Wall Street", etc.) to boot.
Look at it another way: if they're out in the Valley, they're one of thousands of talented engineers, competing against gender disparity in a very competitive social arena. In NYC, they're young, brash hotshots working less hours for more money with all of the ego, bonuses, and cocaine that goes with it, in an infinitely more exciting city. Tell me which setup you'd rather have.
Also, I know people in the situation you describe, and its a grind like every other bonus incentivized gig on wall street. People may tell you otherwise, but they're probably sugarcoating. I know a few people who burned out early and started their own companies, and a couple who are continuing the slog for huge bonus payouts. The no free lunch theorem holds true.
For me, I'll finish up this degree and see where things are headed. I'm definitely not set on SV or WS yet, but it'll probably be one or the other.
The lifestyle appeals to some, but not all, and Wall Street's "very high prestige" doesn't really exist outside NYC and its immediate neighbors. For most people, finance is associated with greed-fueled, drug-addled sociopaths who have to pay for sex and companionship, since that's what's portrayed in "Wall Street," "Wolf of Wall Street," the news, etc., and they don't associate that with prestige.
There are a lot of cautionary tales about Wall Street for a reason. Some in finance get out alive, others end up washed up or burnt out, barely maintaining employment, battling chronic depression due to years of abusing their brain chemistry. A million dollar bonus would be nice, but the other stuff I can do without.
My Dad and I had a relatively lucky streak where we doubled our money day trading Apple on margin after the September 29, 2000 dot bomb:
http://money.cnn.com/2000/09/29/markets/techwrap/
He saved quite a bit of money by being able to trade 1000 shares each time. Since the stock swung +/- 2% a couple of days a week it was pretty easy to make 2% most days, 5% on a good day. It was like a casino where the odds were 2% in your favor, with a ratchet that just sold when it was about to go down. Then you just guess the shape of the day’s heartbeat. He wouldn't even let me sell short because he felt it was unethical, so we only gained half what we could have which was agonizing to endure. I pleaded for him to get out because he was up a couple years of my wage at the time and my gut was screaming at me that something wasn't right. Then we lost all the gains the day 9/11 happened. We sold a few weeks later and ended up breaking even. Then Apple went to 500 over the next couple years with splits thrown in to boot.
I wanted to try day trading myself but they changed the law in 2001 so you had to have $25,000 to trade on margin, so only the wealthy could get twice the gains:
https://en.wikipedia.org/wiki/Pattern_day_trader
I personally don’t buy any of the standard advice about risk because it’s more risky to keep your money in a bank and only get a couple percent a year. I felt so miserable about the whole experience that I worked a bunch of dead end jobs and ran up my credit cards over the next several years. It’s heartbreaking to know how hard typical folks in the world work to make $100 a day when day trading $50,000 can easily earn $1000. But those folks don’t have $50,000 so are locked out. I guess in my heart it felt like stealing, or at the very least finding yourself in the universe where you won and thinking you somehow earned your survivor bias. So I don’t do it anymore, and I’m even hesitant to invest because politics are so volatile right now. Sorry to digress, I should have stopped at the first paragraph.
No, it's good digression, and it brings back fun and painful memories, as I made and lost money at that time.
Trading on margin is something rookies should avoid. It's the easiest way to lose money.
The thing I don't understand is this noise about shorting being unethical. You have more to lose than anybody in the game, and you're not breaking any rules.
From the article, "In the past five years, quant-focused hedge funds gained about 5.1% a year on average. The average hedge fund rose 4.3% a year in the same period."
Interestingly, what these averages do tell us is that the average hedge fund return is not much, if at all, better than the average index fund return, with the index fund return graph being much more uniform in variance.
This is how the rich get richer and the poor get poorer. The rich can trade on margin, pay less tax (capital gain tax is lower than even income tax), and have many more other opportunities. The middle class is shrinking for a reason.
This says how disconnected you are from the reality. Many Americans don't even make $100 a day. $10,000 investment plus $10,000 margin multiplied by 15% is significant for many people to buy food and pay rent.
Just today I noticed a sign posted in the window of my corner store, "Customers of [convenience] store won $1,490,823 from [state] lottery in 2016!" I wish it were required to post the total paid as well. It'd be much better for the public if the sign read, "People in your neighborhood paid $3 million to the state lottery and only won $1 million back."
Similarly, if the brokerage is going to offer margin accounts, they should post easy to understand statistics on how much customers earn and spend. A nice histogram showing returns with sliders to select a bucket for initial amounts invested and proportion bought on margin.
Enforcing free information access is another way that governments can help provide equal opportunity.
Also if you have decent credit you can simply take out an unsecured personal loan for $25k and deposit that in your trading account. This is effectively the same thing as trading on margin.
If the margin trading limit was lower then the news would be full of sob stories about how greedy brokerages let unsophisticated people get in over their heads and then took their life's savings on a margin call. Sometimes stocks go down.
I'd note that the OP lost all their gains and ended up breaking even - and that was with their father stopping them shorting.
If they had been short they could easily have been ruined.
Asking people to have a $25K buffer before they start gambling isn't entirely an unjustifiable idea.
Contrast that to humans doing the picking where there's a fair amount of "lick the finger and hold it up to see which way the wind is blowing"
So for example, managing an index weight fund is simple on the face of it but when and how you choose to rebalance your portfolio can affect your drift. This is the part that requires high math. This is even more important if you're talking a billion dollar fund as opposed to a 100,000 personal account.
Not really. Even if everyone is just throwing darts, half of the players will beat the market, and the more players you have the more extreme the outliers will be. Of course, the converse is also true, but no one pays attention to the losers. Both Vegas and Wall Street prosper from this same principle.
For how long? And if it is for more than one sample (year), then how do you explain the continued success of places like Renaissance Technologies?
> Both Vegas and Wall Street prosper from this same principle.
I think poker is probably the closest thing in Vegas that matches Wall Street. Probably why it is so popular among quants.
This is not to say that all such funds can be explained this way. But most of the successful ones likely yes
Maybe you disagree with 0.5 per day. Let's make it 0.9!
...But that's still 5.7 x 10^-30. How many firms do we need to exist for this to emerge by chance?
This and website bug bounties being sold on the black market are my two HN crusades. I have heard darts, I have heard coin flipping; I hear all manner of analogies from people who stubbornly insist or strongly imply that consistently, purposely beating the market is infeasible in principle.
But no one ever does a modicum of analysis to support whatever trite analogy that's thrown out, they just wave their hands and exclaim, "statistics!".
So here is something actually quantifiable. If you have something quantifiable to refute my analysis of your analogy in turn, please share it by all means.
Lost money trading. If they made $1 trading that counted though I assure you that day was a loser from a business perspective. I also have not lost money trading in 3 years, simply by not trading at all.
It doesn't really detract from your broader point, which I don't have a strong opinion on but that stat is a pet peeve of mine as it's fairly meaningless.
http://www.investopedia.com/articles/active-trading/042414/w...
Source: I worked for Virtu's predecessor Madison Tyler for 4 years and for Virtu after it merged with Madison Tyler for ~8 months and have first hand experience with HFT.
I think he's just saying that particular metric is missing context: a lot of crappy businesses could claim never to have lost a dollar trading, while still as businesses losing tens or hundreds of millions of dollars chasing, and not finding, profits.
I don't think he's arguing specifically that Virtu lost money.
I'm only responding to you, because you're my fav HN personality and tend to have a lot of interesting things to say at ChiSec :)
I just am annoyed when people site the former when talking about Virtu, both using it pejoratively or in praise of the firm, because I don't think its a terribly useful thing to say. We can judge Virtu the same way we judge every other firm on the planet, by actual profits, there is no need to come up with a new metric for them.
Again, I actually don't have strong opinions about coin flipping and trading being equivalent (and I think Virtu is likely a bad place to have that conversation vs a hedge fund because its not really an investment firm its an execution one) but I think claiming trading PnL refutes it is wrong.
these are fairly long term problems so stochastic calculus not too useful here. portfolio risk is a very classic stats field.
it's not really clear that the math is "high" from the above comment but portfolio management, even the really boring sounding ones like index replication, uses the entire gamut of "high mathematics": robust and Bayesian statistics, non-parametric statistics, extreme value statistics, signal processing, numerical optimization, etc.
I do wish the web link was on the index page instead of the comments page so it was faster/easier/more obvious to use it.
We could use it as a prompt. Maybe the comments could be a zone to talk about quants and wall street. That might be useful, but it'd have to be explicit.
EDIT: I've actually wished for prompts on HN, for what it's worth. Other sites typically have ways of starting conversations without articles, and Ask HN's usually die except for a few lucky ones. Usually you end up in a situation where you want to talk about X but have to wait until a tangentially related comment is posted, and then sort of shoehorn it in awkwardly. Being able to spin up a "Quant thread" would be nice without a specific topic or question.
As for using the headline as a prompt for discussion, I'm quite sure that's all that drives many commenters on HN :)
1) my friends and I write to each other every day with more nuanced, researched, investigative journalism/analysis than the WSJ could ever hope to put out. I frequently release anonymous pastebins of my (hopefully impartial, unlike WSJ) analysis for free as a public service and I hope someone promotes them
2) the amount you quoted, $400, is somewhere around 1-2 months worth of total income for easily a billion people in this world. $400 would be considered an excellent monthly salary in a variety of countries, such as Thailand, Vietnam, Bosnia, Serbia, Albania, Ukraine.
So for likely the majority of the world, $400 would be almost unjustifiably high for words on a page that don't really offer any profitable or immediately actionable ideas
You're pitting your intelligence against the intelligence of other quants (not against the complexity of some natural system or something). You have 3 phds? they bring in 4. You bring in 6? they bring in 10. You bring the processing down to 50ms, they go 45ms; you go 5ms, they go 2ms. The game never ends.
It's even more depressing when you realize that, even if you win, you're essentially propping up, what I like to call, a "fluff market", instead of adding-value the way Silicon Valley and the manufacturing industry does.
curl -o 1.htm https://www.wsj.com/amp/articles/the-quants-run-wall-street-now-1495389108
sed -n '/./{/<title/,/<\/title/p;/<p>/,/<\/p>/p;}' 1.htm > 2.htm
Add links to non-essential images (cf. auto-loaded by browser). With available captions.
sed -n '
/./{/div class=.image/,/<\/div/!d;s/ *//;}
/src=/{s///;s/\"//g;s/.*/<a Href=&>&<\/a><br>/;}
/alt=/{s///;s/[\">]//g;/./s/.*/<P>above: &<\/p>/;}
/Href=/p;/<P>/p' 1.htm >> 2.htmThe article had some pictures and graphs (see archive.li someone else posted). They were nice but they weren't essential.
amp html pages look great in links.
The first line uses curl to download the AMP file to 1.htm
The second line use sed to replace some elements in the HTML and writes it out to 2.htm
I added some bare-minimum CSS to make it a little nicer to read. Full command (with in-place sed):
curl -o article.htm https://www.wsj.com/amp/articles/the-quants-run-wall-street-now-1495389108
sed -n '/./{/<title/,/<\/title/p;/<p>/,/<\/p>/p;}' -i article.htm
echo "<style>html { text-align: center; padding: 36px; } body { max-width: 600px; text-align: left; margin: auto; }</style>" >> article.htmI think this would cut down on most of the meta-discussion that happens every time a WSJ or FT link gets posted.