Robinhood reports outage as markets rebound
axios.com
axios.com
I have a couple grand in play money in Robinhood, because hey, zero commissions, but it’s clearly not sustainable for the long term. I’m glad my real investments are all safely parked at Vanguard, where I likely wouldn’t even notice if the brokerage seized up for a day.
The problem is that Robinhood is a young tech company and the "move fast and break things" approach doesn't work when it comes to industries that need reliability and risk management.
I'm surprised they got away unscathed from the lack of proper margin calculations before. That should've led to major fines.
"Nines don't matter if the users aren't happy", to quote Charity Majors. No-one will remember all the uptime for the rest of the month, or indeed year. Just the hours they were down for a critical trading day.
Also rumor has it they “lost user data and the backups are not working as expected”
Edit: This is the tweet. Not sure if it is legit: https://mobile.twitter.com/classrobinhood/status/12345774264...
My bet is still on it being a database issue. If they indeed are using AWS RDS, I wouldn’t be surprised if they have half of AWS support engineers working on it. Once again, if I had to bet, I say they rolled their own "half-baked" database solution instead of using a AWS native offering.
(yes: the URI says February 5th whilst the article says March 5th!)
A company like Robinhood doesn't strike me as a strong candidate for contracting AWS Enterprise support. For the price you get almost nothing out of it.
The question is really does the front-end normally try to query the next day and what's going on in their order execution system on the backend, which we don't know because this source is just speculation based on console logs in the browser.
Robinhood would be using a proper date library anyway to handle timezones. The system was fine on the 29th, 1st and even this morning pre-market. Seems more like a capacity issue that cascaded into failure.
Additionally, if you’re a conspiracy theorist — and who isn't, in this wild 2020 ride — Robinhood was down exactly four years ago today, too, lol
I didn't even know they had a phone number you could call. This should have been emailed out after the first hour of the outage. Or, how about you have a better system down page then just a blank website? If API health checks fail, serve a static site from S3 (route53 can do this). The ops failure of a company that can afford to pay quality DevOps engineers $200-300k a year is beyond belief.
Simple as:
"We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH"
"You're call will be taken in the order it was rexeived. Current wait time: 1,732 minutes."
Why we are building highly sensitive monetary performance based applications under the guise of agile and electron is beyond me.
The app market needs a market correction in design. IEEE, military, aerospace, or aviation design standardizing comes to mind.
Not even mentioning the other hilarious bugs this company has let out into the wild. They don't seem to have a good grasp on what they're doing, and now many other, more mature brokerages offer zero fee trades. Their time is coming.
It's cheaper.
This is not complicated. The entire society is orientated to getting the most for least. Excessive effort in quality is a waste.
Besides, the "serious" people hardly fared better. The notorious TSB failure, for example: https://www.independent.co.uk/news/business/news/tsb-it-fail...
On a month-to-month basis, I can't even seem to crack 99% reliability. (Geez, I should really call them on it, or perhaps get a new modem...)
Point is, how high is Comcast's valuation again exactly?
RH isn't an ISP, and being down for an entire day of trading is a big deal. If I don't like my ISP... welp, I'm pretty much screwed. If I don't like RH I have plenty of options.
Doesn't sound like the smartest thing to do.
I don't check RH every day and I wouldn't have been compelled to check on it at all yesterday except for the news (I'm losing money, who wants to look at that! LOL)
I don't actively use RH as my main brokerage so not a big deal to me, but obviously this is unacceptable. Don't recall any other online brokerage suffering an outage like this.
Suffice to say this throws cold water on any mystic around Robinhood having better tech that any incumbent.
This is very poor communication, sending an outage mail after hours it had occurred, with no ETA for the fix , and absolutely no details on what has happened i.e a hack , or a DDOS, or an infra outage etc.
Brokerage firms have outages with surprising regularity. I remember Sharebuilder being particularly bad, but Vanguard was down for a bit last week. A whole day though is pretty spectacular.
If you are investing, how sexy the app is (if you only use a phone) shouldn't matter. If you are day trading from a phone app (except in emergency), you are, IMHO, delusional.
If you are just doing it to gamble (for fun), a sexy app that lets you gamble easier on the fly better can well be worth it.
I imagine the rules are similar - you must be able to have your clients trade during normal market hours. The regulators understand there can be unexpected events that put systems down unexpectedly (high volume, bug that crashes the system, etc.) and are generally pretty understanding. But I think something like this where they are down the whole day (and STILL down from the looks of their status page, though they claim to have identified the issue), could easily earn them a fine from the regulators for not meeting the license/charter requirements.
> Hey @RobinhoodApp did someone forget to add isleap() to your python module? Imagine losing thousands of dollars because someone didn’t code for the leap year. It’s looking for 3/03 data and throwing errors out the ass. https://pbs.twimg.com/media/ESIuB4nXYAAYImF?format=jpg
Now if I had a trade scheduled ahead of time to execute during the down time, that seems like a more justifiable case for compensation.
[1] https://seekingalpha.com/article/4212397-robinhood-high-freq...
Virtually all brokers are rebated for routing orders to HFTs. HFTs dominate retail order execution. HFTs pay brokerages for access to their order flow, because that order flow is uninformed and safe to quote narrow spreads for.
Like other brokerages, Robinhood makes money on net interest from cash holdings and stock lending fees (it helps that Robinhood encourages people to buy meme stocks with a lot of short interest). Unlike other brokerages, Robinhood also encourages options trading, which are more lucrative for market makers.
You sort of are the product at Robinhood, but it's not really because Robinhood is doing bog standard order routing that every other broker does too.
> Headlines like Robinhood Gets Almost Half Its Revenue in Controversial Bargain With High-Speed Traders are virtually meaningless, as the above discussion of other monetization options should make clear. At scale, once their userbase actually has material cash in their accounts on any given day, they’ll likely make net interest, but pre-scale they may not even have the treasury function to do the work required to earn it safely. It’s a commentary on how low their revenue is (growth-stage startup optimizing for reach over revenue, film at 11), not a commentary on how beholden they are to Wall Street.
https://www.kalzumeus.com/2019/6/26/how-brokerages-make-mone...
https://www.reddit.com/r/wallstreetbets/
Folks were leveraging up all last week and buying puts, thinking the market was tanking to zero and we're all going to die of "beer virus" as they call it -- and are getting a harsh lesson about timing the market today.
They’ve been giving so much leverage to broke young people, I wouldn’t even be surprised to read that they’re insolvent someday after a big, unexpected move.
The folks that post there see their gambling for what it is.
I've read some sad stories about, for example, a student betting student loan money in the market and losing it instantly, lured to make risky bets from the other stories of dumb luck success.
Here's someone I don't think is in on the joke: https://i.redd.it/at7gloeb5ck41.jpg ... oops
Posts like that are par for the course. Its called “Loss Porn”. The subreddit is absolutely geared towards stock/options trading as gambling and if you think otherwise then you’re wrong. They are very self-aware and do it for fun. And yes some may lose a lot of money but they know exactly what they are getting themselves involved in.
Loss and gain porn are a normal part of the culture as are the slurs used.
The gains and losses are often real but this is the community you go to the laugh about such things
I remember being in college in 2005 and seeing people going wild with Forex and penny stocks. WSB is just a small subset of traders who like to discuss it on social media.
Edit: I stand corrected, see below.
They could easily have 1M worth of options that have now decreased in value, which is what it looks like since AMZN went up Friday and today.
> 5QxjKb7SI2j1d9Zs3jcs · 809 points · 11 hours ago
> As a proud cuck, not gonna lie I get off on being locked out of my account while it gets pummeled and I can do nothing but watch.
> BippityBoppityZop · 156 points · 10 hours ago
> Don’t stop I’m almost there
> SensualKoala · 47 points · 9 hours ago
> "Oh yeah FUCK my savings harder"
> TRAIN_WRECK_0 · 84 points · 9 hours ago
> Anyone feel like barfing? It's almost like my wife's boyfriend stopped fucking her and starting fucking me, and there is nothing I can do to stop it.
@nostromo can't have read very much of that thread. It's one big in-joke.
Or, you know, buy for the long term and ignore today's news.
Regarding leverage: buying options is a fixed loss against the cash in your account. RH also only allows selling secured options. There's no way to lose more than you started with (at least with the current app, they had problems before).
Retail investors using standard accounts can't buy options on margin, it's only used for faster settlement.
I was certainly caught off guard but still managed to make a bit of money (~6%) off of one of those idiotic volatility ETNs today.
None of this changed on Monday.
The position I kept, I got fucked hard by this Robinhood downtime. Ended up losing all of it, (4x of my original 13x profit), whereas I was trying to exit it early Monday. If I lose money on the market and it's my fault… fine. But when my broker causes the loss, I'm pissed
This aged terribly. Anyone who did that received a massive windfall.