Robinhood Is Down Again?
status.robinhood.com
status.robinhood.com
Thread from exactly 4 years ago, March 2 2016: https://old.reddit.com/r/RobinHood/comments/48mep4/robinhood...
Thread that pointed it out: https://old.reddit.com/r/wallstreetbets/comments/fcoaev/and_...
As a customer who lost a decent chunk of money yesterday (and today) because their system was down, I'm angry.
I hate to say this, but I do not think they're competent enough to continue to operate as a broker. I, for one, will move all my money off as soon as I can. I'm also going to file a complaint with finra[1].
[1] https://www.finra.org/investors/have-problem/file-complaint/...
Why? Aren't they paid succulents sums of money for their work?
What is going to happen to them? Overtime?
At this point for me its not so much about the lost opportunity as it is about what the hell actually exploded so hard that they need to be down for 2 consecutive days. From my perspective, it's looking less and less like some one-off technical or hardware screwup and more like a fundamental limitation of their core architecture.
Seems like infra problems.
As for believing/unbelieving, I mean, I've seen dumber mistakes in financial products I've worked on. Not as high stakes or damaging, but definitely dumber.
That at least means that some portion of their stack used roll your own datetime library. That would not actually be a problem as long as the entire stack got the same library or the same rules for datetime. The problem is of course that it probably did not so the libraries were not bug compatible which of course caused errors and those errors need to be handled, preferably every fast.
Error load and error handling is the least tested part of every system because it can only be properly tested in production and, unless your company embraces the Chaos Monkey approach to testing, the C-level would have a heart attack when anyone proposes doing it.
https://api.robinhood.com/markets/XASE/hours/2020-03-03/
vs
https://api.robinhood.com/markets/XASE/hours/2020-03-07/
This whole discussion lacks context to the nature of the requests, aka a front end code review.
Somewhere within the infrastructure there's going to be an assertion such as
if (max_drift_delta > delta(order_live_date,system_live_date) {
# oh crap, something is completely broken in our system where did this come from?
blow_up("terrible things are happening! how did we get this order")
}
which is an excellent and correct catcher for "terrible things are happening" since those things should never happen. That blow_up() code path is likely to be very expensive which kills performance of the system, which in turn means that it no longer can handle the load.And since RH has lots of people who use apps, it is not that they can just push an immediate bugfix.
I spent a better part of a year working on it (along with other things). Modeled the interface after Python's datetime module (which I think is one of the simpler and easy to use date-time libraries across various languages I've used). More than $20B USD trades using that library.
The motivation was the firm used to use RogueWave's date-time facilities, but we moved away after they jacked up the licensing. Think we used to have a site-wide license, but they were moving to a per-core licensing, wanting something like $2K per core annually.
Needless to say, testing was extensive. Overflows were found in Boost in far distant dates, had to work around those. Tested against a ton of historical dates. Tested against Northern and Southern Hemisphere daylight saving time (a lot of people don't realize that Southern is inverted from Northern). Learned a lot about timezones and the history of timezones along the way.
[0] https://www.boost.org/doc/libs/1_72_0/doc/html/date_time.htm...
I was a little concerned they were front-loading orders. Still a valid question, but they say no: https://www.m1finance.com/blog/how-m1-makes-money/
Not to be too spammy, but here's a referal link if you want the 10$: https://mbsy.co/swlqd
This isn't the first time that it's happened. Robhinhood should have been prepared. This is unforgiveable.
And if there's one thing people hate more than going slow it's spending more money.
What's wrong with people who think that they can have SDLC without a proper QA? I've seen this many times in the SV. Developers thinks that QA is just waste of time and their stellar code doesn't need any testing, because: "Hey, if something goes wrong I can always roll back". I don't buy it.
I know not every company needs it. I was the second QA engineer at a food-tech startup in Boston 5 years back. QA got laid off after 10 months but they are still successful without QA right now. They hire great devs and have a great process. At the end of the day, if the site stops working, people will just get some late food deliveries. One the other hand, I've a friend working at Ford in Michigan and he says they have no code. You have hundreds of engineers and everyone manually tests their own code. He says sometimes he will test for a couple of minutes and just push his code. I don't blame him. Devs are great at writing code. Let them write code but have someone else test their code please.
I've a small options position open on Robhinhood. I'm not too worried about it but I imagine some people have tens and hundreds of thousands. I've read some rumors that this outage was caused by the leap year timing and it happened 4 years ago too. This is just unacceptable.
I understand what Devops (a similarly amorphous term) people do, but I've never had a clear picture of what the day-to-day life of a QA engineer would look like.
> To those of you who deposited funds to Robinhood and are now down your deposit or more, these are technically ACH transfers, and if you genuinely believe Robinhood defrauded you, or otherwise failed to provide service as understood, you can issue an ACH chargeback through your bank’s web portal or by phone for any funds deposited in the last 60 days.
> RH may try to call you into arbitration, but more likely they’ll just ban you from the platform. It’s not a terrible idea if you plan on leaving the platform anyway. Just make sure you can substantiate your losses or they might have a fraud case.
> [If RH dings your credit], you‘re able to dispute with the credit reporting agencies or, if the agencies won’t delete, hire a debt lawyer to resolve. I doubt they would even if able. Any representation they have is likely about to be drowning in FINRA and arbitration prep, some people keep obscene amounts of money on their platform. They also likely can’t furnish any signed documentation of credit pending payment unless you signed up for margin trading.
> If the funds haven’t been deducted from your bank account or are in a pending state, you may also be able to just issue a stop payment. Again, anyone who tries this should ensure that they can substantiate losses related to the payment they stop or chargeback.
-Zookeeper, Kafka, Spark, Airflow, Faust (Kafka Streams in Python) -Kubernetes (was/is SALT + Terraform) -AWS Aurora, PG, ES, Influx -Presto via AWS Athena, Redshift -homegrown solutions for a data lake & managing k8s microservices
Fwiw, their team does appear to have a rigorous internal post-mortem process (SEV reviews) [0] drawn from industry best practices.
Let's hope a public one will be made when issues are resolved. (FINRA will be asking for one anyway.)
Not piling on. I genuinely feel for all the infra engineers staying up all day/night fixing this.
[0] https://robinhood.engineering/creating-a-sev-process-that-sc...
I would not be surprised if a “move fast and don’t test things” approach to margin stress testing would collapse in the market we’ve had for the past week.
https://twitter.com/askrobinhood/status/1234861941413351434?...
Sounds like Kafka / message bus to me
> Why?
EDIT: Because their systems are garbage and their support non-existent. You can do better literally anywhere else (Fidelity, Schwab, Interactive Brokers).
Blanket statement like that without reason would equal please switch to self hosting git instead of using github.com cause it went down last week or so - both have their purpose / skill level / cost associated with them. So a 3rd party app went down for a bit. Happens often (as most of us know)
The fact that they’re trying to get people to increase trade volumes while being stuck to one very limiting tax strategy is bad.
If you’re buying stocks you want options and only having FIFO is not a good option
Second, now your investment might be small, if it grows, and represents a portion of your savings you might not be as delighted as today, to see them being offline during such a busy day.
Individual investors are at a huge disadvantage when it comes to intraday trading.
For everything other than a child's hobby account, intended to teach emotional stability through gains and losses, a <$1,000 stock-trading account is value destroying.
Also, one need not trade a small account intraday. I make small trades, but they are always with a multi-year perspective. Give me limit orders and small commissions, and I am happy.
(N.B. I use a different broker/don't have a first-hand perspective on Robinhood.)
The key to avoiding faffing around with limit orders not filling when you want immediate execution is to place a reasonable limit that accounts for the day's volatility. At other times, I'll place a limit order and let it stand for weeks. When it fills, the counterparty and I are both happy.
I'm rarely using limit orders to shave pennies if I want quick execution. I use them to prevent the unexpected. Under normal market conditions, if the limit is set 10% beyond the expected clearing price, it provides me with free protection against a completely unexpected surprise. Something will have to have gone very wrong with my investment strategies if I'm desperate to buy/sell at any price.
In the long-game case, if I purchase a stock at $0.95 that I think can sell for $0.99, I'll immediately place the limit-order for sale as soon as I've made the purchase. No need to hide my hand -- I'll be happy if you want to buy it at that price (and I might get faster execution by being earlier in the order queue).
you want me to risk > $1000 when I don't know what I'm doing with a real brokerage that I don't know what their benefit is over something like Robinhood because.... ?
I'm willing to loose a couple hundred to learn and understand something vs giving someone I don't know thousands and "trusting" their opinion.
More so - testing the waters myself may not make me as good as someone who does this for a living - but maybe after a few hundred and a few months, at least I have a better understanding of what a real brokerage tells me to buy than to just blindly say take my money and quadruple it.
Brokers should never be telling individual investors what individual stocks to buy, at least not anyone with less than ~$500,000 in assets. If a brokerage is giving you buy/sell lines for individual securities, that's a red flag.
A good investment platform (or adviser) guides you in portfolio management. In encourages long-term strategic thinking over short-term trading highs. The former builds wealth. The latter lines professional traders' pockets.
You can learn the ropes without wasting money.
You can buy and hold, or "Invest", on any platform which, in my opinion, has to meet three minimum requirements: It should offer commission-free trading on a variety of low-cost index funds, it should offer a variety of investment account types (at minimum, taxed accounts, Traditional IRAs, and Roth IRAs), and it should be stable.
Robinhood only meets one of these three criteria. There are dozens of other brokerages out there which do meet all of these criteria: TDAmeritrade, Schwab, Vanguard, and Chase are four great ones.
Robinhood is on the same level as a betting app for sports; so, fine; use it if that's what you want. But do not even mention the word "Invest" in the same sentence as Robinhood; its grossly irresponsible.
I wouldn't advocate anyone trying to build wealth to pick individual stocks, but I do think if someone were investing $1,000 to get their feet wet I'd recommend they put a bit in individual stocks. The financial system is complex and I think it's worth stripping away one level of the complexity for educational purposes. (It's typically also a humbling experience.)
Now I have a much larger account, and I'm trading / investing with a much lower risk profile, because of what I learned on my >$1000 account. I consider that the exact opposite of wasting time.
EDIT: It seems it's completely removed now. https://www.interactivebrokers.com/en/index.php?f=4969
They're also known for mispricing essentially _every_ morning, showing nonsensical pricing and account values for the first 30 minutes of trading. Overall, their software is clearly lagging behind something like Think or Swim from TDA.
Robinhood kind of made sense for low-value accounts that wanted to avoid commission. They've changed the game a bit in that most brokers offer low to no commission now. I'll echo the parent comment, I just don't see the point in using them given that most of the alternatives are better.
If you want to use one of the better tools, see TDA with ToS. Schwab has great service and is a great bank for a one stop shop kind of experience. IB is one of the least expensive ways to trade if you meet their minimums.
RH OTOH has shown time and again they play a bit too fast and loose for comfort when money is involved. They have had to pull their entry into banking a couple of times now. The app seems to have issues at the worst times. From what I have read, support is abysmal. Not someone I want to deal with when it comes to my money.
If you are trading much more frequently/short term, or WSB style options, then yea, I think that a proper brokerage (IB) is likely better for you.
Also perusing twitter --- lots of people were planning on executing that _perfect trade_ if only RH was up... lol
EDIT: Why am I getting downvoted for this..? That's the reality of the situation here.. If I missed something, please reply instead of just downvoting.
https://www.cnbc.com/2020/03/03/trading-app-robinhood-experi...
Robinhood, a pioneer of commission-free investing, gives you more ways to make your money work harder. Robinhood, a pioneer of commission-free investing, gives you more ways to make your money work harder. It's Time to Do Money.
My company's 401(k) provider's site was also experiencing significant issues. Seems like last week would have been a fun one to be a system administrator in the investment/brokerage sector.
Robinhood has the worst customer service and security. My account got hacked, I changed the email, phone number and enabled 2FA and it got hacked again. I was trying to resolve it for a month.
I really don't see why anybody would use Robinhood instead of an actual brokerage, now that all of them are free.