Robinhood goes down again, during another historic trading day
cnbc.com
cnbc.com
If this was Wells Fargo we'd see the pitchforks. But it's a SV company so all is good.
Not that I expect them to do any of this, just pointing out that 'shut everything down' is not necessarily the best approach.
IMO anyone who opened a position on RH in the last week should not be shocked that the same thing happened again in this high volume crapstorm.
Provision the resources and take the financial hit in costs.
If your architecture wasn't able to scale horizontally because it was poorly designed, heads should roll. This isn't some social network - these are financial platforms where literally the individual customer is financially dependent.
Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
A system like robinhood can have hundreds of moving parts, if 99 of them are horizontally scalable but 1 is not, eventually that 1 piece will become the bottle neck, and the fact it hadn't been made horizontally scalable yet is more likely to be a testament to how much work would have to go into doing so.
> Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
This is an unfortunate viewpoint. How quickly did we forget that robinhood is literally providing a service that no other company was able to do before. You want 100% uptime, you won't find it in an online service, let alone an online fee free service, you'll find it on the floor of the exchange.
ftfy
Haha, true. Call up you AWS account representative and you may find that certain service limits can’t be increased for love or money.
$0 commissions aren't a big deal either. If you just buy and hold, it makes no difference. If you trade actively then a real broker with better tech and order management is worth way more than the fees.
Mistakes, any mistakes, are very hard to excuse when people, or other financial entities happen to lose money.
Comparing it with a site like reddit is, frankly, a bit disengineous.
Your customers are paying for speed and performance .
In a industry where users spend millions to be closer to the exchange if you are not provisioning for some multiple of your peak load you are going to get slaughtered in the market
The last blog post blamed overloaded DNS so it doesn't seem like a major architectural problem, which is why it's surprising that they're down again.
Nearly every write-action on Reddit appears to hit a queue. If your vote or comment doesn't go up for 30 seconds, that's not a problem.
If your trade takes 30 seconds, that could be a huge deal.
Somehow, in my mind, the responsibility of safely handling money of millions and redesigning the entire system don't really mix. Ditto for strict regulations and redesigning the entire system.
Please tell me you do not manage a team of software developers.
Rolling out fixes fast, even if they require intensive changes, is completely reasonable and expected in many industries though. They should have the talent and procedures to get it done. This isn't some startup web app, it's a multi-billion dollar broker managing millions in client funds. It's bordering on incompetent to have 3 outages in 2 weeks.
EDIT: What exactly is everyone disagreeing with?
This is a financial trading platform. Do you understand the risks of potentially introducing a different bug?
Changing a single line can introduce a different bug. Use proper QA and testing to catch as many as possible as with any development.
My emphasis is on getting things fixed quickly. They need to do whatever it takes to get systems online asap. Not sure what's so controversial about that.
I'm surprised by all the misinterpretation in this thread. Seems like it reflects the laid-back West Coast/SV attitude that isn't a good fit for high pressure time-sensitive work in other industries.
This is the part you don't understand. There is a difference between digging ten one-foot deep holes vs one ten-foot deep hole. People need time to plan how to coordinate and then get on the same page so that everyone can work at their own pace. That is the part that is not parallelizable and is the rate-determining step.
If this sounds unfamiliar or onerous then it's because you and others might have never experienced teams that do this. Robinhood is clearly lacking this experience and disaster planning.
RH wasn't prepared with any contingency. They should have a resolution for their users - even if they can't find or fix the original cause. That's the failure I'm talking about.
See the 2 other users in this thread that describe similar high-pressure situations.
Reality is, without knowing more about what’s causing this it’s impossible for either of us to say. If there is indeed some fundamental bottleneck that was previously not known, then I certainly won’t be surprised if it takes a while to sort out.
Now you can say they should’ve load tested, capacity planned etc etc. But we are where we are. Still can’t go back in time to turn this into a quickly fixable problem if it’s currently not.
Edit: also pretty disappointed that we don’t know more about the root cause. As an user I’d want to know what the issue was and what they are planning to do to about it to evaluate if I should trust them going forward.
So I'm not sure if I know that the state of the art for trading platforms is as rock solid as everyone is implying, and Robinhood seems to be way far off from whatever gold standards there are, see infinite leverage bug. So I don't think it's crazy that they move quickly to fix it.
I'd never be able to stomach the pressure, and I wouldn't wish it on others, but it doesn't seem crazy.
There are many ways to resolve these issues FAST. The easy way is to throw money at this. Go tomorrow (literally) and bypass all procurement controls and go to a mega big provider and scale this asap. Any financial services with a half decent IT has done the paper exercise to this scenario (at least for the purposes of BCP/DRP). The slow/better/mature way may be too slow, especially with the current market conditions.
The RH folks will definitely get a visit from SEC, their external auditor, and their external auditor will get a visit from SEC.(their auditor will be in the deepest of shits)(how come they failed to spot such a going concern issue?)(what the hell were they looking for on their audits?)(did they only send juniors over there?)
I feel sorry for the retail traders that got knocked down. I think that anyone locked in buying at 27-28k (US30) should wait 6 months to breakeven and after the US elections (irrespective of the winner) there will probably be a rally.
They are probably at ~10X peak traffic at times.
Just are not prepped for a hardcore crash.
[1] https://www.ccn.com/dow-crashes-so-hard-might-close-stock-ma...
[2] https://www.businessinsider.de/international/stock-market-ne...
[3] https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u...
Robinhood and other brokers don't execute any trades, they just forward orders from users to their exchange partners and show up to date quotes. They have a much easier load to bear than actual exchanges.
It's a lot easier with something like Robinhood. So, ethics aside, the level of measurable risk they have now is existential. Lawsuits will flow.
What Robinhood is facing is like that, but with no notice. It means they ~always need to be ready for a 4x(?) increase in load.
> It is well known that one of the best services a retail broker can provide is not answering the phones during a crash. The market is down, the customers panic, their timing is terrible, they want to sell at the bottom, they call you up to say “sell everything,” you say “we’re sorry all our representatives are assisting other customers, your call is important to us,” they hang up and get distracted, the market rallies, they forget about selling, you have saved them a fortune, good work.
[0] https://www.bloomberg.com/opinion/articles/2020-03-03/robinh...
In the movie (based on real events) the same thing happened: during the housing crash suddenly every bank had 'technical issues' when people tried to sell their stocks.
Disclaimer: I'm not in finance, don't know much about it, but it's what I understood while watching the movie ;)
Yeah, right, it's so nice platforms prevent people from selling, just as the big fish unload their own stocks.
Reality is the regulations on the financial system are a feature. Being able to have charges reversed if I get my card skimmed and someone makes a bunch of charges is a feature. Being able to get my money back if the seller ships me broken goods (or nothing at all) is a feature. Having regulations on banks front-running their customer's transactions is a feature (one that Robinhood unfortunately gets around somehow). Having regulation on securities not being scams is a feature.
As with anything, the answer is sometimes.
You want those rules for your retirement account, because if someone can steal a six figure sum from you instantly with no takebacks, that is bad.
But the overhead of that system is not always required, and it's the major reason that we don't have e.g. micropayments. The overhead of allowing transactions to be reversed eats the entire transaction amount for very small transactions.
So it would be good to have a system where you could, for example, transfer $50 into it from your bank account, wait the month or so until the transaction can no longer be reversed, and then transfer it from there a few pennies at a time with insignificant transaction costs because the small transactions are instantaneous, anonymous and can't be reversed. Which isn't nearly as problematic because even if you get scammed, your loss is limited to the $50 you put in. Instead of getting wiped out, you pay a reasonable cost for a personal lesson on security and trust.
And there is no reason we couldn't have both. Then you keep the bulk of your wealth in the inefficient safe system and some petty cash in the one with lower transaction costs and get the best of both worlds. But the existing rules don't allow that, which is bad.
For example, if we took a simpler example of something more people are familiar with like options...let's say you buy a put at a strike of 100. The stock is trading right around 100. The market tanks like it has been doing and the stock goes down to 80. That's great for you since you had the put. You technically have the right to buy the shares at the market and shove them onto someone and force them to pay you 100. Most of the time, since we're dealing with derivatives, they just trade the value of the option itself and skip dealing with the underlying security because it takes a lot of capital (100 shares * 80 bucks each in this case). So the value of the option (the derivative) should be around $20 as you approach expiration and the premium fades off. If the market maker for the options has way too much exposure and wrote naked puts (meaning they didn't have the underlying security) they took on a ton of risk and basically got fucked. In this scenario it's like Burry calling and finding out his options are worth like $1.50 instead of $20 simply because they are thinly traded and therefore can't get a good price by anyone. This actually happens on some options - you'll see a huge spread between the bid and ask on some options so it's not like it doesn't happen even for this type of retail-friendly derivative.
In the Big Short case, the banks were acting as market makers, or more technically in this case dealers- this was more akin to going to a used car lot. You read a lot of reports saying that the new Fords were flying off the lots for outrageous prices, and to people with dubious ability to pay for the sticker prices they did because they got loans they shouldn't have. You go to the dealer and say I want to sell these cars short and buy them back later. The dealer presumably thinks you are an idiot, or wants to hedge some risk (the analogy is breaking down here), but says ok sure, thinking he is going to sell them for even more in 6 months. 6 months later comes- you were right- there is a flood of Fords on the market at half the price because so many people had them repossessed or are desperately trying to get out of these loans that are killing them. You go to the dealer and say "Ok bud, I'd like to buy these cars back at half price..." and they say "Nah, these are still worth 98% of what you sold for... how about that price?" and you are kind of stuck. You bought specific used cars that are kind of but not really entirely fungible. You can't just go down the street and buy those cars and replace them. You have to hope that they feel the pressure to lower those prices because they start feeling the squeeze for cash, or a regulatory agency comes in and puts the pressure on. They can kind of live in la-la land and avoid the reality of the situation as long as they have no requirement to sell.
Eventually though, they blinked and once one bank started taking write downs, all banks did, and once that became acceptable, they all followed suit- and they also needed the cash at that point.
Anyway- now back to stocks/options- these have well known discoverable prices and you can sell them on open markets where the brokers you connect to might be on the other end of the deal but its really unlikely. A broker/dealer like Robinhood or Schwab or Interactive Brokers should theoretically have an entirely flat position at the end of the day.
TL,DR: Robinhood didn't go down because it doesn't want you to sell your stocks.
This means that transaction fees spike to 10-100x their normal rate during those crises, or else your transaction just sits and takes a week to get confirmed (if it ever does at all - eventually it will just time out after a few weeks).
As much as I love Rails, it's extremely hard to scale an application written in it to handle millions of users. All the Active Record transactions have to be rewritten as database transactions if you want to make sure that the databas doesn't lock the tables for a long time.
Groupon wasn’t anywhere near real-time that a trading system requires
The choice of a language or framework is only one factor on the success of failure of a system.
You can write a broker in brainfuck if you are so inclined. You can make it rock solid. Your pick may require more or less engineering resources, but that's all it does.
You mention ActiveRecord. That too is premature optimization. Crafting optimized SQL queries will not save a badly architected app. They might even be worse if whoever is optimizing them doesn't fully understand what they are doing.
> if you want to make sure that the databas doesn't lock the tables for a long time.
You didn't mention which database you are using. You didn't say anything about its schema. This is yet another generalization. Not all databases will "lock the tables", no matter how badly queries are written. And, if they do, optimize that.
Usually, only a handful of services are critical. Microservices are all the rage, optimize those to death.
When volumes go up, the servers run out of money, and everything grinds to a halt till humans intervene.
The humans typically want to do a lot of checks they aren't being tricked into refilling the online wallet too. Those checks are usually manual and take time.
Not being able to exit these high-risk positions won't just bring your account to zero, but potentially put you in the six-figure negative if you are unlucky.
I want to be on the other side of a LOT of trades today and I'm unable.
The future is unknown to all of us. In some cases, looks are not deceiving. When the market appears to be on the cusp of freefall, it can in fact accelerate downward.
(1) They trade on margin, so huge drops are much more dangerous to them
(2) Their boss sees a Profit & Loss for the trading day that evening, if they don't like what they see they might give their portfolio to somebody else.
(3) Customers are calling the sales desk and they want to sell; hopefully sales can slow them down, but you may need to sell to pay for redemptions. Many customers may need to sell to rebalance their portfolios, get liquidity, etc. This is a mechanism which can carry instability from one market into another one which would otherwise be doing just fine.
There are a handful of hyper-geniuses that can make gobs of money by actively trading. For the rest of us, there's buy and forget.
Because the Harvard endowment, like most all other funds out there, generally fails to beat the market and has had down years even while the market was up. This is probably due to lower risk tolerance, but it doesn't really show that hyper geniuses are running it either.
Where I live real estate was massively overpriced, so still hasn't rebounded, more than 10 years later.
* this is obviously the caveat and where things went wrong for people.
You're also forgetting that they had somewhere to live for a decade for the cost of inflation of the house, which is significantly cheaper than normal rental prices in many places.
To come out even after a decade, after housing costs, seems a pretty "safe" investment to me. Certainly not as lucrative as having invested in an index tracker, but it beats the 0% interest my bank pays on my current account balance.
Why?
I am not an economist though, this is just my layman’s view.
What you actually have to figure out is: is there any unexpected future efficiency gains that the market doesn't expect. You have to know something that the market does not.
Isn't this ignoring opportunity cost, uncertainty, etc.? If the expectation of future gains is priced in, wouldn't treasury bonds sell for face value + remaining interest on the secondary market?
You don't actually need to improve efficiency/productivity if the government has tools like the discount rate, open market operations, the ability to alter margin requirements on derivatives, fed funds rate, as well as programs and policies like MMIF, TAF, CPPF, ABCP, TALF, ZIRP, to manipulate the money supply and the velocity/flow rates of money.
Assuming you have no pressing needs for your invested capital (e.g. most retail investors), the market cannot actually stay irrational longer than you can remain solvent.
Well, the companies that don't go bust.
It's good to make decisions with a sober view of risk and reward. I feel so much market advice that filters down to us laypeople is whether we should or should not buy or sell, when really it is all based on our level of risk tolerance, how much in terms of assets we have, what our time horizons are, what our expertise is. I would say a healthy young person with plenty of cash to burn might well look at buying stocks now. But a person nearing retirement, perhaps with health issues, they should probably board up and get ready for the storm. Most people below retirement age should probably just stop freaking out and keep contributing to their retirement plan as they (hopefully) have been doing, and make sure to have some cash on hand.
Our society does not do well with self-control or with gray areas. We want easy answers, which, unfortunately, don't usually exist.
That's because you hear them when they are making gains, never when they are losing.
I could be wrong, but I'd see survivor bias as "at the end of the day, everyone still in the game reports their results", not acknowledging that those doing really poorly might walk away from the game.
This is where the panic goes to though: they might have 2-3 months cash right now, but might feel like they need 12-24 months cash to ride things out. People start worrying they'll lose their jobs and then their runway is at the mercy of the market. If a significant chunk of your wealth is in your house, that isn't easy to get out of during a downturn. Your tolerance for risk goes even further down if you have a family. Even further if your spouse doesn't work.
These are rational decisions that look like panic selling.
Tech work is a field that likely won't have to worry about this though.
With tech stocks tanking and economic growth likely sluggish for at least the next eighteen months, I can see large tech firms putting a hiring freeze in place, making it more difficult to switch jobs and more difficult to get hired if you DO get laid off.
Moreover, all that RSU comp that makes such a big portion of many tech workers take home pay just took a 20% haircut from a few weeks ago. It could well go lower.
Then there is the startup scene. I can definitely see current events massively impacting funding (see the recent Sequoia post). This could quite reasonably cause many startups to fail just like they did in 2008.
The IPO market is also going to be hit. I don't see any major unicorns doing an offering until the recovery happens. A lack of IPOs means, again, that the people willing to contribute to earlier stages of the pipeline, dry up.
Then there are secondary and tertiary effects. If the average tech worker sees their total comp drop by 10% to 20%, and quite a few lose their jobs, do you think the home you just bought in SF is going to be worth more in a year, or will there be some reduction (in appreciation if not absolute values)?
Right now it really isn't that bad. I'm in London and going to work normally, the city is busy and the trains are still full in the mornings. Our company is starting to migrate people to working from home though. So a proportion of each team is wfh and staying there, for 14 days or until otherwise notified, and more will probably follow not because there is immediate danger, but because we want to be well prepared when there is.
If the markets are like this when there are a few hundred cases in the UK and US, what will they be like when there are tens, or hundreds of thousands of cases and major cities are in lockdown?
Back to investing, stocks are down significantly and it's quite possible now is a good time to start buying, for those with the capital. Maybe spreading out a balanced equity investment portfolio over the next several months. I don't know if or when the market might go lower, or by how much, but it seems unlikely it's going to rally all that much given that we can be pretty sure there's plenty more bad news still to come. So investing too much now might miss further falls, but IMHO buying over several months is unlikely to miss the dip. That's how I see it anyway.
Investors are trying to judge the likelihood of these very events and selling / buying accordingly. Markets are based on predictions.
For example it’s likely several more airlines will go to the wall, but we don’t know which ones yet.
I flew from Seattle to San Francisco and back this weekend. In both cases, my "prime time" flights were less than 1/3 full. Returning yesterday, there was literally zero wait at bag check and security, I walked straight up to an agent. Haven't seen an airport like that in years, let alone a major.
Some conservative predictions are arriving at millions of cases in a couple of months. There are too many unknown variables for an accurate prediction but, without really drastic measures, it's difficult to argue with exponential growth.
At some point it will plateau and become another seasonal flu. And we may even get vaccines. But I agree that things are going to become way worse before they improve. Most worrying is the strain on the health care system, which may by itself worse prognosis on other, otherwise unrelated, diseases.
Why not? Assuming you can weather the bear market, the bulls will eventually return, no?
The high was $74 per share. The low was about $2 a share.
I’m on the East Coast of the US, and though it makes sense, I haven’t seen it before.
It was was also used for the 1900-1909 decade as well.
And that's bad. And it's the kind of behavior we tend to see at these overheated moments where the market has gotten away from the fundamentals and all the players are trying to find "tricks" to keep the gravy train rolling.
Failure in moments of market stress is the worst way for a brokerage to protect its position in the broader industry, because downtime damages customer confidence (long-term income) in addition to reducing order flow (short-term income).
Order flow is more valuable in moments of extreme volatility because spreads widen when the market goes nuts.
It happened aplenty in 2008, when banks were rushing to offload toxic crap that they owned on to the market FIRST, and only THEN were they willing to help their customers do the same.
Let people sell, that's on them.
Market access is a major component of liquidity. If your broker already holds a phone in either hand, he won't be able to pick up your line when the light next to your name flashes.
Imagine that you have all of your liquid net worth in GLD and a huge cholera outbreak hits lower Manhattan. Bad shellfish. GLD rallies and you want to close your position and take profits. The problem is, the largest firms that make markets electronically on GLD have sent their employees home and shut down for the week. Their absence from the market makes it difficult for you to get a fair price for your large position.
Imagine that you use Verizon for your mobile phone and internet service. You think their service is great, so you buy shares of VZ. One day, Verizon announces that due to a malicious hacker, their network is experiencing cascading nationwide hardware failures that will not be repaired for at least a week. VZ stock plunges, but you can't modify your position because you can't get on the internet or make calls.
A less fanciful example would be owning stock when trading has been halted by the exchange. Perhaps there exists a buyer during the halt who would be willing to trade at a price that is favorable to you, but that transaction cannot happen.
Tail risk, in some cases, is a scenario where stocks get cheap. In other cases, stocks can't trade at all. (And in the worst of situations, tail risk is a scenario where guns get expensive.)
When we're talking about an enterprise scale electronic trading system that should already have tested these sort of load scenarios, especially in a world where AWS/Azure/etc. exist and elastic computing is the target roll out for applications that probably don't even need scaling, that's not a reasonable case. That's a case where something that should already have been planned for and tested thoroughly slipped by. It's especially damaging when it happens repeatedly over a short time span. There's nothing unpredictable here. Management and IT at Robinhood surely read the news and knew there would be massive load today.
Surely, someone designing systems and software in this domain understand that the market could be highly volatile and their system needs to be ready for those cases where it's flooded by customer requests or clearly conveyed to customers that it's not ready for such cases.
"You should assume you can reasonably control your assets at any given time." Sometimes you can't reasonably control your assets, because you can't control them at all. So you shouldn't assume that you can reasonably control them in those times.
I hope it is clear that I am not trying to be argumentative; I actually think it's a complete let-down that Robinhood has left so many people frustrated as their wealth whip-saws around in the market. I'm just saying that tail scenarios don't usually fit into our predictive model of what's "reasonable," even when the solutions seem crystal clear in retrospect.
On a more general point, I never understood why market returns are assumed to be gaussian - these "one in a billion" events seem to happen far too often.
> "Other firms were less fortunate, and in a number of them confusion gained the upper hand so thoroughly that some brokers, tired of trying in vain to get the latest quotations on stocks or to reach their partners on the Exchange floor, are said to have simply thrown up their hands and gone out for a drink. Such unprofessional behavior may have saved their customers a great deal of money."
"Oh, hey, we saw your message but we were just too overwhelmed to execute it. We did however, manage to do the next best thing, which was to purchase an option, expiring today, to sell at the prevailing price at the time of your call, and, to make things right, we'll go ahead and exercise that option, unless you'd prefer we just debit you the cost of the option and let it expire without exercise?"
'Debit! Debit! Just debit!'
Brokerages aren’t doing investors any favors that are purposefully trying to buy a dip or sell for cash, they are only doing themselves favors as the middle man.
Lame arguments from brokerages don’t change that. Has there been a recent case of “unexpected” downtime during a huge market rally?
Their target audience is casual investors. The kind of people easily influenced by reddit/twitter sentiment.
The fact that wallstreetbets made a PREEMPTIVE RH goes down thread [0] (2500 comments) is pretty conclusive proof that they're never living this one down.
[0] https://www.reddit.com/r/wallstreetbets/comments/fftyri/pree...
Still useful enough, and if you want, you can get to your own dashboard... but things like this (inconsequential as they seem) can be the difference between going down or not.
Each non-personalized element on the page that's removed saves at least one DB query (and in a fancy app like Robinhood's case, maybe a half dozen).
Innovators dilemma?
Which is somewhat ironic considering the attitude around things like Trade Secrets leading to a plethora of Chesterton's Fences being basically guaranteed to litter the economic landscape.
The problem is RH itself.
It's like saying Airbnb is taking down root level DNS servers.
I didn't realize they were a big player in this game. Citadel definitely is, though.
This is strictly an RH problem. Their last blog post said it was a DNS problem.
Since all securities are affected, it probably actually is their OMS which is malfunctioning.
Loads of VC money means you can take more time up front to architect things better. Maybe have more experiences engineers and architects on the team from the start.
As much flak as I’ll get for saying this, why Python? It seems like Robinhood went all in on an ecosystem that is easy to get an MVP off of the ground with, but certainly not as fast and reliable as you would expect a system moving billions everyday. Language typically doesn’t matter for end users, but man, there are at least 10 solid options to build a critical trading system in that are faster and safer than Python.
Technology and scaling are tough. Reddit, Twitter and CloudFlare have gone down in the last 12 months, despite being constantly stress-tested.
Especially not when this happens for the third time in a row.
I wonder if people will leave the platform en masse to alternatives as most brokers have no fees on trades these days.
Most folks I know who use RH just use it as a fun little side gamble/game. Small-ish amounts of money, something they could afford to lose, with the rest in more stable funds or investments.
On the other hand, I don't really care. I'll be fine. Most of the folks I know who use RH are just joking about it being down. It's annoying, but no one's broken up about it. A lot of the folks casting their anger at RH seem to be folks who already dislike it and probably don't use it?
Professional or large-scaled investors with large sums invested probably aren't using RH for a lot of reasons even before you get to the downtime issues.
It probably works fine, but I am never as confident on a touch screen. I even prefer to order pizza on a desktop.
You are correct in that it's more likely their system just can't scale.
However, engineering HEADS SHOULD ROLL. In the era of on demand computing, near limitless cloud resources, etc, these engineering teams should have already had capacity planning ready to go for these trading platforms. I'm extremely disappointed - these engineers LIVE for this stuff, and they dropped the ball.
Again, heads should roll. This is not some data center that caught fire, this is capacity planning at its lowest common denominator.
I don't know if it was the case for the two other platforms OP mentioned but Robinhood seems to have some issues handling the request volumes.
Should be pointed out that BoA decided to build their own "cloud"
1) you make it sound like scaling a system, especially one doing lots of financial transactions, is trivial. Just because you can get access to nearly unlimited computing resources doesn’t mean everything just magically scales infinitely.
2) why should heads in engineering roll? Do you know it was incompetence on their part? Do you even know what the issue is? How did you diagnose the fundamental issue here without any further info?
I’d hate to do a post mortem with you. Sounds like you’ll be basically calling people involved idiots and for them to be fired.
After vents like this juniors turn into seniors, seniors into VPs, and VPs going on sabbatical.
I was an intern in a team where everyone who's been in the company for more than a year was a VP.
And its not unheard of. My sister-in-law worked from startup contributor to Cisco VP by applying for the next job up, one at a time, from company to company.
Senior Developer in the "outside world" typically means Associate in the banks, and VP is the next rung up from Associate, so VP really is just one hop from Senior Developer.
Just how they do things there.
Honestly, I don't. There is enough public information about the brash attitudes of this company's execs that anyone working for a company like this should know that the lowest level engineers working on a technical problem like this are eventually just going to get thrown under the bus.
I hope this is a lesson learned for smart and impressionable engineers about where to work.
My point - it's difficult for me to feel the same anxiety someone might feel in this situation when they equally ignored the risks of something that is easily identifiable. They very likely felt elated when they joined because the company was seen as cool, hip, and by all objective means is/was successful (valuation, # of users, etc).
1. https://blog.robinhood.com/news/2020/3/3/an-update-from-robi...
> We now understand the cause of the outage was stress on our infrastructure—which struggled with unprecedented load. That in turn led to a “thundering herd” effect—triggering a failure of our DNS system.
I was planning to wait for the post-mortem because I dislike speculating and I'm not personally invested here, so I prefer to wait. I'm curious when they'll release the full one because it's going to be one of the most interesting ones in a while, given the large scale of the failure and potential implications... (ignoring today's outage)
Faust is a stream processing library, porting the ideas
from Kafka Streams to Python.
It is used at Robinhood to build high performance distributed systems and
real-time data pipelines that process billions of events every day.
Nothing wrong with Python in itself. It just feels like the wrong tool for the job.
I suppose that's what I get. I realize that I should be looking at yearly data and I've only had them for a couple of months, but it's a little hard to look at.
Not to mention with all of the other RobinHood issues I want to move to another platform I'm already on (I've got an account with Schwab for work stuff and they seem fine), but general wisdom says to hold and let this craziness ride out.
THEN: dollar-cost-average into the market. Put a fixed amount to work every month. This has been a terrific strategy over the last 50 years or more.
But, caution, you probably don't want to buy a few shares per month of a stock or an ETF in a taxable account. That's because each transaction is a separate taxable event when you go to sell. It would be a nightmare for you to track this yourself. I have no idea how well RobinHood tracks that for you.
If you are investing in an IRA then no worry. The transactions aren't taxable. Of course an IRA has its own problems. Outside of an IRA long-term capital gains are taxed at a lower rate than ordinary income. But when you withdraw funds from an IRA you are taxed at the higher rate.
A Roth IRA doesn't have those tax disadvantages. I've seen a lot of press lately on this issue. For example, in Oregon, the state itself is promoting Roth IRA investing for the "little people":
Participants saving through OregonSaves beneficially own and have control over their Roth IRAs https://www.oregonsaves.com
There are also tremendous advantages to having differently priced lots -- you can sell the expensive ones, and donate the cheap ones to charity to minimize taxes.
In fact, starting your investment career during a 15% drop is great to inoculate you against panicking in volatile times.
I say a 10% dip should be motivation for you to buy more of this ETF
Word of warning though: Buying and holding for a few decades is excellent advice, but be sure to log in every once in awhile lest you get escheated[1] out of your nest egg!
[1] https://www.npr.org/2020/02/13/805760508/when-your-abandoned...
https://topstonks.com/blog?post_id=robinhood_down_for_the_co...
Needless to say, people are definitely considering moving to other brokers.
Context: my dayjob is being responsible for trading system uptime at a major bank.
I took stability quite seriously. My first boss was fired after knocking is out of the options market at open on a Monday. Refused to accept responsibility and opened the firm up to a potential $600K fine. He was out the door before noon.
Now I can do free trades via my Merrill Lynch account and that works for me. There are many other options.
See replies to Robinhood's tweet: https://twitter.com/AskRobinhood/status/1237016846282280961
The vast majority of Robinhood's customers aren't day trading clamoring to get their daily dose of tendies. The proportion of RH's customers who are actually effected is probably quite minimal, and of those some might change to another broker, but most will probably just put up with it. The market isn't going to be this active for long, and so once it calms down RH will put in place some fixes for its systems. This will be forgotten after a couple of weeks, it'll perform better next time and it'll have almost no impact on RH's value.
I believe this will be Robinhood's undoing and the company will likely just firesell itself (less than $1B) to Schwab/TD so they can acquire the younger demographic.
See my earlier posts about staying away from RH.
I couldn't be happier to see them crash and burn.
FTFY. Just because you work at a company doesn't mean you were around to experience what ever event provided some sort of "learning" experience. If you don't experience it personally, what ever systems the company has in place to protect itself from said events just look like bloat/legacy cruft ripe for a young upstart to disrupt away.
At least in such a volatile time period.
A sufficiently bad positive feedback loop can also drive you into the ditch no matter how monstrous your hardware is.
Robin Hood took market share by being cheaper than other brokers. Cheaper rarely means better (more robust, reliable, etc). That's the price you pay for not paying higher fees.
Full disclosure: I shop around for low fees. But that's because I buy and hold boring index funds. It makes no difference to me whether I can trade into or out of a position today or even this week. If you need to change positions quickly or die, for God's sake, don't pick the cheapest supplier.
Explain-Like-I'm-Five: Why would you use Robin Hood instead of a normal brokerage like Vanguard, T. Rowe Price, Fidelity, Charles Schwab, etc that.. (1) already have commission free trades, (2) an app to make those trades, and (3) don't charge any fee for transferring your holdings in-kind to another institution.
- Still today, the UX of all of those other apps/websites (that I've used) is horribly dated, clunky, and slow. Robinhood is amazing to use.
- Robinhood has a Cash Management account that pays me 1.35% interest on all my non-invested cash. None of these other services come close to competing with this.
- Robinhood started a progressive movement with the mission of making stock trading more accessible to the lower/middle class. I love this company and brand. Schwab is just another faceless corporation who only offers free commissions because they had to to stay relevant.
https://blog.robinhood.com/news/2020/3/3/an-update-from-robi...
Ugh. This is a tough issue to design for, and is expensive to run as well.
I have no faith that this company is operating in good faith.
They make more money when trading volume is high. What they can't afford is losing customers to either competitors or customers realizing day trading is a bad idea.
When people sell stocks on RH (or anywhere else) it’s not the broker paying the sales rate. So you’re alleging something like RH having fraudulently sold customers’ stock before?
And the market was moving the other way when RH also shut down last week. This would be the only broker in history that had its system set up in a way where they suffer with any trade.
They don’t even come here to defend themselves and people still act surprised
This is the same frothiness I remember from the .com when ETrade etc. just became big and all these zillions of people who thought they were investing plowed their money in.
In 2001, the bubble was burst finally by big shift in interest rates from the Fed, and some insane valuations which cause the smart money to run, causing a stampede.
Most analysts have been raising red flags for a long time now. The market has been operating on hype and irrational good vibes for at least a few years.
Warren Buffet started hoarding cash some time ago - long before the Coronavirus.
Listen to Robert Shiller's interview earlier this year on the FT concerning 'narratives'. I don't think it's a new concept, rather just a modern articulation of it. The mania we're seeing is just a more extreme form of narrative.
So far the S&P 500 has lost just one year of gains. I havn't done deep analysis but to me that still seems consistent with expectations that the economy will slow down for months due to sickness and lockdowns all over the world caused by the pandemic (+ this weekend's oil price thing).
The PE ratio of the S&P 500 is still 20, so there is still some way down to go before I would call this the bubble busting .I don't have to hubris to predict how much more the market will fall tomorrow or in 3 months.