Hacker News Reader

Comment by blantonl | Hacker News Reader

Merrill Lynch / Bank of America's trading platform is also down... so this isn't limited to Robin Hood. I was completely unable to place any trades or even check balances during the market opening this morning.

However, engineering HEADS SHOULD ROLL. In the era of on demand computing, near limitless cloud resources, etc, these engineering teams should have already had capacity planning ready to go for these trading platforms. I'm extremely disappointed - these engineers LIVE for this stuff, and they dropped the ball.

Again, heads should roll. This is not some data center that caught fire, this is capacity planning at its lowest common denominator.

View on HN

A couple of things wrong with this:

1) you make it sound like scaling a system, especially one doing lots of financial transactions, is trivial. Just because you can get access to nearly unlimited computing resources doesn’t mean everything just magically scales infinitely.

2) why should heads in engineering roll? Do you know it was incompetence on their part? Do you even know what the issue is? How did you diagnose the fundamental issue here without any further info?

I’d hate to do a post mortem with you. Sounds like you’ll be basically calling people involved idiots and for them to be fired.

he is the customer, not a employee. As a customer I don't actually care about postmortems, i just want things to work. If it doesn't work, I use another product. I'm leaving robinhood due to their issues, and it sucks it takes so long due to their transfer limits

That's fine and for every customer to evaluate for themselves. But there is a difference between saying "as a customer the product isn't acceptable so I will use something else" and "engineering is incompetent and someone should be fired". Especially if you don't actually know what happened.

People are emotional, especially when they lose something of value. And they should not be fired, they should just cover the loses...

The market hit a circuit breaker this morning when it was 7% down. No one could trade at the open, not even professionals, because the market was stopped.

No, I couldn't event place any orders whatsoever, and couldn't check balances etc. Order PLACEMENT and order execution are two different things. The former was impossible as well.

That doesn't seem to be the reason Robinhood is having issues.

I don't know if it was the case for the two other platforms OP mentioned but Robinhood seems to have some issues handling the request volumes.

Designing a real time system to scale to this magnitude is not easy at all. I wonder how many engineers would be capable of designing such a system, I sure couldn't. This isn't like most scaling where delaying/queuing can be an acceptable trade off.

Maybe they shouldn’t be in business if they can’t hire the engineering to do the job. Every other discount broker seems to have no problem hiring this level of talent.

Sometimes engineers are not at fault. They are well aware of technical debt and possible performance mine fields. It's the management that does not properly prioritize performance and scale work because there is no apparent "value" to the customer. Until it's too late.

Agree that heads should role, but it may very well be management decisions that brought them to this point. We don't know if engineering was resource/strategically limited (seems unlikely at a well funded startup but it may have been). Having worked places where terminations were the result of failures, it's not always the party most responsible that is let go.

Bad idea. You would be just rolling heads of people who just learnt a valuable lesson in high throughput services. Keep them around, they'll get better than hiring someone else who will have to spend more time learning the same lesson.

> In the era of on demand computing, near limitless cloud resources

Should be pointed out that BoA decided to build their own "cloud"

There should be accountability for sure. I am not sure if rolling heads is enough or even the right thing by itself - but there definitely needs to be a 3rd party audit, and a public report.

Reply on news.ycombinator.com