Hacker News Reader

Comment by xir78 | Hacker News Reader

Are you able to talk about the changes you had to make to the login service? Would be interested in knowing what the scaling issues were.

View on HN

The core of the issue was that EIDM, the old system, was an assembly of products designed for corporate intranets on the orders of 1000s of users -- it was never meant to be used for a consumer system. So all this super complicated permissions logic was bogging everything down.

They mitigated the effects for a while by running it on bigger computers - when we came in it was running on Oracle's BIGGEST computer, the Exadata. $6MM per environment. Terabytes of ram. But still just completely bogged down by a set of software not designed for its use case.

Thanks for this detail; it's vivid and upsetting. One assumes that VAR pricing was at play here as well: not only did the government get sold a pointless super-computer, but the contractor made points on that sale.

Exactly! The root of all this (I think) is the contracting structure for building jet fighters -- cost plus percentage -- was applied to software, which has such a different cost structure. With that structure there is no motivation to ship working software, or work efficiently at all. So people who value that won't work for any company that behaves like that, hence the shift of talent elsewhere.

God. I thought cost plus contracts were only used in exigent circumstances such as war, but apparently they are getting used more now, such as for the F-35.[1] Are software contracts really cost plus? These contracts shouldn’t be used for fighter jets or anything else. They should be banned except in critical situations and even then used sparingly. I remember stories about the boondoggle in Iraq, $20 coke cans and such.[2] Really sad if these are being used more now. Indicative of a lazy, corrupt government contracting process.

1. https://en.wikipedia.org/wiki/Cost-plus_contract

2. https://en.wikipedia.org/wiki/Iraq_for_Sale:_The_War_Profite...

Na, this happens everywhere with software, startups too. The dev just wants to throw more resources at a problem than fix the complexity problems that make it inefficient. Perhaps it's a sign the dev is just out of their depth.

Just want to confirm 6 million US dollars a months for a computer?

6 million dollars for a computer.

We basically rebuilt things from scratch on a node/MySQL/nginx stack while providing the same API as the old system, and were able to swap it out. We have become masters of the strangler pattern of software design...

Reply on news.ycombinator.com