Hacker News Reader

> We put databases on Oracle that should be handled by Postgres or MariaDB since DoD prefers Oracle.

I mean, if you’ve already budgeted the CapEx for some additional Oracle licenses, the OpEx efficiencies of having unified tooling and a unified ops doctrine are no joke.

I haven’t worked with an Oracle DBMS, but I think this is analogous: I’d sure hate to have to manage a cloud infrastructure where parts were on AWS, parts on GCP, and parts on Azure. Sure, there are generic tools that treat all three the same, or over-layers like K8s that don’t care about substrate—but what if the projects on each platform were taking advantage of that platform’s specialties? What if I was using SNS on AWS, or BigQuery on GCP?

To bring that back through the analogy, what if our Oracle projects were tuned using Oracle-specific query-planner hints, while our Postgres projects did their ETL using PG-specific Foreign Data Wrapper connectors?

In both cases, the only real solution is hiring and retaining O(N) specialized ops headcount, one team for each stack. And that cost gets a lot higher than just paying for another darn Oracle license.

Even with an abstraction layer like kubernetes you still have a lot of duplicate work if you're multi-cloud. Services need to be exposed with load balancers, and those will have different configurations to be setup. Same with any Volumes. And then you have platform updates on both sides, bugs, quirks. plus the maintenance and upkeep of two clouds - two bills to inspect, two account managers to deal with, two sets of permissions and overall account configuration to setup, maintain and audit. Multi-cloud is really a set of requirements that changes the whole game in terms of operational overhead. And we limited our example to kubernetes. I imagine any non-fictional-for-the-sake-of-example company would want to make use of other platform specific tools as you mention.

I think a lot of this speculation requires inside knowledge of what the DoD's use cases actually are.

Are they doing a lot of compute, a lot of ingestion, a lot of output, and a ton of networking? Are they primarily just doing one of these things?

Who knows?

There's a lot of cases where having multiple clouds could be fine -- maybe even a big benefit. There's also a lot of cases where it could be a major headache.

> Who knows?

I know a little about it from a previous employer.

Even the narrow slice I saw was all of "a lot of compute, a lot of ingestion, a lot of output, and a ton of networking", and more.

I think that the internal inefficiencies in the DOD datacenters are so enormous that any kind move to something more 'standardized', no matter what company it is, even with all of the artificial overhead, etc, will likely be a big win.

>>DoD's use cases actually are.

We all saw T3, we know it is skynet

I'm sure that these deals are complicated at the size they're going at and maybe laymens pricing models just get tossed aside, but one of the biggest things I spend time on in cloud architecting is all around data ingress/egress in order to control costs. I simply don't understand how it's possible to go multi-cloud and control those costs, I feel like you'd either blow costs out not caring, or blow costs out throwing engineers at very complex solutions.

Not just governments. I worked at a F100 company that had a special type of internal funding called SQP for spending on (mostly) IT people-hours on projects. Year after year, the biz would penny-pinch on their SQP to the point that we struggled to keep core contractors on staff. Then around Oct/Nov, biz would come to us begging to spend their SQP on anything (else they could lose it next year) as long as that project stop billing at fiscal year end (not a day afterwards; as that would be charged to next year's SQP). That meant we now struggled because we didn't have enough people on staff to burn up all the SQP. Famine then feast! I would suggest (half jokingly) that we should head over to the nearest retirement home and get a bus load of old folks to join us at $150/hr for two months to just sit there doing nothing but burn SQP.

We are always seeing threads with junior people/new grads talking about how they are having so much trouble breaking into the industry. Just hire them and give them interesting tiny projects. Would be super helpful to so many people to get going and still allow you to spend all your money.

You can't ramp up hiring that quickly. By the time you've onboarded several new devs, the year (and budget period) will be over.

What's hysterical to me is that Microsoft was actually making this point in their proposal because they very much expected for AWS to solely receive go-ahead with this contract.

As a former longtime contractor I totally agree. One thing often overlooked is this 'Silver Tsunami' that will wreak havoc on the old guard (existing giant contracting companies). The existing workforce is aging, the new blood is uninterested in the old ways (cruft, old tech) and frankly there's tremendous opportunity for a 'Space-X' style small contractor to get a serious foothold.

Some of these companies need to replace 40% of their workforce in the next decade. Who would a new grad choose?

> there's tremendous opportunity for a 'Space-X' style small contractor to get a serious foothold.

Sure, but the procurement process is incredibly difficult to crack into for small companies. There is tremendous amounts of red tape designed to keep other players out.

> Some of these companies need to replace 40% of their workforce in the next decade. Who would a new grad choose?

Most will go to where the money is...unless you go FAANG, it's harder to find the kind of high salaries that defense contractors can throw around in the private sector.

> Most will go to where the money is...unless you go FAANG, it's harder to find the kind of high salaries that defense contractors can throw around in the private sector.

There is a reason the DC suburbs of Northern VA and Southern MD have some of the richest counties in the US...

> There is tremendous amounts of red tape designed to keep other players out.

Call it what it is - bureaucracy, and corruption.

Is it really unusual or inefficient for the Department of Defense logistics chain to have lots of spare capacity all over the place w.r.t the Cisco chassis? Especially with global force projection, I'd expect there would be massive, unavoidable waste just to maintain operational capabilities, stretching into every corner of the industry.

> how did we buy 2 extra $50k Cisco chassis and then keep them in storage for 3 years

Military budgets... Spend it or lose it.

It's better you find spare cisco chassis, than hundreds of zip disks or obsolete laptops >.>

This was an accident, they meant to buy 2 blades for more SAN ports and they ordered the wrong parts.

They did this with a few database servers as well, they were supposed to order 8 core processors, but they ordered 10 core instead (it was a typo), they had 4 processors per machine it was a $40k extra mess not including the extra Oracle licenses needed to buy since Oracle licenses by cores (8 extra cores per machine and 3 machines).

But i do understand the spend it or lose it, which is another reason no one tries to be efficient.

My cubicle at my last job was surrounded by a mountain of expensive printers because they would loose some funding if they didn't spend all of their budget each year.

If we ordered a server it would show up in 6 to 12 months and sometimes would be sent to the wrong place. The people in charge of ordering would replace SSDs with spinning disk and we wouldn't know about the change until the wrong parts arrive months later. Fun times.

I understand that the existing approach results in a worse and more expensive product, but doesn't this approach also allow agencies to focus their efforts on assuring the vendors of their critical infrastructure aren't comprimised by bad actors etc...?

(This isn't my area expertise, so I'm open to the idea of being super wrong)

You can lock down YOUR infrastructure, but then are 100% dependent on the cloud environment to maintain and patch theirs. Since you have no insight to what the underlying infrastructure consists of you really have no way of knowing if they are secure. Do their storage arrays have open CVE's? are they employing people who are mentally sane? You just need to trust them.

So in the cloud just migrates the a lot of the security to another team. I do not know for a fact, but I am pretty sure the DoD cannot just show up at the AWS or Azure facilities and start auditing them (maybe they can and it is in a contract, someone else might know).

Considering AWS already has 2 entire airgapped dataceters for the US government, I'm pretty sure this contract will entail microsoft building entirely separate airgapped datacenters for the DOD and thus they probably will be able to just show up to their datacenter because nobody else is going to be running anything in those datacenters.

Correct, but those air gapped data centers already exist- there are separate Azure regions for certain restricted civilian government workloads, DoD unclassified work, and cleared work.

They are almost certainly getting their own privately built cloud which they can 100% show up and audit whenever they please.