Pentagon cancels $10B cloud contract that Amazon, Microsoft were fighting over
cnbc.com
cnbc.com
Linode can give us servers for probably 1/10th the cost of AWS or Azure if you're not going to actually let us use any managed services.
And people want to know why all defense projects are constantly behind schedule and over budget.
I don't know... that kind of sounds like a good idea to me. Vendor lock-in is serious, and we don't need a 50-year DoD drip-feed going to whatever cloud vendor wins the contest today.
An example of where I'm coming from: https://www.outsystems.com/blog/posts/vendor-lock-in/
I think past vendor lock in wasn't so much about the technologies involved, but the inability of organizations to transform. If you have the right approach to application development/deployment/management there isn't a way any vendor can lock you in for any significant amount of time. I see more people wasting resources developing to avoid lock in when the applications they are developing won't live long enough to ever worry about having to move off of the systems they are developed on.
I'm not saying we should never be concerned about vendor lock in, but I think the concerns need to be balanced against the costs (and complexity!) of engineering everything to be immune from "vendor lock in".
At the same time, the near-guarantee of the vendor that has the lock means they have far fewer incentives to be responsive to changing needs by the client.
In addition, given the lock-in, whenever 3 or 5 years maintenance agreement contracts near their end, the Vendor has significant leverage to demand more $$$ for pretty much the same level of service.
The overall result is a lower quality product that can eventually put the buyer years behind the current best-of-breed capabilities, all at a significantly higher cost.
Using standardised primitives, like the case with Kubernetes, leaves your infrastructure prepared to be migrated to a different cloud provider with much less friction and effort. You will still probably need some shims and/or abstractions for common patterns (caching, persistence, etc.) but if you can decouple completely your platform from the vendor's platform your organisation gets much more leverage, both for negotiation purposes as for regulations, in case there is a need to swap cloud providers your work is more-or-less cut out for you. When you didn't prepare for a cloud agnostic environment you will have years of migration to be done, for services, libraries and so on.
It's a major pain in the ass to be vendor locked-in and a huge cost, in time, opportunities and money.
The fundamental problem is that you aren't the one choosing or buying these services. The "people who give a crap" is some mix of political and industrial people with a very light sprinkling of high level program management which are driving the requirements.
So it has nothing - literally nothing - to do with what you as the actual person who is doing the work, want or need to be effective.
This is how the government acquisitions process is written into law unfortunately and I've pushed hard within the government to change that in the past (to some very limited success).
I thought AWS's Security & Compliance features[1] was the reason why it was selected before for such purposes. Do any of the low-cost alternatives offer similar security and compliance levels?
[1] https://docs.aws.amazon.com/whitepapers/latest/aws-overview/...
P.S., it's worth mentioning that it was the Manchurian who restored slavery in Qing Dynasty. The word Nucai in Chinese or 명사 in Korean, meaning Your Slave, was such an honorary title that for more than 300 years until early 20th century only those who were trusted by the royals could use. Yeah, don't wanna be a slave? Build a good army.
I suppose if you consider 1898 a few years ago.
Unlike AWS and Azure, Google doesn't offer any GovCloud regions, only Fedramp which seems to apply to unclassified data only.
AWS and Azure also have IL6 regions (able to hold secret information).
Finally AWS runs a top secret cloud for the intelligence community. I believe Azure announced a similar project but not when it’ll be open for business.
I've had demos from GCP sales reps, and the platform is shambolic. It doesn't even work during sales presentations.
Not sure where you’ve gotten this information from. GCP serves many “large organizations” including government agencies [0].
> I've had demos from GCP sales reps, and the platform is shambolic. It doesn't even work during sales presentations.
This feels very hyperbolic to me. Either way, demos don’t always work out the way you intend.
Also, it's evident they cannot service those customers. Australia Post has a large contract with AWS. Whoops.
I've worked in many large organizations. I've received GCP pitches. I've participated in evaluations of GCP as a technology solution. It's substantially worse than Azure, plus according to their leaked massive losses on the product it'll be shuttered this decade.
>This feels very hyperbolic to me
Their dashboards are typical Angular heavyweight clunk, with confusing error messages, infinite loading spinners and things break constantly. It's been discussed here extensively. [1]
Their actual decent products (BigQuery) are hamstrung by the rest of the failing platform.
Especially when you're Google (our customers aren't as smart as us) competing with Amazon.
As for anecdata, the only cloud platform I've heard nothing positive is Azure.
Except the most important part of Twitter -- timelines, of course. [1]
> Snapchat
Except the $1 billion contract from AWS, of course. [2]
[1] https://techcrunch.com/2020/12/15/twitter-taps-aws-for-its-l...
[2] https://fortune.com/2017/02/09/snap-inc-signs-big-aws-deal/
Multi cloud means cross cloud redundancy, but there's none of that here.
https://www.zdnet.com/article/google-heres-why-were-pulling-...
( Their offerings were so bad they were forced to sell their vSphere as a service arm to a low cost hosting provider. Even with the popularity of that dumpster fire in DCs and most companies moving away from DCs they still couldn't capture any market share)
Disclosure: long MSFT.
edit: re: downvotes - the article states this in the sixth paragraph
A couple of points. The whitehouse blocked any disclosures of discussions / pressure even to the inspector general. They also issued notices to staff not to talk to IG. That's a pretty darn good sign that there WAS involvement by the whitehouse.
The DoD is dropping this case because they were asked to reveal some of this information. Instead of disclosing it, they elect to cancel the contract. This was a specific issue in the current case, would DoD have to reveal this type of communication from whitehouse.
Yes - people claim no pressure / no involvement - but it's blindingly obvious that there is a very good chance there was pressure.
I have past experience with govt contracting so have a sense for what the flows look like.
Contract selection put on hold, involvement of a political person (probably esper here). Ban on anyone talking about the potential discussion to internal auditors. Those are all the red flags.
What? You're trying to pass off absolutely zero evidence as proof.
The IG review specifically noted all the ways that Trump / Whitehouse blocked the answer to that question.
The Whitehouse then also got rid of IG Fine in April 2020 - I mean, this guy served 28 years in justice then through bush / clinton / obama terms as an IG.
Amazon also dug up stuff where Trump promised to GET involved.
No one "cleared" trump in this. He stonewalled as long as he could.
https://www.nytimes.com/2021/04/28/technology/trump-jedi-pen...
This was AMZN thinking that all they needed to include in their proposal was market share numbers. MSFT's proposal was more responsive to the DoD's needs.
Can you summarize/quote the relevant parts of the article? What evidence did the court find?
“I will be asking them to look at it very closely to see what’s going on because I have had very few things where there’s been such complaining,” Trump said. "Not only complaining from the media — or at least asking questions about it from the media — but complaining from different companies like Microsoft and Oracle and IBM. Great companies are complaining about it. So we’re going to take a look at it. We’ll take a very strong look at it. Thank you very much everybody.”
He head already shown at this point that he was an old boys club type of guy that would go after anyone who disagreed and support anyone who kissed his ass, so anyone having a hand in the selection of the winner of the contract that wanted favor with the president would obviously attempt to influence the outcome to not be Amazon.
He should have kept his mouth shut on such a massive pending contract, but it's pretty obvious by now that keeping his mouth shut isn't something he's physically capable of doing.
Now instead of having started on a crucial service that is needed by our military we're looking at likely further years of delays. All because the president was butthurt over some mean words in a newspaper.
That doesn't matter. We're not talking about criminal proceedings, where anyone has to be proven guilty beyond reasonable doubt. This is, at most, a civil law process. And it's the DoD canceling its own contract, which they are probably allowed to do for whatever reason.
Trump administration had asserted a “presidential communications privilege.” when asked if they leaned on the DoD to deselect Amazon.
Pentagon lawyers instructed Defense officials not to talk with the IG about any discussions they may have had with the White House about JEDI.
So basically - you know for SURE that there were communications about this.
Her last program held a voting pool to pick a name and ended up with "Totally Not a Laser Death Ray Facility" unfortunately coming in 2nd place.
"Star Wars" itself was the name of a previous defense program, so, I'm sure this debate was had out decades ago.
1. https://en.wikipedia.org/wiki/Strategic_Defense_Initiative#C...
https://www.copyright.gov/circs/circ33.pdf
You're probably thinking about trademarks, which only protect the use of an identifier when used in the business of a particular trade.
I mentioned in another comment that George Lucas tried to sue over people calling the SDI "Star Wars" back in 1985. I can't find any evidence now, but I believe he once sued either the Air Force or NRO (can't remember which) over a mission patch with an x-wing on it.
They’re cancelling it because they don’t need it anymore. They need something like it, but the requirements have become archaic.
The fact that this was kept up in litigation until the technology became archaic is really, really disturbing.
And also, in a way, really really good. It gave them time to know what they actually need and not waste money on something they (incorrectly) thought they needed.
Even in Defense contracting land, If you delay a contract by 5 years then nobody from the original proposal will still be around. Had the contract originally gone through there would probably be incremental development to keep it up with the times.
"Mr. Johnson, I don't give a damn whether your plane ever gets built. But my forms will be signed and procedures will be followed."
He found the bullshit tiring because he felt that he didn't deserve to deal with it, because he trusted in his own competency and honesty.
You can see how all the pieces fall together. Somehow, the government has reached a default stance of not really trusting in the honesty (and maybe competence) of their contractors. And frankly, Ben Rich didn't really believe in his competitors either. He just didn't care because he only writes about projects that he's also competing in, so he thinks that his competence will carry him through (which it generally did... not for the F-16 though lol). What about all the competitions that doesn't have the strangely competence and honest vendor?
DoD saw what a gong-show single source could be (look at F-35). They figure multi-vendor, open architecture is the way to mitigate that risk.
In a DoD facility? pretty hard
Do they not have computers in DoD facilities? Do they never run RHEL? Don't the installer disks have QEMU on them already?
This would be far better for society. Compute-for-all.
If we were serious about fixing this problem, existing federal pay scale constraints would not be a barrier to entry.
Legislation could be passed yesterday that would add arbitrary pay scales for special purposes such as these. Attach an AWS architect salary to a government job and I think you will immediately find the skill gap filled. Bonus points if you put incentive structures in the employee contracts so that the brilliant minds are directly incentivized to deliver, rather than via proxy of their lobbying container organization.
Or maybe add real progressive taxation that would get the $500k comps back into normal territory (by, among other things, taxing dividends, share options and the like accordingly).
Tough cookie!
If you can't do that, well, then you better consider secession or something. I don't want to live out my old age in a "New America, Yukon" military junta rump state because some people sit down, blushing, seeing Rubicons on all sides in 2021.
(The sad irony of course being that crossing Rubicons made military Rubicons then, I know. But it was a lack of meaningful other reforms too. So find a Gracchi metaphor instead, I guess.)
From there, we can give the agency the budget to do the job, and let someone who's actually close to the situation decide whether it makes more sense to contract it out or build an in-person team. A bunch of legislators who are well-known to be perpetually too busy conducting fundraising lunches with lobbyists to actually read the text of the bills they're crafting and voting on are never going to be making informed decisions on matters at this level of detail.
That said, appropriations is no barrier to hiring the necessary talent. There are plenty of well-run private data centers in the federal government, staffed by both federal employees and government contractors.
Also, it's my understanding that the micro-management is more a problem on the state and local level, like NY state politicians such as Cuomo forcing the MTA to take on debt for these stupid stations rather than improve service.
I'll match you against somebody willing to take $150k for the exact same work. I'll match 4 of you against 8 of my hires. I'm willing to bet that my $1.2M team will outperform your $2M team. I'll very happily match 50 of your team against 75 of mine. You could poach 25 of mine who prove to be superstars, and I'll hire more.
That doesn't mean that you are dumb or incompetent or less skilled. I just think that there are some very smart and capable people willing to work for $150k. I think that many of them are as good as any AWS hire. And I think they could be motivated by something besides money to create something as good or better than anything from AWS. Indeed, less money could itself be a strong motivation to outperform.
Thankfully, we don't live in a world where there are only 10 geniuses to go around. The world is much much cooler than that!
I'd take ONE top faang engineer over 8 government software contractors. There's so much tribal knowledge out there that people in the Midwest just don't have, not to mention a can-do attitude.
It blew my mind the first time I worked with a guy from Microsoft on something. We were having issues with some code, and he just popped open the kernel and started actively debugging things that I'd only vaguely even heard about. I feel like I was twice as good at engineering after that experience alone.
I had the same experience from the other direction later on.. a team of smart, hard-working coders had been dealing with stability/perf issues for years on some product. I was able to root cause and straighten all of them out in a couple of weeks, even though this was in an area of software development using a set of tools I'd never touched before.
I'll bet a single FAANG-hardened code wizard could outperform 20 or 30 government coders.
the hardest-core thing I've done was step my way through Windows startup into the container subsystem, on a real, physical target machine I had connected to my dev machine over FireWire. I felt like Indiana Jones. (It helped having the source code though!)
Don't be afraid to dig through heap dumps or decode assembly if you have to, although when you hit that point you're probably just going to have to swap out whatever component is misbehaving
I've done this at a company full of ex-Googlers. Some programmers are simply incompetent, no matter where they've worked.
The reason is not talent, but process. Governments are bureaucratic by nature, companies are meritocratic by nature. Obviously this is a vast simplification and there are exceptions but I wouldn’t bet on your team being one.
Although I've never worked in public sector so I can't make a cross comparison.
Wasting 500k on an expensive school that pretends you're going to be an engineer, then learning how to reverse a tree on a blackboard so you can be hired at Google or Amazon doesn't make you a code wizard. That's a very expensive code monkey.
If you think people's claims that they aren't motivated by money are anything other than lies they've learned they're expected to tell, perhaps you're just as ignorant as your $150K hires who have no idea that FAANG companies pay their developers $500K or more (which is the only reason you are able to hire them).
To your second, well, /shrug. A reasonable number of folk might well jump at the chance to have a job like that on their resume, and there's likely a fair amount of possible cross pollination w/ various folk in the DoD/military to bring up bodies and relevant expertise (i.e. security).
You still have to pass a marijuana drug test to work there, for heaven's sake.
I hate to break it to you, but that's normal in the vast majority of companies. So far, I've only lived in two states where it was legal, and in both the laws specifically allowed employers to test for pot. I'm in healthcare, and the company I work for tests everyone on entry, and people randomly after that.
If it's ever legalized at the federal level, expect employer testing to be specifically allowed. Transportation companies, surgeries, heavy industry, and a lot of other companies will filter for this, either because they or their insurance companies don't want the risk.
We're talking tech here—the Digital Service—and I've never worked at a company that drug tested their programmers. Not once, since the 90s, during the full-swing War on Drugs.
Anyway, the marijuana test isn't the only antiquated, sclerotic thing about the USDS (and Federal employment in general). But it's certainly a representative example.
What security engineer worth his salt is going to accept half the pay to work for the government where things move at a snails pace?
Problems of red tape are not limited to only government ,it is function of most large organizations
Plenty of security engineers are constantly frustrated at the lack of security focus in corporate sector, investments are only made when it affects revenue . Even after all the ransomware companies some times do the bare minimum and buying insurance
Government on the other hand cares about security for different reasons, they take it lot more seriously. They don't sacrifice security in the interest of sales.
I would anyday prefer to work for some org where security actually matters not where it is inconvenience
https://www.nytimes.com/2020/12/17/us/politics/russia-cyber-...
Government doesn't care about the security of your personal information, hence why they treat their security engineers with contempt by offering them paltry wages and drowning them in red tape.
Most companies I know treat Security engineers similarly and pay not all that much higher. There is world of enterprise IT outside the silicon valley, the pay is not all the much higher. Despite all the recent threats and ransomware attacks security is not treated as it should be.
Windows still dominates most orgs, but never thought I would live to see any significant numbers - but here we are.
Is that how people approach it? Yes some do, and there are fantastic people who are working the government in service to the nation. Certainly not all and almost never for government contractors.
That's the spirit at least. I don't think it's actually working unfortunately because trust in government is so low. There was a time though...
Compensation is only one aspect of a job. Plenty of people work in defense while private sector pays more, public service motivation, job security, benefits, better work hours, more seniority driven promotions there are a ton of reasons.
Doesn't mean it is the most efficient way to run an org, point is there are enough qualified people who value the other benefits more than just the cash.
Funny you mention 170k Amazon is known to have a fairly strong cap on base salary compensation. Usually at 150-170k range.
Also, the fact that government jobs are so capped like that is a conspiracy to make the public sector suck, basically. We should fix that too, but I recognize it cannot be done in time, hence allowing that the design work to be outsourced.
Or in another words, you're trying to find the one honest person, in the stack of AccgemloitteBMizantPMGataBCS proposals. And everyone else is bidding with the profit expectation of tossing entry-level, over-hyped developers on it, and then crossing their fingers nothing goes wrong.
hahaha
-----
I am trying to structure the contract to avoid this stuff, by making the deliverables include both the contractors running their own working deployment, and the government also running their own. For the latter, there should be a tech transfer period where the contracter needs to do training and promptly answer questions, and then a government independence period where the contractor is not allowed to intervene at all to prove the knowledge has actually been transferred, and government isn't entering an IT-support protection racket. Only then is full payout given.
Yes, that means finishing the entire project will take quite a bit longer than doing all the dev work. Yes, that also means the project is riskier for the contractor and has less ongoing reward, and so they will need to be paid more initially. But it's worth it in the end.
If there is some gotcha here where the private sector can get out with too much proprietary IP or a weak and dependent customer in the government, do let me know, that is not my intent. I'm trying to envision a transitional project to get out of today's privatized world to one with better US state capacity.
> This would be far better for society. Compute-for-all.
Yeah, but that could also be the perfect being the enemy of the good. IIRC, there have been many, many massive failed military IT projects (e.g. they've spent decades trying and failing to replace legacy accounting systems). If they tried to build their own cloud, a very real possibility is that billions of dollars are spent creating an open source cloud platform that literally no one wants to use.
Also, my employer actually did implement their own private cloud using open source in our own datacenters, but that's been abandoned in favor of AWS and Azure. If the military builds it, it doesn't mean anyone else will come.
Good in this case means meeting most or all of the project objectives. If they pursue a "perfect" solution, they may not do even that, let alone realize the other social goods you're hoping for.
Pithily paraphrased, "Those who would give up essential [self-sufficiency], to purchase a little temporary [efficiency], deserve neither".
From the Pentagon's perspective, they're probably meeting that with either AWS or Azure. They probably look at self-sufficiency as a national, not organizational thing. So long as the thing is built and located in the US and owned and operated by Americans, it means they're not giving up any self-sufficiency.
https://cloud.gov/docs/technology/iaas/#the-infrastructure-u...
So, no I don't think it really fits the bill because underneath it's using a solution built by a private business.
I guess its "cloud.gov" vs "cloud.mil".
Cloud space is already commoditized through AWS, Azure, GCP, Oracle and other smaller players right? Is this more of a NIH thing?
> This would be far better for society. Compute-for-all.
Can you expand on why/how would this be better for society than what we have available today?
(My coworker made an astute prediction (only semi-jokingly) that eventually all the standardized binary interfaces would eventually transitively refer to all the other standardized binary interfaces as the inevitable conclusion of that phenomenon, and few stingy large orgs properly separating their external interfaces from internals.)
The solution here is to pay a specific contractor to do the design and validate with a demo deployment. Oncely once it is proven to work, and the government can then set up their own in-house (i.e. proove it is real open source by doing the tech transfer) is the full money paid out.
This is the computer equivalent of a drug bounty, basically. It's high time publicly-funded engineering doesn't just deepen private intellectual-property motes.
I was under the impression that not having to build and operate their own datacenters was the idea behind outsourcing cloud. Since commercial players are already doing it, the government could piggy-back on the economies of scale already being made. Plus, they can decide not to renew their contracts and not be stuck with a datacenter on the balance sheet.
It would make much more sense to let Microsoft, Amazon or Google to span a new data center and let the government wrap it entirely with their own monitoring and possibly also a specialized access solution.
That way they leverage all the hard work which these US companies have already made, all their expertise, and are able to keep an eye on all of it.
Easy money! If they don't mind their mote being filled, that is...
It'd be like the government designing their own ASICs or making their own OS.
But the part about avoiding vendor lock-in would seem like an appropriate measure.
To avoid some "idiots at the wheel situation", part of the tech transfer exercise will ensure that the deployment isn't too brittle. E.g. the government should be able to make some trivial Kernel modification and redeploy everything without issue. Maybe even also fab and install a slightly modified ASIC too.
Our government is not capable of efficiently building and managing AWS style infra. Or designing new ASICS for consumer products.
Helping with R&D? Strategic planning? Sure. But not doing stuff like that at scale.
Like say, gov.uk, or a large number of public services in Australia (both state and federal). I've dealt with our equivalent of DMV and it was effective, efficient and there wasn't the stupidity of queueing that I've seen in US discussions about the DMV.
The US is one of the few advanced economies where there is a deliberate effort to screw with the government (eg defunding IRS auditing) and then complaints that government can't deliver services, followed by corrupt contracts to private enterprise that deliver worse results for higher costs.
Sure, most governments and bureaucracies aren't the most efficient, by their nature, they have rules and processes imposed on them. But that's more of a function of the "management" than of the services themselves.
Edit: DARPA -> Pentagon
> Shares of Microsoft were down about 0.4% following the news
So the MSFT share price changed well within the norms of daily volatility. It doesn't mean anything.
FWIW the stock is now back at what it was before the news broke.
*unless he is trying to launch rockets, which somehow he catastrophically sucks at
I admire the faith he demonstrates in his team, by being among the first human lives at risk in a launched Blue Origin vehicle.
But maybe 57 is just the new 27.
It is still not easy , but as difficult as say 20 years back.
To an extent NASA's Commercial Crew program and other commercial engagement vision worked in triggering commercial industry
I think it more that technology has changing a lot making it less expensive to attempt building a vehicle. Today, for example Terran is considering building a lot of their rocket with 3D printing, Rocket Labs does that already for their rocket parts. Talent availability with the right skills has become easier in the recent years probably helps.
> The AWS Top Secret Region was launched three years ago as the first air-gapped commercial cloud and customers across the U.S. Intelligence Community have made it a resounding success.
AWS has provided air-gapped regions to the US Government for 6+ years, according to that blog post.
But presumably Internet connectivity is the entire point of why they'd want a cloud contract: The government already has servers in buildings. It sounds like they particularly want the global reach of a cloud service, probably for activities intentionally intended to be on the Internet.
I mean, bear in mind, if you're a state-level actor that engages in cyberattacks, you need to be on the Internet to do them... Not saying that's the purpose of the JEDI contract, just that... there's plenty of scenarios where the military wants Internet-connected things.
On the other hand, let's be honest... In strategy for a modern war, Microsoft and Amazon's major datacenters are already on someone's priority target list.
Every nation-state actor knows where every datacenter is of every major digital player, because they've already implanted spies in those companies and the location maps of those datacenters are not internally secret. It'd be very hard to make them internally secret, since those DCs communicate with the rest of the network and must be controlled in realtime by site reliability engineers.
The DoD's job is defense, so it's about fighting wars, not about building infrastructure and supplies.
The security of the military apparatus will stay in the DoD. The implementation and operation of the security infrastructure will be delivered by government contract.