Someone Is Waging a Secret War to Undermine the Pentagon’s Huge Cloud Contract | Hacker News Reader

Someone Is Waging a Secret War to Undermine the Pentagon’s Huge Cloud Contract

defenseone.com

198 pointstheDoug90 comments

Well... To be fair, it's an AWS spec sheet reworded as an RFP. Other cloud providers - even Google and Microsoft - can't meet the requirements, because they don't have the same marketing jargon. For example, there's a requirement in the RFP for a "ruggedized" storage product. That's a made up word that Amazon uses to describe AWS Snowball, so no other company satisfies the requirement.

Some other AWS-only requirements:

* exactly 32gb of RAM. Google promises 30, Azure promises 28. We all know how realistic this isn't. * 150 miles between data centers. This arbitrary number is exactly what Amazon advertises. Others can manage 100 miles, or 130, but this number comes right out of Amazon's advertising package.

Not to mention the idiotic idea of a single provider solution.

On the other hand, this is a 10 year contract worth about 1% of Amazon, Google, or Microsoft's bottom line in one year. So there's a bit of mountain/molehill going on here

Isn’t this commonplace in government contracting? You have a preferred vendor already, but must show the appearance of an open, fair bid, so you put your RFP out there—but is made as specific as possible, matching your preferred vendor’s product spec word-for-word, so only they can meet the requirements. This is kind of standard, run of the mill corruption that I thought was well known.

The folks downvoting you have obviously never been in the rooms when RFPs are drafted. I've been in the room half a dozen times and in almost every case the RFP is written so one particular firm will win the bid, and at most 1-2 other firms are even qualified to respond.

Edit: Once while sitting on the private sector side of the table (yes RFPs were often drafted in consultation with the company that would eventually win them, if just informally), a [state] government employee expressed concern over the wording of a particular requirement because he was worried that it would open the door to our primary competitor being qualified to fulfill the bid.

I’ve always recommended that software folks do a short tour of duty at a government contractor, purely for perspective. Its amazing to just watch how that sausage is made and why it’s made. You’ll come away with a renewed appreciation for working in the private sector on products that are made for people.

Having worked in the sector in the past, I can’t imagine anything I’ve gained for my time in there besides bitterness and cynicism for business, and this attitude has had only negative impact upon my career. On the other hand, I see a lot of parallels with the private sector, specifically in enterprise software. B2C software has its own awfulness but at least it’s a bit sexier than the insane Rube Goldberg Machine of, say, Payroll software. These days, I don’t really see a difference between an IBM, HP and Lockheed, Northrop - they oftentimes compete for the same contracts even.

I totally understand that. I think it depends on where you work though. I've worked for a couple gov contractors and it has it's upsides. It always seems to be effectively hourly work with a 40hr max work week. Projects can be (if you push for them) short and frequently varied which keeps things fresh and lets you learn/maintain a lot of different skills. The main downside IMO is that the DFARS are nuts but with the security environment we're in, their purpose is at least understandable if not their implementation.

Been there, done that. I feel much worse about paying my federal taxes now. Federal contractor (to whom we were a subcontractor) spent at least five times what they should have, and very little of it trickled down to us. They also were a “broken telephone” between us and the end customer, so we didn’t get much user feedback. Any call (not _video_ call mind you, a phone call) with them had a minimum of 10 people, and after the call everyone on their end would have immediate amnesia. 1/10 would not recommend.

Stop me when you’ve seen this one: Prime contractor is a one or two-person shop. They don’t do any work, but subcontract the entire contract to the huge company that is doing the actual work. The prime contractor’s job is to be an Alaskan Native American-owned or Female Amputee-owned or some other underrepresented minority-owned business so the government can say they work with minority-owned businesses. The prime does nothing but pass communications back and forth and skim $$$.

Mind blown—have a contract with one of these currently.

Ours was very large. In fact, I'd say, a 2-person do-nothing prime would be a vast improvement over what we had, especially if they did not have any illusions about their own level of competence. To see the situation in the positive light, however, I have vastly improved my project management skills. Making _anything_ stick on the other end was a non-trivial endeavor.

This applies just as much to private sector RFPs too. If you're a vendor and you get an RFP out of the blue, you're not going to win it.

RFI process is usually far more open.

My first job in the '80s was doing RFI/RFP shotgunning for a mom & pop (my mom & pop) defense consulting firm out of the GSA register. We would essentially cold-respond to everything in our domain. From that experience I learned that there were some genuine requests with no fix in.

There definitely are! However if it's an established contract (most of my experience) or a big contract, the odds of it going to an unknown entity are slim to none. But it's certainly possible to break into it as a small unknown firm, you just have to start by picking up the scraps.

This is really interesting and I would be very keen to hear more about it, like a blog post or something. I bet a lot of other people would be interested too.

Interesting. My experience is all commercial, not government. Pleased to hear there was some good behavior (at least in defense in the '80s!)

I was going to say this too. Especially if you are huge enterprise.

It's not even necessarily corruption. The company they're wiring the bid for may indeed be the best option, they are just trying to minimize inefficiency.

Having said that, just as often, it's due to cronyism.

> Having said that, just as often, it's due to cronyism.

The line between cronymism and efficiency is in the eye of the beholder.

If you look at some of the protests that have been filed, the protesters are often worse than the awardee.

Some corporations are more equal than others.

That is right. Not sure if it's called corruption tho. No one is getting paid on the side.

The practice is called "single-sourcing".

It's pretty clearly an example of corruption; the entire purpose of single-sourcing a contract is to defeat the RFP, or bidding-out, process. A decision is being made by someone who isn't supposed to have the authority to make it.

To turn that around, RFPs themselves are an example of populism, where a meta-decision (the algorithm to choose the winner) has been drafted to play well to voters' ears, rather than to actually accomplish anything. The decision-making process has been corrupted by political pandering.

"Defeating an RFP", then, is frequently (behind closed doors) agreed to be in the best interest of good government. A chosen expert—despite the bad optics—can do better (usually much better) than the RFP process.

I'm not sure "corruption" is the right word for this, since the goal here isn't to line anyone's pockets, but actually to choose the consequentially optimal, highest-ROI option. It's not quite "two wrongs making a right"... maybe "three lefts making a right?"

Do you drive the cheapest contraption that could technically haul you from A to B?

Did you type this on the cheapest thing capable of making HTTP requests?

Do you live in the cheapest structure capable of maintaining its interior at a non-life-threatening temperature?

I submit that lowest-bidder rules are patently absurd. People who have to deal with something day in and day out ought to have the authority to select a reasonable one.

If lowest-bidder rules are patently absurd, how does that stop bypassing them from being "corruption"? You have a job and you purposefully cheat at it by doing something you're explicitly prohibited from doing. It's just as corrupt whether you think the results are good or bad.

The contract is not "the job," but a means to some other end. Corruption is not a synonym for rule-breaking. It implies a subversion of legitimate purpose, serving the wrong master, prioritizing the wrong interest.

Rigging a bid in a way that compromises the real task at hand in order to personally enrich yourself is clearly an example of corruption. Rigging a bid in (what you think is) the interest of the task at hand, but in violation of contracting rules, is rule-breaking and insubordination but not corruption.

at best it's subversion of a policy that is intended to enforce some amount of competition.

It certainly is that.

This is very very common and happens in every big company and in every government around the world, even so call "not corrupt" countries. For big contracts and for small contracts.

1% of Amazon/Google/Microsoft's bottom line is a metric shit-ton of money.

It's also, more importantly, opening the door to new markets like banking, insurance and state/local government. Many compute intensive workloads in state/local government remain on-prem beacuse of the lack of Federal guidance, which means higher compliance risk.

Silicon Valley drives innovation, but the federal government drives implementation of technology.

Google lets you pick an arbitrary amount of RAM and number of vCPU's when creating an instance. You can easily dial up 32GB of RAM. (Or 37.25 gb of RAM if you feel like it that day.)

https://cloud.google.com/compute/docs/instances/creating-ins...

> On the other hand, this is a 10 year contract worth about 1% of Amazon, Google, or Microsoft's bottom line in one year. So there's a bit of mountain/molehill going on here.

a) Have you ever known an IT project, let alone a government IT project, to end up smaller than or equal to the projected size/cost? Or anything near it?

b) If you ever have the “pleasure” of dealing with government IT, you'll see how much easier it is to use a combination of things that has already been “certified” by another part of government. Winning a contract worth 1% of your company's bottom line is huge itself, but as a foot in the door, it's massive.

Yes, I assumed those were part of this corruption document being shopped around that's discussed in the article.

Your points are all the ones raised in that... wapo? nyt? article that you're alluding to, which was discussed here and found to be largely nonsense, IMO. For example, that 28/30/32GB thing is wrong. Are you really saying GCP and Azure don't offer 32GB options? That's a totally standard and common configuration these days.

This isn't accidental. Despite all the rules about fair awarding of contracts it's mostly ceremony: the government agents who draft it have a contractor or small set of them in mind when they draft the proposal. It's the same thing that happens when a hiring manager wants to bring a friend in: they'll write the job description such that only their friend qualifies, while wording it in such a way as to give some cover.

Ruggedized is a real word, though. Googling it at the moment gets you definitions and links to laptops, tents, hardware firewalls, etc.

This looks like a regurgitation of a Vanity Fair article that I found a bit flimsy.

The idea for example that MS are effectively excluded from bidding because they can't achieve the feat of putting 32GB of memory in a server is a bit absurd when you think about it. For a big contract like this I reckon MS techs could probably just about manage it.

Have the people drafting the requirements done a bit of copy-n-pasting? Definitely.

> Not to mention the idiotic idea of a single provider solution.

I'm surprised this isn't being discussed more. With a huge $10bil cloud I would think you would want to select a secondary vendor to distribute redundant workloads and data storage. Even if Azure or GCP couldn't meet some requirements today it's a ten-year project and both are expanding quickly. Azure has recently brought online government-specific locations and Google has already has geographically dispersed zones within regions that meet these requirements.

The Pentagon already has programs like MilCloud in place for redundancy. JEDI is far from the Pentagon's only cloud program.

Btw, Google is years away from getting the kind of security certifications you'd need to run a cloud from the Pentagon. Oh, and the Project Maven thing.

You're just restating points from the Vanity Fair article, which was sourced almost exclusively from the dossier. In reality, "ruggedized" is a fairly common DOD term for battlefield IT. The 32g RAM is mostly arbitrary and others, including Microsoft can deliver it. The single provider solution makes sense when you're trying to centralize a fragmented cloud environment.

the concerns you raise are so arbitrary and copied word for word from the vanity fair article that was published a few days ago. FWIW Google, and Microsoft have datacenters well more then 150 miles apart. Also ruggedized is a term used all over the place in military spec's I mean its has its own wikipedia entry

https://en.wikipedia.org/wiki/Rugged_computer

This is the kind of astroturfing that should get your account flagged.

When the USG writes an RFP/BAA, they already know who they want as the provider.

I looked at the RFP itself and I disagree with these arguments: https://news.ycombinator.com/item?id=17754900

> "ruggedized" ... [is] a made up word that Amazon uses

I'm pretty sure "ruggedized" is a common term for pseudo-milspec computer hardware. You see it on the marketing of regular ol' external USB hard disks quite a lot. If Microsoft and/or Google had a product of the same type, they'd very likely use the same adjective to describe it.

AFAICT the above is from the reporting that DefenseOne is criticizing; that doesn't make it false, but it means we shouldn't take it at face value.

It was discussed recently on HN and, IMHO, mostly discredited. For example, the RAM amounts are wrong for Google and Azure, and certainly they have 150 miles between data centers.

https://news.ycombinator.com/item?id=17754079

It's basically an attempt to sole-source a contract while pretending to be an open competition.

Having said that, AWS may realistically be the only provider that can meet their needs.

The 32GB of RAM is silly, because virtual machines, and PCI buses lol.

But I suspect the 150 mile marker has some other objective driving factor behind the reasoning for that number, which Amazon incidentally has some advanced insight on, due to other commitments.

It probably draws directly from line-of-sight targeting at visible horizons for some sort of specific altitude. Zero altitude, mean sea level gets you maybe a thirty mile shot with conventional artillery, targeting a multi-story building. But then again, rail guns seem to be capable of sending shells or slugs over 200 miles [0], so who knows...

[0] https://www.cnet.com/news/futuristic-navy-railgun-with-220-m...

It's Oracle. Safra is super close to the administration [1]. The only reason this contract is out of the hands of Booz Allen et al is Oracle. This might be the worst-kept tech secret in D.C.

[1] https://www.bloomberg.com/news/articles/2017-04-12/trump-tea...

There was an article in The Register a couple weeks ago about a pension fund suing Oracle over their excessive cloud hype, and what sounds like outright threats to punish customers who don't try it out. From the piece:

The complaint alleged Oracle has been coercing customers to adopt its cloud services, to inflate revenue and hide market disinterest.

"In truth, Oracle drove sales of cloud products using threats and extortive tactics," the complaint, filed on Friday, stated. "The use of such tactics concealed the lack of real demand for Oracle’s cloud services, making the growth unsustainable (and ultimately driving away customers)."

https://www.theregister.co.uk/2018/08/13/oracle_sued_cloud_n...

I read something similar on a German IT news site a while back. It claimed that Oracle's sales people pressured customers into buying cloud services, and if the customers did not "comply", they were threatened with a license audit that apparently included suspending the customers' licenses until the audit was completed. And if your ERP system is supplied by Oracle, or the database your ERP system runs on (or your financial accounting package), that means you can basically shut down your company until the audit is complete.

That's when you go along with their demands for just as long as it takes to get completely off their products/systems all together.

...and the Oracle sales folk won't care - as long as they meet their (current) targets. Be very careful about the behaviors you incentivize.

Oracle is a cancer. Their hostility to their own customers is inexcusable. My company recently suffered through an Oracle audit, and we're being basically extorted to buy Oracle cloud products to avoid a lawsuit.

Their corporate culture is just rotten from the top down. Their sales reps' word is worthless, and the attitude is not "How can I help the customer," but "What's my leverage?" Sales people are trained not to find the right fit for their customers, but to exploit their ignorance. They take advantage of customers using opaque terms of service, bullying tactics, and outright deception. It's like they got their business ethics from watching Goodfellas and The Sopranos.

Oracle is currently buying out entire DC Metro stations with ad campaigns that consist of the words "ORACLE CLOUD" in white text on a bright red background, so I would imagine this is all part of one big ham-fisted strategy.

Or course it's Oracle. Given how much of the rest of the US government they have by the short hairs, they're desperately trying to legitimize their last-horse entry into cloud computing by leveraging existing relationships as hard as possible.

I don't need a 100-page dossier to see that, even from here in the midwest.

I can't believe Oracle would do something so underhanded and unethical. Oh, wait. Yes, I can believe that.

Oracle was my first guess too, but this isn't exactly proof either.

Booze Allen and Oracle should both be banned from government contracts like this. Considering their histories they are clearly inept.

Is your reference behind a paywall ?

You could always click it and find out.

What are you insinuating? Booz Allen isn't even a cloud provider at all. They will likely inevitably be involved in the rollout regardless of who the gov goes with, AWS, GCP, Azure etc. but I'm not sure what your point is.

I'm also genuinely confused by Oracle being pulled into this, are their execs really so deluded that they think they could actually win this contract or is this just a wild guess?

So you were hoping that Booz Allen Hamilton would have some kind of role in helping the gov't procure this important service? Have you ever met them?

> you were hoping that Booz Allen Hamilton would have some kind of role

I have no horse in this race. Not sure how you got that impression.

I have. They're awful

Looking at Oracle's website, after getting past the hilarious Cookies Policy modal prompt, you really wouldn't think they were genuinely in the cloud business. It just looks like sales fodder with no substance behind it.

They aren't "genuinely in the cloud business". They are a vampiric ancient giant slug that lumbers on by drinking the blood of other ancient giant slugs. They have zero interest in marketing cloud infrastructure to, say, startups.

The lamest possible version of warring elder gods. I love it.

https://goo.gl/MZFbQ9

Manager, Startup Ecosystem Business Development, Oracle San Fransisco

Govt contracts have their own unique sales process and having a detailed webpage is not relevant

They are very much in the cloud business. Oracle salesfolk are essentially not compensated for selling things other than cloud services.

I’ve been getting what I assume are geotargeted ads for this http://monopolyamazon.com/ for months on twitter (I work a block from the White House). For example https://twitter.com/setonmotley/status/981262337905602560?s=...

Me too, every day since April. That sort of Twitter campaign costs tens of thousands of dollars. My bet is Motley and the dossier are related.

I've gone to monopolyamazon.com, then proceed to http://lessgovernment.org link in the page bottom where I was served a banner calling Net neutrality a solution looking for a problem. This website have very strange affiliations.

Also questioning both climate change and the harm of second-hand smoking. There's a whole constellation of right-wing think tanks that raise money from Exxon, Koch brothers, etc and then lobby "against regulation" in a variety of topics of interest.

Lost me at the first line: "Jeff Bezos is one of President Donald Trump’s biggest critics."

How do Jeff's political opinions affect AWS' suitability as a cloud service provider in any way?

It doesn't, but this is the US we're talking about. Thinge get shady here.

Trump is the President and theoretically can call the shots at the Pentagon if he wants to. Bezos is the CEO of Amazon, which, despite having the most popular service, is a company that Trump doesn't like.

Oracle has spent probably billions at this point over the decades lobbying Washington. Safra is working very hard to pull strings to keep this contract in the good ol' boy network, considering the hundreds of millions of dollars in revenue each and every major branch of government spends on Oracle stuff every year.

It's also not true. Bezos has said very little about Trump. Trump is mostly pissed because he owns WaPo which reports on his misdeeds.

Lots of people on that thread who seem to think that it's only Trump supporters and/or Russian spies who would be concerned about Amazon's dominance and monopoly power.

There is another side of this, which is workforce training and capabilities. As much as AWS / Azure / GCP are alike, their services and behaviors are very different at a detailed level of overview, requiring separate tooling and best practices. I could see someone wanting to procure a specific cloud vendor as their toolsets already support this.

I am fairly sure the 'secret' is One Rich A Called Larry Ellisson. Quite possibly in a knock-off suit provided by Hammer Industries...

You forgot the -sshole.

It's off-topic, but this is what succesful censorship looks like: when you get to the point that people voluntarily self-censor (even on a pseudonymous network), the mission is accomplished.

I don't know about you but I don't want to come to HN and read a bunch of comments of people cursing. There's a difference between self-censorship and just being somewhat professional. I rather like the idea that people here will actually construct a coherent comment instead of just calling other people assholes.

>SBD helped defense industry clients navigate the Pentagon bureaucracy and hone their marketing messages to government buyers, according to company spokesman Price Floyd, who was the assistant defense secretary for public affairs when Donnelly was on Mullen’s staff.

at least lobbyists are supposed to register. "Navigation and honing" in government contracting seems to be completely free wheeling.

> at least lobbyists are supposed to register

I read this with a tone of defeat. :(

The lobbying situation is very bad, it is just corruption with extra steps. And its effects are the same one that corruption. Money gets diverted from public utility towards private hands. Cut lobbying and this "Navigation and honing" is also going to be reduced. When a behaviour is normalized it replicates at all levels.

I suspect the winner will end up being the company with the least morals.

Google is likely already out of the running; between Maven and Dragonfly, the Pentagon has to be aware that there's no commitment they can make that doesn't come with an "if our employees don't revolt over it" asterisk. Given the questionable crud our defense department gets into, they'd probably want to avoid that condition at all expenses.

Microsoft probably wouldn't be too far behind on that train, I've seen some Microsoft folks upset about some their projects too. Not sure about Amazon, have we heard about upset Amazon employees on a cloud project yet? Oracle has never been too ashamed to play the villain, so I'm sure they'd be willing to do whatever the Pentagon wants.

And as a sidebar, single-sourcing all cloud services for the department of defense is still a dumb idea.

Government contracting is a bizarre place. Eg it’s the only place where an incumbent can sue the client (government) to delay award of a new order to a competitor, while still being paid millions of $$ to continue working for the same client.

Oracle's secret/top secret cloud is completely stealthy. You can't even see it.

Is this not a repost? Quick search for the magic word 'ruggedized' turns this up from last week:

https://news.ycombinator.com/item?id=17754079

If there is a "secret war," it's to undermine the corruption of the process. The contract is written, as most preferred vendor contracts are, in such a way that at most AWS would have 1-2 competitors which would still not meet the full spec...so AWS (the preferred provider) wins before bidding even begins.

Government does this all the time, it's a form of "capture" within the system.

Unfortunately Trump has given Bezos cover on a lot of things, simply by criticizing him. Because now if AWS loses the contract people will say Trump intervened or effected the bidding process. Amazon is growing in ways that other companies aren't, and it is unfortunately heavily subsidized by tax dollars, or deferred tax dollars.

Example: A recent piece about Amazon negotiating electricity rates, which in the end stiff locals with the brunt of the bill...then there's their corporate welfare rent seeking of HQ2...then there's this contract. They skirt taxes more than average. They treat their floor employees far far worse than Walmart or McDonald's.

It's beginning to get a bit ridiculous. If conspiracy theorists want to say Trump is behind the backlash (or a rogue corporation like Oracle,) go for it, but that's not the (full) truth. That's just a cover now, since anything (rightfully in many cases) connected with Trump's criticisms are tainted. Don't fall into that trap, Amazon is beginning to become a troublesome company. I don't know what to do about it, but rewarding them government contracts that are, in the end, rigged to be "no bid" isn't helping. Then on top of that, buying the Amazon line of not using multi-vendor fail overs is baffling. This may be a 10 year contract, but it's built to vendor lock the government.

Secret? I thought it was quite obvious. Anyone who is going to miss out on the cash. that's who.

Why would the Defense Dept. choose Google or Microsoft cloud? I wouldn't put US military cloud on GCP or Azure after the Project Maven and ICE thing. Even if they were cheaper. Those companies are full of people who have demonstrated willingness to sabotage America's defenses. Their names are on anti-America petitions around the internet.

Amazon has proven willingness to assist military and law enforcement using cloud facial recognition technologies in the face of criticism from detractors. Of course AWS will win.

Reply on news.ycombinator.com