Pentagon names Microsoft and Amazon as $10B cloud contract finalists | Hacker News Reader

Pentagon names Microsoft and Amazon as $10B cloud contract finalists

techcrunch.com

287 pointsRmDen142 comments

I tried to build a VC backed startup that sold a SaaS product to state and federal governments. The procurement process for the government is so horrible and broken that it blew me away. From first conversation to signed contract will take 2 years minimum. It's filled with legal challenges, back room deals, inefficiencies, a ridiculous RFP process. You run out of money before you even get your first contract signed.

It was so strange to me to sit in meetings and talk about how we could make them so much more efficient and they would talk about having the problem solved in 5 or 10 year timelines. Having spent most of my career at Microsoft and Netflix where a year feels like an eternity, it's just a completely different world.

the other side of it is the decision you make like this in government will have a 30-50 year impact. making changes is not easy for anyone, but you can change an entire infrastructure at microsoft in a tenth of the time you could do it at the DoD

Well said, buying for an entity that needs to survive to the end of time is a totally different ballgame.

For comparison, the DoD's 2019 budget ($686B) is six times Microsoft's 2018 revenue ($110B). It is one-hundred and sixty-three times Netflix's $4B.

Add that to relative technical aptitude, regulatory compliance, etc. and it doesn't surprise me Government is going to take longer.

The procurement process for the government is broken because our political system is broken. You need deep pockets and political connections.

This is actually Elon Musk’s greatest triumph; almost all of his post-PayPal businesses are based on exploiting weaknesses in government regulatory (Tesla, PayPal) or procurement processes (SpaceX, boring company).

SpaceX was legitimate innovation in a broken government contracting process, but Tesla is starting to look like it was only a viable business with the subsidies — I would anticipate their business model to shift away from vehicles soon and turn towards large-scale grid infrastructure (aka government contracting).

Yes and no.

Separate from the political issues at the top level of funding, the entire FAR (Federal Acquisition Regulations) is a steaming pile of reactionary bullshit, grown by accretion to minimize risk to bureaucrats at the expense of serving the tax payer efficiently.

The whole "bribing and political connections" side is sexy and interesting and headline grabbing, but the bulk of the issue is just far more banal and depressing.

The sad part about it is that the state of the FAR is such that our government, even if you are a person that wants it to play a bigger role, isn't really capable of executing on big things anymore. Combine that with the inept and apathetic staffing of (typically) mediocre at best clock punchers, and you end up with an organization that has a deep, deep spending efficiency issue.

The two political parties are both clueless. One thinks a massive, bloated organization with a track record of spending inefficiencies needs more revenue. The other thinks that a nation of 350 million just doesn't need a funded federal government.

> Separate from the political issues at the top level of funding, the entire FAR (Federal Acquisition Regulations) is a steaming pile of reactionary bullshit, grown by accretion to minimize risk to bureaucrats at the expense of serving the tax payer efficiently.

I don't think people understand just how much of the procurement system has been built out of risk aversion.

A thing needs doing, politicians / government workers procure a solution to that, and that solution ends up going wrong somehow. Lots of egg on the relevant politician's faces, and plenty of ammo for anyone running against them. "$foo wasted millions of taxpayers money on solutions that don't even work!" So they put in a new mechanism to try to ensure that mistake never happens again. Each individual mechanism on its own generally makes sense.

But over decades and decades it becomes such a complete mess that if you take a step back and look at it, it's all kinds of absurdity (There's a fun parallel there to the infrastructure and systems we tend to make as developers and sysadmins). Worse, the kind of absurdity it has become actually ends up costing taxpayers more money than if a bunch of the regulations didn't exist, cheaper things were tried and failed, and replaced with one that does work.

By the same token, if you normalize failure in a structure that large, it’s hard to get anything done. It’s more enriching to get paid to fail 5 times than it is to do the thing once successfully.

Failure is great when there is oversight and feedback mechanisms that help the organization learn from those failures. Less good when repeated failure is profitable.

The major driver of procurement bureaucracy is exactly this sort of blatant, somewhat cynical, distrust in government.

Every requirement is designed to protect against arbitrary decision making, especially corruption but also just general "I like the vendor/product".

Corruption is self-explanatory, so let's focus on the second:

Instead of buying a Toyota because you liked the last one you drove, or because they offer a fancy color you like, you have to create a 500-item checklist of features you want in a car, and a scheme to evaluate and weigh all possible options.

If you forget a single important criterion, say "available in colors other than pink", the police department will now be driving in pink cars.

The problem isn't that the process is really broken (though it is in many ways) - the problem is that too many companies fail to understand there is a major difference between government and commercial markets. Two totally different mindsets.

Yes, the sales cycles in government are a lot longer (2 years is at the extreme end though, 3 to 6 months is more average but with a lot of caveats) and yes, there are a lot of rules and regulations that don't exist in commercial deals (FAR, DFARS, etc. etc.) - but at the end of the day, you're talking about business-changing scale when you get awarded a decent sized government contract (millions of dollars over a period of 3-5 years on average or longer). You're talking about being able to grow to 100's of employees with just a half dozen or so government customers.

The problem here really is that people need to invest the time and energy into understanding the government market and they can't expect it to be like the commercial one. If you do this, then the payout is very much worth it (and, no, it doesn't require "deep pockets" but it DOES require relationship building, a strong commitment, and a lot of learning).

If you give up in frustration because "it's too convoluted" compared to the commercial market, then, yeah, it will never pay off and it will feel like everything is stacked against you. But if you stick it out you'll find yourself with some incredibly stable and predictable long-term revenue streams.

I think the concern is that the reason it is so different is disturbing to taxpayers: If it is filled with more bureaucracy than a similar commercial deal, then they're wasting taxpayer money. If that bureaucracy is the only way to prevent corruption, then they must be horribly easy to corrupt. If you can get a guaranteed revenue stream to grow your company to 100s of people with no day to day risk of losing this customer, then they're not being good stewards of that money. In short, them being different in these ways are bad sign for the taxpayers who fund this.

But the bureaucracy is sometimes for good reason (usually driven by Congressional mandate). What people need to do is learn to work within (and around) the bureaucracy. That's key to the government market. There is a reason for the way it is, and you need to learn to work with it instead of fighting against it.

I'm not sure you're understanding really how the system works, which is exactly what I was pointing out in my statement. I never said "guaranteed" - I said "stable". There is ALWAYS risk of losing the customer and the government can terminate at any time for almost any reason. But if you perform, then you can maintain the same customer for a decade or more.

This is how the system works and it works this way for a reason. There is more accountability in the government market than people realize - moreso than in most commercial transactions even.

I think people would be more willing to accept the bureaucracy if it delivered results. If you've ever been the customer of a government tech product (even a public-facing one like healthcare.gov) you know it does not.

Take a look around the world and look at the average state of governance and you will find a lot of appreciation for American bureaucracy.

And it is indeed independent bureaucracies that are responsible for making agencies and institutions run. Bureaucracy gets a bad name in many places, but the alternative to bureaucracy isn't just automatic improvement, it's most often politicization. Which is significantly worse.

Historically, government contracting and the public sector has been incredibly corrupt, as it's basically unlimited funds with no market-based mechanism to address it (in a private entity, the owners will be angry about money taken from their pockets by employees, and in places where corruption is too high, you end up with family firms or other external trust structures, very small entities, etc.)

This is why the heavy-handed regulations exist in US contracting (along with massive risk aversion and other things). The Government uniquely cannot be subject to market forces and punishment for bad actions.

That's true in places like India, but not really in the U.S. I own a company that does government contracting and have never "paid someone off" or done anything remotely "corrupt" but we have been wildly successful nonetheless.

There are NUMEROUS mechanisms for oversight and redress at every level - but the typical commercial-focused company doesn't understand them, so they don't research or learn what they are and just give up thereby developing a very misguided interpretation of the system akin to what you believe it is like.

This was true in the US before we created all these stupid regulations, which is why they were created. 1700s/1800s, "spoils system", etc. That is the whole reason the regulations (and attendant overhead) exist.

(I bootstrapped a US government contractor and have worked for several others.)

Can you clarify your point, were you saying that the US had 3rd world-levels of corruption before the regulation, or were you saying that the US was as fine as it is today, before the regulations.

The former- before the regulations in the 1800's there wasn't even really a concept of 'third-world levels of corruption' as it was universal to every government.

The former. Google the "spoils system."

I've worked for commercial, government and semi-government organizations. It's not corrupt in the sense that it's illegal. The problem is that while the stakes are high within the private sector there are no such stakes within the government sector. Yet people do get payed high stakes salaries. There's no incentive for efficiency. In the government realm when the end of year is nearing, departments start burning through their budgets to ensure they don't receive less the next year. Inefficiency is rewarded. It simply means more people get to pay for nice mortgages.

> If that bureaucracy is the only way to prevent corruption, then they must be horribly easy to corrupt.

The bar is much higher for government procurement, though. As an example, if I decide to use Codacy as my static analysis tool at work because I know the founder, nobody would bat an eyelid. If that same dynamic were reproduced at the government level, it would amount to corruption.

Having worked for a government contractor, it was my impression during my time there that they're so worried about oversight of the money and what miss-spending and corruption looks like that they spend most of the money making sure it doesn't get miss-spent. Often the amount spent recovering miss-spent money far out-weighs the cost of what was miss-spent in the first place.

One example my attention was drawn to is that a government employee overspent by $1.50 for a drink that was misassigned an expense code for the amount they spent. If they'd used the correct expense code or spent $1.50 less, no eyebrows would have been raised. Instead everything was processed and the person was pulled into a tribunal in front of other government employees who were being paid to oversee spending. Lord knows what those employees were paid to investigate this miss-spend. On top of the fact this employee was paid to attend the tribunal. This kind of behaviour is a gross waste of time and money... for $1.50 at a cost of many thousands of dollars to reclaim - or more likely re-assign to the correct expense code... and for what?

If you want to see your tax dollars at work, it's right here, making sure none of your hard earned money gets miss-spent.

It's accountability in the age before we had AIs that could spot strange patterns of behavior across millions of government employees. Presumably now there might be more efficient ways to do this, but it's a combination of inertia and fear of what the consequences would be if such a program didn't work out. The upside of the current system is that it leaves huge paper trails and everything is auditable; the downside is that actually auditing all this stuff is so painful.

It also doesn't really help that a majority of the politicians and a majority of voters foam at the mouth about government waste, no matter how minuscule and how much it would actually cost to eliminate. If we implemented more efficient algorithms and saved 90% of administration but some waste slipped through they'd say the government was getting sloppy. You see this a lot when people rail against, say, fare evasion on public transit; people would rather have 10x of a bunch of highly paid employees to manually check every ticket, instead of having proof-of-payment and random inspections with fines, because it feels like the right thing to do.

Here's the scale of problem the government deals with:

https://www.youtube.com/watch?v=4JkcrtvN60s

That's an experimental YF-22A crashing due to a software glitch. That's a $300-400M hand-crafted state-of-the-art-on-a-hundred-dimensions supersonic jet, almost killing the test pilot (an engineer who flies planes) 50 feet off the ground at 150 mph. Just the cost of training that pilot is easily in the millions of dollars.

Everyone agrees the budget process is nuts, right up until something like that happens. And then everyone who ever signed any document related to that airplane is able to sleep at night. Except the software teams that wrote or approved the code. They're going to loose sleep, and they know why.

Oh, and the DoD is a rounding error compared to CMS. They have routine fraud complaints that would sink a major defense contractor.

I think fraud detection and holding people accountable for safety lapses are two completely different problems calling for different solutions. Just because you have a hammer doesn't mean everything's a nail.

If we didn't spend so much money on fraud detection via immense bureaucracy, that would increase the amount of money available to provision services.

Serious investigations of seemingly-trivial events can produce some startling information. Look into the ongoing Kealoha scandal in Oahu. One broken mailbox is tearing down a massive criminal conspiracy.

Stupid question, but how can it not require deep pockets when it can take up to two years to get to a single signed contract?

I think because the implication of the phrase 'deep pockets' for most people implies a graft or bribe of some kind.

Reading it as - you need enough money to make it two years without payment - is fair, but I would argue not how most people read that statement.

From what I gathered when I got an offer for a government contract position: the company has an idea of how many people they want to hire (or things they want to buy) to fulfill the contract.

This can involve making "contingent offers" to get people interested, which to my knowledge carry no actual legal weight and is really just an informal agreement.

This way, you get a reasonable pool of candidates without actually having to hire anyone.

> It was so strange to me to sit in meetings and talk about how we could make them so much more efficient and they would talk about having the problem solved in 5 or 10 year timelines.

Maybe their experiences or observations with "VC backed startup[s] that [sell] a SaaS product" has shown them again and again that you cannot trust those in any way to be even around for 2 years.

Seems unlikely to me. The barrier to entry for startups selling to the government is so high because of the regulations to prevent corruption force such a long sales cycle.

Another thing that surprised me when it came to selling to the government is that it seemed like most employees had no idea how to purchase something, even if they had budget and authority.

The federal government is generally biased in favor of large entities that'll be around in 20 yrs because that is the timescale over which the government operates, which is why their technology is always significantly behind.

As a startup your best bet is partnership with the big guys, meaning you'll have to craft your solutions in a way that it can piggy-back on all the checklists (things like FeDRAMP) the big guys already have checked.

Don't even get me started on the bureaucracy full of people whose sole purpose for existence is to ensure nothing can get done:)

Not true. 23% of all government money goes to small businesses. Over 30% in certain agencies and up to 60% in some. Like I've been saying here, you need to take the time to learn the market because if you don't, you will just believe it is stacked against you when it really isn't and is actually more friendly to smaller companies than certain commercial markets.

My response was in relation to a VC funded startup business, that is very different from the "small business" segment you're referring to.

There are loads of "small business" entities out there who get government business merely by being in existence and meeting the sizing requirements.

I however wouldn't quit my job to start one of these because these are really just people grazing on the federal acquisition process and provide in most cases little to no value, it is not much money either.

I know this because I have experience with working on government projects where you can easily have 20+ contractors involved and it is not clear what their roles or input are to the project.

question:

are most of those small businesses partnered with larger entities on RFPs to win those contracts/money?

A large percentage of them are. When starting out many small businesses exclusively work as subcontractors to larger entities which must partner with smaller businesses to meet set asides.

Some contracts require the prime contractor to be a small business. Often when this is the case and the contract is large the small business acts almost purely as a pass through to a larger contractor. Taking a percentage off the top and filling a couple of slots on the contract while the larger contractor does fills most of the slots and takes the majority of the money.

true in some cases, but some set asides require the small to do a certain percentage of the business

Which is why I said often and not always.

I've worked in Canadian Federal sector after decade+ in commercial sector. My mind was blown - they do not compare:

1. Speed is an order (or three) of magnitude lower. On anything - from macro-scale items such as signing a contract, designing the solution, creating the plan; to micro-scale items such as opening a port (2weeks-2months), creating a VMWare VM (2weeks-2years), obtaining a standardized laptop for new team members (2-weeks-3months). (Yes - you read that right - there is a set of two well defined VMWare windows VMs we have literally been waiting on for two years)

1b. After my first project, I reviewed the openly available "lessons learned" from variety of Gov't IT projects, and in the top three, uniformly, you'll find a sentence like "External vendor did not have sufficient appreciation for nature, complexity and speed of government processes". Truer words never said. We became faster when we slowed down to the oragnizational level: if we tried to do something in 2 days that government does in 2 weeks, it'd take us 4 weeks of hitting our head against the wall and causing havoc. Once we understood the 2 weeks it takes and built it into our plans, then 2 weeks it took.

3. As a citizen, I think of "The Government" - a single, big, overarching entity. Forget about it - it's really a lose and overly-formalized set of practically-independent organizations. IT integration of Department of Agriculture to Ministry of Labour is like integrating Walmart to Tesla. Different standards, processes, networks, firewalls, etc. It can take couple of years to agree and consume standardaized webservices across departments

4. Like many large organizations with variety of responsibilities and products, stability and predictability are much, much more important than speed. They need a solution that will last 20 years and a vendor that will exist and support it for that long. They can't change frameworks across 42 datacentres and 300,000 employees every 2 months. When their budget works at best on yearly basis, they cannot change their CIO roadplans quarterly.

5. This is a personal revelation and people may disagree: we want, or rather demand government to be that way. Due to requirement on transparency, fairness, documentation, we the public end up pushing government in the corner. They cannot just pick the easy right choice in two weeks - that'd be in the Globe & Mail (think Washington Post;) as unfair process, favouritism, lack of transparency... other companies would outright sue. So you get a two year procurement process that ultimately WE demand. Same for every decision at every level - it must be examined, discussed, commented, documented, covered. Everything has to be supported at an order of magnitude higher level than in commercial world (and note it has to be a well-supported, not necessarily right decision). Government procurement and processes have documentation and checkpoints which in some ways can be wonderfully stable and structured, in other ways immensely frustrating.

My mentor told me early on: think of everything you do in the context of front page headline: "Consultant does X on public money" So I can't buy lunch to my colleague I've worked with for years ("Vendor tries to bribe public servants"), cannot have a kick-off dinner for the project ("Consultants eat and cavort on tax-payers' money!!!"), etc.

My personal fear/nightmare is how I'll integrate back into the market having adapted to the public sector way and speed of doing things. Reading about neck-breaking speed of JavaScript frameworks and public cloud changes out there "in the real world", doesn't help my sense of peace :P :P :P

>we want, or rather demand government to be that way

This is something so many people miss. So much of the endless red tape is a direct result of public concern over wasting taxpayer dollars. The irony of the situation is the government would be able to save a lot of money if taxpayers stopped caring about where their money was going.

I often wonder whether we'd be better off if we "tolerated" a small amount of corruption. What's the point in spending $1 million on red tape and auditors to prevent $10,000 in fraud?

Just like the war on drugs, you're fighting a moral crusade against the basest part of human nature. You're always going to lose - better to just accept it and do what's needed to minimize net loss.

Government is inherently inefficient, yet some people still insist that it controls everything.

Classic reverse causation there.

I've worked as a government contractor in a few industries and my experience is just like OP's. The government is slow, inefficient, contradictory, expensive, and it under-delivers. This isn't an accusation, this is just fact, and it's not even a secret.

However, if you are able to land a contract or two, you have a lot of money pouring in. Maybe enough to the point where you don't need any further outside funding, or a reduced amount of outside funding. And since the contracts are for 2+ years and you have to really screw up to have it canceled, then you have guaranteed income for years. You can't run it like a consumer startup.

Large enterprise customers can take just as long with an even less transparent process. It's not a particular problem of gov't, it's a problem that comes with size.

I work for a Danish municipality, we’ve bought a few SaaS products in recent years. The fastest implement was two days, and we had a working product before we were even done with the contract. We bought it from a local software house, that knows how political organisations operate, they host in AWS, but they could have hosted it in Azure for all we care. (We would prefer Azure because they are better in Europe, but AWS is fine).

So government operations and contracts can work fast. Until you meet a certain amount of cost that is. We have laws that govern how we buy software, and once they kick in, the process is long, tiresome and really, really, ineffective.

Because often the company who wins a bidding war, underbids, and end up delivering a lot less than promised unless you pay ridiculous extra money, and nothing in those laws protect us from this, not really.

The big issue is that government isn’t similar, so you have to govern in for the lowest common denominator. My ciry has technical competences on staff, this allows us to actually utilise SaaS products that require integration into our business processes and 300 existing systems. Our neighbouring municipality doesn’t, and as a result, they don’t even really know what the word SaaS means. The government has to write laws that work for both of us.

I think the real issue is lack of technical competences in the public sector, and the reason is really beyond me. Sure I have a vested interest, but I think I can defend it by my ability to write software in a few weeks that would cost us half a million on the private market. That’s not the primary importance of technical staff though, I don’t think we should build all those 300 systems ourselves, but we need to know how to cooperate with the private sector, and we need to build an enterprise architecture to support that, and to do that, we need technical staff.

The fact that this isn’t obvious in the political command structure in 2019 is just silly. Especially in Denmark where we’re competing with Estonia for having the most digitised public sector in the world. But to add some numbers to it, we have 7000 employees, they use a digital device or a computer for an average of 5 hours a day. We have 245 physical locations with network (schools, libraries, child care institutions and so on). We have more than 300 different systems, some major some minor. We’re expected to operate this with 5 people in operations, two people in development and architecture and 3 people in support.

I think I know why though. We don’t have a single elected official in my entire country who knows how to program.

> From first conversation to signed contract will take 2 years minimum.

If you are trying to sell to government for a “need” they aren't actively seeking solutions for, which your description sounds like, that seems plausible. Then again, a major purpose of the safeguards in the procurement process is to prevent sales going in that direction in the first place. The intent is for the government to identify a need, and then solicit solutions, not to have a vendor with a vested interest in the government perceiving a need in a way which favors their solution sell the government on the need.

It's what Avisare [0] is trying to address (I have no connection to the company).

We have some government contracts pending and we decided to give them services free of charge rather than going through the procurement hell.

[0] https://www.avisare.com

This is further compounded by how Gov employees are compensated. Gov IT is not a meritocracy with RSUs, so "efficiency" and "performance" are not intrinsic goals.

To truly succeed in Gov contracts, solutions must be aligned with the retirement and pension goals of procurement officials who want to "ride out" long-term projects to the sunset.

Pro tip: don't sell to the government. Regardless if it's in the US or outside

Rather, sell to someone (or partner someone) who is experienced and let them do the talking

Also don't do it as the 1st thing you do, you'll have much more success selling it to people/(small) business first.

I would only advise selling to gov first if 1) you absolutely know what you're doing and it is something that only them can buy and 2) you have money for someone to walk in the higher circles (as in both salary + "extras")

This can be a very difficult market, but there are a handful of different projects designed to help with this. In-Q-Tel [0] has been accelerating adoption of new technology for the Intelligence Community for many years and often invests in and helps deploy technology from startups of many sizes. SBIRs [1] can be an effective way to quickly get gov money for research and even culminate in a procurement vehicle for sole source contracts if you get through Phase III. There are also a handful of accelerators like DIU [2] and MD5 [3] designed to help small firms navigate this difficult space. It still isn't easy, but it can help level the playing field a bit when you are just a startup.

[0]: https://www.iqt.org/

[1]: https://www.sbir.gov/

[2]: https://www.diu.mil/

[3]: https://community.md5.net/md5/landing

The dirty secret of government contracts is that most firms, no matter how good their tech and implementations are, couldn't win the contracts because they don't understand government procurement well enough. This leads to the same companies winning contracts over and over again just because they know how to win the bids, and not because their solutions or prices make sense. Many big contacting and consulting firms are really just experts in the procurement process.

This then leads into third parties acting as procurement contractors, where their only job is to win government contracts for others.

There are other middlemen taxes like this. Ever seen a proposal that says that it must go to a small business or a certain percentage of the award must go to a small business? Just find a small business to bid on the contract, take their cut, and then sell your product as the solution.

Government procurement is a dark place.

Overpromise and underbid, there are few if any consequences. YMMV

This is 100% spot on in my experience.

Is it horrible and broken? Or is the grueling process a feature? By filtering out VC-backed start-ups they are left with only bidders who they can be sure aren't going to vanish if funding dries up or a few key staff leave.

I guess Oracle's desperate attempts (to the point of suing the government) to ensure a cash cow for the next decade despite clear tech inferiority were not successful in the end

Glad to hear it. I'm disappointed that Oracle keep getting away with their shady and bullish business practices.

The government LOVES their oracle databases. They are very entrenched there. I am actually really surprised they weren't selected.

Maybe the government tolerates oracle; have endless pockets and dont care about the financial aspect of using oracle dbms.

We built software for data analysis for the government and had to integrate with their data storage systems. I wasn't exaggerating when I said they loved Oracle. In my experience, they seemed to think Oracle databases were superior in almost every way.

I think you misspelled loathes.

Both AWS and Azure support Oracle Databases. This is the selection of a cloud provider, not specific software that would run on that provider. They may very well run Oracle Databases in the future.

They do run oracle databases - or at least AWS does, in its rds.

That may be the case but as my employer is learning it takes a ton of engineering effort to actually run a complicated, large scale oracle DB on the cloud. At every turn you either have to compromise on design or expend a ton of effort getting over the hurdle. Even at large scale running it bare metal and then distributing that around the world is easier (that's what we've been doing). Time will tell if the effort is worth it.

I’m a cloud True Believer but even I wouldn’t suggest it is the right choice to run any large third party proprietary database like SQL Server or Oracle on AWS.

At least with AWS’s custom Aurora/MySQL and Aurora/Postgres offering they are able to rewrite parts of it to take advantage of their other technologies, but when you look at their RDS offerings, it doesn’t really buy you as much if you need to run third party commercial databases at scale.

Background:

Oracle made a complaint to the Government Accountability Office last year that was essentially "it is unfair to award ALL $10B to just one company." The GAO disagreed with the following reasoning:

> the Defense Department’s decision to pursue a single-award approach to obtain these cloud services is consistent with applicable statutes (and regulations) because the agency reasonably determined that a single-award approach is in the government’s best interests for various reasons, including national security concerns, as the statute allows

So now Oracle is suing the US Government with exactly the same complaint.

Oracle Senior Vice President Ken Glueck is a statement claimed that they're worried the DoD is going to be "locked into" another cloud vendor and that a single vendor "is out of sync with industry’s multi-cloud strategy" Make of that as you will.

Yes. And try to get Oracle to play nicely with Microsoft Technologies.

Having multiple vendors is great, but here they are just talking their book. Do they pitch multiple database strategies too?

At my employer we're working on just that. It's a hell of a task.

Good riddance- Oracle cannot die fast enough.

Their claims about JEDI needing a "next generation cloud" and Amazon being a "legacy cloud" that they'll be tied into is laughable as well.

Oracle hasn't been next generation ANYTHING since the 90s.

On that note, what's your take on $ORCL's "autonomous database" that they've been marketing over the last couple of years? I saw the Ellison presentation on it when it was first announced and it seems plausible you could tune databases on-the-fly and maybe optimize indices "automatically", given your comment I was curious what you think about this specific offering.

https://www.oracle.com/database/what-is-autonomous-database....

Is it really anything new? Informix was self tuning in the 80s.

I suppose you could automatically identify slow queries and identify what indexes would be useful, sort of how a JIT identifies hotspots in code and optimises them.

Personally it sounds like a play to keep lucrative enterprise customers on side by allowing them to cut the DBA role that is a necessity for a decent sized Oracle install.

Glad to see the attempt to split the contract up by appealing for a multi-platform approach failed.

A multi-platform approach destroys most of the cloud's value proposition in the name of avoiding vendor lock-in. At that point you're essentially only using the provider as a glorified data center manager, and I'm pretty sure the DOD has more than enough money to run their own data centers.

I would love someone to explain clearly the strategy of "multi-cloud" products to me, what they do, and how they are long term viable. (I do mean this genuinely because I'm not getting it.)

1. Surely there is no way for any "multi-cloud" company to keep up with the constant changes and growth of products at all these companies that they can abstract between them in any meaningful way without constantly falling behind.

2. And if there is some purpose right now, is that purpose sustainable? Why would Amazon not have every incentive at some point to make it as easy as possible for an existing Microsoft customer to migrate workloads over to their platform, and vice versa. And if/when they get into the game, isn't that essentially game over for any independent product?

> Surely there is no way for any "multi-cloud" company to keep up with the constant changes and growth of products at all these companies that they can abstract between them in any meaningful way without constantly falling behind.

This is the appeal of kubernetes.

And yes, it does make cloud providers glorified datacenters. I think k8s is an existential threat to aws long term. They're not going to be able to maintain their profit margins on ec2.

And Kubernetes still can’t give you all of the benefits of managed databases, queuing, messaging systems, etc.

If you’re only using the cloud to do VMs at scale, you’re probably spending more than baremetal and spending just as much on maintenance. Kubernetes adds yet another level of complexity.

k8s was a genius move by Google, and it's working. Especially with OpenShift, it will commodize PaaS.

I'm hoping the same thing happens with Knative.

That's a great move by Google, but it's a terrible move for Google. I'd love to see PaaS and FaaS commoditized, but precisely because I want to write apps that run on both vendors and transfer capacity back and forth depending on who is cheapest at the moment.

And that works in Google's favor currently, since they are small player entering the PaaS market, and they need to compete with AWS on price, because their offering is less mature and they don't have counterparts to many of AWS services. Making migration easier is a reasonable strategy for them.

> Why would Amazon not have every incentive at some point to make it as easy as possible for an existing Microsoft customer to migrate workloads over to their platform, and vice versa

This is what flies in the face of multicloud efforts for me.

Now, most of the big cloud provider's new revenue comes from enabling migrations to cloud infrastructure. But that's a finite amount of work.

At some future point, everyone who will has already been migrated to the cloud. At that point, all the cloud provider's new business will be in the form of enabling migrations from existing clouds.

And then you'll have the resources of Amazon / Microsoft / Google working to enable migrations. Which seems a lot more successful resource match.

(That said, at that point cloud vendors could also really start actively turning the screws on making that difficult. But in a three-party world, where two parties don't, it seems unlikely that would succeed in the long run.)

For something on the scale of the DOD, it makes very little sense to operate in the cloud, I am still unsure why this is a goal for them, particularly considering both the cost and the security concerns.

But single-sourcing your cloud destroys any remaining value proposition: If you become dependent on a single cloud provider's services, they can charge you as much as they want once the current contract is up, and your lock-in means you're going to end up paying it. (Or we all are, since it's the DOD and we pay taxes.) The ten billion dollar contract is nice to have now, but I'm sure Microsoft and Amazon are both really licking their lips for the forty billion dollar renewal ten years later.

The value prop is the same for them as it is for any other company choosing a single cloud: it unlocks a ton of services and basic building blocks which empowers and speeds up software development, as well as providing fundamental stuff like access control and auditing.

The concern over vendor lock-in and pricing is one that gets thrown around often, but doesn't really make sense to me. When has Amazon, Microsoft, et al gouged one of their larger customers in the past? Why would they do that when it would destroy their reputation and scare away business?

A large customer working with a large company is like any other negotiation, both have leverage and there isn't any feasible scenario where Amazon or Microsoft go completely crazy and start gouging their customers for some short-term gain.

There's nothing "short-term" about having the entire Department of Defense locked to your services. Our government still runs systems from decades ago.

When you're signing contracts at DoD scale, typically you negotiate a "vendor can only increase prices X amount per year" clause.

Companies that get screwed are those who aren't big enough to negotiate that kind of deal up front.

But no, a Top 50 or large government type contract is never going to be at the mercy of unexpected price hikes.

They will, though, be at a disadvantage in pace of new features, hardware refresh, etc. The gov version will lag far behind the general offering. It's also possible the general offering will go down in relative pricing, like cost/mb or same price, but faster CPU.

"A glorified data center manager" is exactly what AWS and Azure are.

I don't have enough eyes to roll at the cluelessness of your comment.

I know both companies want this contract badly, but if your company lands the contract and they say, "we need engineers to work on building this out," RUN AWAY. Anyone at AWS who worked on GovCloud and similar projects can tell you it's going to be a s*show.

Yeah...Azure Gov is the same. They'll always be behind the commercial offering.

I heard the guy who wrote the government regulations for cloud computing promptly resigned after it passed and went to go work for AWS.

As long as he has no interactions with his former colleagues, I don't see the issue. You want to make sure your company is following all the rules: who can tell you that better than the person drafting those rules?

The government hired us to write the RFP to purchase our software. Apparently this is common practice. Ironically, they ended up selecting a different vendor in the end.

I'm still in favor of the revolving door surtax.

https://www.usatoday.com/story/opinion/2013/01/28/revolving-...

Presumably, he was chosen to write the government regulations for cloud computing because he knew a lot about cloud computing.

Would like that anyone that ever worked for the government forever be banned to work in their domain of expertise for a private company?

The $10B number seems funny. Government procurement contracts and projects routinely overrun their budgets and schedules by a factor of 2 or more (something software developers should be familiar with). I wouldn't be surprised if the winner of this contract got considerably more than $10B from this contract alone.

It is a $1B/year contract for 10 years, but the DoD has a number of early-outs if costs overrun or the vendor under-performs.

Where are the details of this? I'd like to give it a read, thanks.

In the article this thread is about.

Can dig into the details here https://www.fbo.gov/index?id=3860a4f4fe9d9ffc31e722ece82a143...

No surprise, I believe they are the only two that even came close to the requirements.

There have been news of group of employees opposing this internally at Microsoft. Employees feel that they have been cheated/mis-guided my MS.

Think that was with Hololens. Previously there were qualms with Office too. Though I imagine there'll be an open letter about this one too.

Dunno, I try to be sympathetic to concerns, but don't work for a business that is willing to do government contracts (or have the government as a customer, for that matter) if you don't want anything you work on to be used by the government.

>you don't want anything you work on to be used by the government

Is this the work of some kind of extremist libertarian branch inside of Microsoft?

the open letters and what not? I have no issues with their, or Google employees for that matter, frustrations on a surface level, but at some point you either have to make peace with the fact that what you're working on has the potential to be leveraged in a way you don't agree with or move on to somewhere else where you have some semblance of the corporate ethics and/or morality aligning with your own if that matters to you as a person. Unintended consequences stink, I totally agree as I did a bunch of enhancements on a system once with the intention of making everyone using it have an easier time but it resulted in the company needing less people to do the job (not by firing thankfully, but these roles were filled by temporary workers so as attrition naturally happened, instead of being a revolving door, they just weren't replaced as they were leaving). Felt miserable about it when it happened and it still informs me to this day, but I also learned from the experience that you can intend one thing yet the result can be very different.

The lower you go on the stack, the greater likelihood you're building something that could be used in a way you disagree with.

I don't know how you can work on a cloud provider ANYWHERE and not have this issue. I don't know how you can work on any open infrastructure, honestly.

There are worse actors than the government already in the cloud, if you ask me.

Which choice betters my career prospects if I'm just getting into AWS development/integration?

Pick one, build something non-trivial for it, learn and form opinions about it. Rinse and repeat this process while exploring new tools and technology. Try to automate as much as possible. If you can stay relatively sane after a handful of iterations, you are not only an expert on cloud computing platforms, but an expert on the kinds of decisions, skills, tools, techniques, and experience needed to succeed in the industry. You can do it!

Choice between what?

AWS vs Azure is about like Java vs C# in the enterprise. AWS is twice as popular, but you won’t go wanting if you’re good with Azure technologies. No GCP is so far behind in adoption and mindshare it’s just not worth it to me.

Great to see Pentagon plays Amazon like a champ.

It'd be slightly amusing to see the Pentagon work Amazon over on negotiations like Amazon worked cities over on HQ2. Amazon ran into an entity that was larger than itself.

You think Amazon cares? They will happily eat this as a loss-leader. Neither MS or Amazon is fighting over this for a measly $1B a year. They are fighting over this for $100s of Billions of contracts with other parts of the government.

Not just other parts of government. There are a lot of big defense companies (ie Boeing, Lockheed, Raytheon, etc). Being on the same cloud as the DoD is going to be a big selling point.

“The nose of the camel under the tent” strategy.

Oracle = lawsuit, go home oracle your drunk!

No real surprise here..

No suprises there

How long until the staff at Microsoft/Amazon strike?

Why would they strike?

Google employees went on strike because of a government contract to improve AI used in the targeting of drone strikes. The fallout of that project lead to Google dropping out of the running for this contract citing its "AI principles".

https://www.nytimes.com/2018/04/04/technology/google-letter-...

https://techcrunch.com/2018/10/08/google-will-not-bid-for-th...

Microsoft employees are not like Googlers. There was a protest letter that was passed around and I think it got more ex-MSFT than actual employees to sign it.

I suspect a few people from both companies will complain but it won't be enough to dictate change. MSFT employees complained about the company working with ICE and while that did get a response, it didn't get the contract canceled.

Do you think the hiring process at Microsoft somehow selects fewer people with strong political views?

Or perhaps the employees are the same, but the corporate executives and culture encourages or discouraged certain behaviors?

Google is wired to have more "activist" type people. I don't think Microsoft has that type of culture based on the people that I have spoken to that work there, though I'm sure they exist.

I think there is a big difference as well to work on something generic that the military happens to use as well like a database or truck compared to building something which is mainly part of a weapon. Luke a drone targeting system or a truck mounted AA gun.

That’s not really the same thing as generic cloud services. At a certain point you need to draw a line for responsibility. Is it renting computer time? Probably not.

I think most people would distinguish between government vs. military contracts (despite the outsized proportion of government spending that the military occupies).

Um... never? Why would they strike? If anyone cared, and I don’t think anyone does, they would just leave.

How much % do you think cares about protesting a gov contract?

Strike about what?

A large percentage of the technology we use daily found its roots in projects funded by DARPA and related entities.

Businesses need customers to pay the bills, the salaries, and perks enjoyed by those employed by these companies. Someone is going to provide services to the government, so it Makes sense to compete for the business.

It is reasonable to wonder if there is going to be infighting about this decision at Microsoft and Amazon, because Google employees have already demonstrated that it is possible.

And when this topic came up with Microsoft, Satya went on record saying they were continuing with pursuing the business.

Coordinating with the Chinese (a foreign) Government to implement spying and censorship as Google (and Cisco prior) did, at least for me, is completely different than competing for hosting contracts with the government.

Not everyone is American - what may be foreign and untrustworthy to one person, may be native and honorable to another.

It is not black and white at all, which makes some of the hoo-haa about this all a bit strange to me (with people saying "... but this is helping our military keep us safe!..." neglecting to consider the other side of the coin of the citizens of another country on the receiving end etc etc)

True, but in the context of the employees (in the US) they are either American citizens or on some form of US Visa. It would seem somewhat contradictory to be a non-Citizen protesting the policies of a US company while employed at that company on a visa issued by the US government.

Also, having had a relative that was high up in one of the services and dealt with logistics. Not every contract involved with the Pentagon is specific to a direct conflict effort. It could be HR/Personnel systems, inventory tracking, etc.

During my undergrad/grad years, I worked under a DARPA grant around weather prediction leveraging GOES imagery. Money coming under the auspices of the DoD, but related to research not specific to a military usage. Amusingly, some campus individuals who were anti-DoD put up wanted posters of the professors involved with work under this same grant. (I know different than providing cloud services, but just an example of government military budget going towards efforts that dovetail into civilian uses)

> they are either American citizens or on some form of US Visa. It would seem somewhat contradictory to be a non-Citizen protesting the policies of a US company while employed at that company on a visa issued by the US government.

With respect I disagree.

I don't think it is contradictory at all - you are not obliged to have unwavering and unquestioning support for whatever the company you happen to work for, or the country you happen to live in, decides to do. It is 100% ok to disagree with what your employer or country is doing.

Just because someone is on a visa, you think that they suddenly are not allowed to have an opinion or any morals? Please - take a step back and think this through from different perspectives than your own. Many people on a Visa/dual-nationals/US-natives etc still have full and complete lives outside of the US - friends, family, colleagues, property, cultural, etc. You cannot expect people to abandon all that and just go along with the US military killing innocent men, women and children outside in their home countries/near-by countries just because they have a job and/or are living in the US. [1]

1 - https://www.washingtonpost.com/world/national-security/after...

There will be no Google-comparable response. Amazon and Microsoft are very different companies and cultures compared to Google. Microsoft and Amazon have much smaller, diluted blocks of the culture that spurred the infighting at Google.

If Bezos says they're going forward with it, they're going forward with it. If a couple hundred AWS engineers mutiny, Bezos will gladly show them the door and replace them. That's how it would work at Amazon for better or worse.

Providing database solutions to the whole military is much more morally grey than helping the military build a ML model to bomb people more "efficiently"

At least, to me

Amazon already has a cloud computing contract with the government, https://aws.amazon.com/federal/us-intelligence-community/.

If there were to be protests in Amazon, it would have happened already.

Powerpoint is by far the most important piece of software in the military...used in planning for literally every mission from where to build latrines to mission planning for seal teams. Do you think Microsoft employees should strike over the use of Powerpoint?

I remember seeing video of ISIS actively using Google Earth images in order to coordinate their attacks, and they were not the only one in Syria using this Google product.

Google earth started as a CIA contract.

With a different company that Google bought

Should have given it to Linode.

Reply on news.ycombinator.com