Hacker News Reader

My suggestion that governments shouldn't rely on third-parties for their infra, didn't go down well with HN:

https://news.ycombinator.com/item?id=25639520

FWIW, I agree with your opinion. That includes not letting politicians onto these websites to do campaigning. This was a big mistake. The number of former public servants who have gone to work for Facebook is alarming, IMO. The number of possible future public servants who have worked there is perhaps even more worrisome.

Don’t forget Apple, Netflix, Twitter, Google whose board members, “senior policy executives” are all ex political staffers or cabinet officials.

It's been a revolving door for a long long time

The question is how do you fix that? We should want domain experts in a lot of these government roles, but the primary way you become an expert in an industry is working in that industry.

Let's image we put a ban in place to prevent this revolving door. Why would someone working in the industry be willing to forfeit their post-government career, especially if they are a political appointee and might only have the role for a few years until the next election?

That is a huge disincentive to participate in governing and will likely lead to more career politicians and bureaucrats which isn't exactly an ideal situation.

I think for starters it's important to recognise that the expectation of a "post-government" career is itself a part of this problem. If we instead built long term competence within public institutions, including competitive remuneration and career development like training programmes, there wouldn't be a need to have a door between the private and public sector quite as much in the first place. The reward should be delivered in situ for continued constructive service, not be an upgrade to a private sector gig after holding your nose and/or making decisions you feel will be looked upon favourably by future employers.

Private and public sector employees could collaborate where it makes sense, just like companies sometimes collaborate on shared initiatives like working groups or open source projects.

Government can actually work better than it does in the US -- as it does in other countries -- but one must first accept the ideological premise, which seems to be what holds the country back.

And working conditions. The state often can offer better working conditions. Things like high pay and fixed hours - does not have to be a fantasy salary before it is very attractive to people with families.

> If we instead built long term competence within public institutions, including competitive remuneration and career development like training programmes, there wouldn't be a need to have a door between the private and public sector quite as much in the first place...

That sounds great in theory, but the problem is that many of these leadership roles if not elected directly are nominated by the president and therefore they are political in nature and those politics can flow down through the organization.

For example the FCC chairman, Ajit Pai, is resigning this week. Would he have resigned if Trump was reelected or would he have been re-upped for another term? Is Biden obligated to find this Republican a job somewhere else in his administration? Is Pai just supposed to retire at 48 years old? Can he go back to working for Verizon like he did for a period earlier in career? How many other people at the FCC have or soon will leave the agency due to the change in leadership?

> That sounds great in theory, but the problem is that many of these leadership roles if not elected directly are nominated by the president and therefore they are political in nature and those politics can flow down through the organization.

That could be changed though. It's not how it works here in the UK. The ministers in charge of departments are political appointments, but we also have a substantial politically neutral civil service with lots of headroom for career advancement. (politically neutral to the point that you are not allowed to publically express your political opinions - posting them on social media could be a firing offence)

The US also has an apolitical civil service infrastructure, just not at the top of these agencies. And it is the leadership that people refer to when they worry about the revolving door. No one cares when some low or midlevel USPS worker goes to work for FedEx, they care when the former FCC chair Kevin Martin is tapped to head up Facebook's US Public Policy.

Well yeah, half the people on here seem to draw their salaries from selling cloud infra

Working for a third party that provides govt infra, we/my org see it as a license to print money and an opportunity to lock govt in. Really perverse incentives.

> we/my org see it as a license to print money and an opportunity to lock govt in. Really perverse incentives.

That's so fucking wrong, but I bet the government continually falls for this trap.

They pat themselves on the back for falling into it. Congratulations and drinks all round after you've signed millions over to big blue for a glorified application level firewall.

It's already in the trap. Very hard to justify the $$$$ to be platform neutral when sales reps are so good at pushing the savings of jumping in fully with one SaaS.

Your idea is honorable, but do governments have the know-how to do it? It's not only about costs and responsibility, I don't think this could be feasible in most countries, maybe only in China and US.

If they paid better, sure they could. That would require more taxes though.

Would it though?

Compare other hard problems and how they have been handled. Healthcare comes to mind immediately.

Lol this is kind of funny given my other post but in addition to having worked in government I’ve also worked in healthcare.

I made about 50% of what I do now at that vendor and was glad to leave. I truly think if people had an incentive to make things better in government and healthcare then things would get better. It was a terrible job but a good starting point. I wonder if health care vendors paid as much as google or Facebook how good things could be.

In fact there are a few people that do try to make things better, but they value their progress over money. Same with teachers. We know they make a fraction of what a tech worker does but the value they provide is immense.

I don’t know if you’ve lived on the west coast and worked in tech but if you have surely you know someone who makes an absurd amount of money at Facebook, or google or whatever and also hates what they do. Imagine if you could pay people faybu bux but to make government work. They would do it. There would be an impassioned soul who is want to make government better and would do so. Instead the government pays half rate and gets half rate old dudes who want to retire in place.

A former Googler got called in by the Obama administration to try and put healthcare.gov back on its rails after the disastrous launch, and the things he learned about it were fascinating. He talks about some of them here

https://youtu.be/0albm_hhQzM

It's not only the pay rate attracting employees who prefer stability to income... It's an entire decisionmaking ecosystem centered around minimizing blame coupled badly with compartmentalization. The end result is employees are heavily incentivized to do things that minimize risk not for the entire project but for their individual department. couple that to the percentage of the work that is done by third-party vendors under strict contracts written before the full scope of the project is understood, and it's a bad recipe.

One of the first problems the embedded team identified with the whole design of healthcare.gov was that every team had responsibility for its individual component, but nobody was empowered to have responsibility for integration. It was possible for every subcontractor to satisfy every bullet point of their contract and the result to not actually be a functioning healthcare exchange website.

I’ve experienced this first hand. I really think more money would solve this problem.

If the departments weren’t structuring themselves to survive indefinitely out of fear, then we could engineer in government.

> I don’t know if you’ve lived on the west coast and worked in tech

No. I don’t live in the US and don’t work in tech. I work in healthcare and have done so in the private sector and public sector. I don’t think you have pay a an amazing amount to get good people. Decent conditions go a very long way. My public sector pay isn’t worse than my private pay, and depending on how you measure it, the public pay may be better.

Would I be right in thinking that your healthcare experience was all private sector?

Definitely all private sector in the US. That’s almost exactly what I’m getting at.

Not especially. Government has largely predictable load, and hosting on third parties is eyewateringly expensive (only approved suppliers, who overcharge).

In jurisdictions I’m familiar with it would require (nearly impossible) reform of the bureaucracy to make it possible.

I was involved with a replatforming project in a state government that failed expensively.

I lasted six months before I left. It was infuriating and depressing.

I think with enough money the nearly impossible bureaucracy reform could be possible.

This is just a priority problem. There are plenty of tradeoffs that could be made to fund this kind of thing.

Fact is computer and software, services lobby groups want the business. In the US, said business can be bought, and is.

> Your idea is honorable, but do governments have the know-how to do it?

Yes. Governments hosting their own servers is still the norm if anything, the same is true in plenty of large companies. It's not exactly black magic, especially with modern technology. Outside the HN bubble the cloud hasn't really taken over.

> I don't think this could be feasible in most countries, maybe only in China and US.

Smaller countries are easier, there is far less scale to worry about and computers are powerful enough that most government services could just about be run on a single server (in practice you probably wouldn't).

Can you expand on why you believe that (and why you don't find the counterarguments convincing)?

In particular, why isn't this concern addressed by the government negotiating a more favorable contract than the standard terms of service?

(Alternatively, why can the government not address the problem with legal means? In the US, the Defense Production Act can compel private companies to accept and perform government contracts.)

For a valid reason. The government isn't a tech company or an infrastructure provider. You wouldn't ask the government to build their own roads or bridges, so like everything else software and services need to be contracted out as well. Of course that doesn't mean they should use AWS by default, but asking every single government agency to build their own data centers and CDNs is foolish.

> You wouldn't ask the government to build their own roads or bridges

Bad analogy, because we're not talking about the building of bridges (software), we're talking about where they are built.

In your example, the government surely wouldn't pay to have a bridge built on private property where access could be rescinded immediately if the land owner was so inclined.

Why is the assumption that three letter government agencies are signing the same contract with Amazon as a random startup? Amazon built an entire new instance of AWS just for government use. I'm pretty sure they wouldn't be able to deny access on a whim.

Who builds your roads? You contract out road building because it's too difficult for the government to do?

Where do you live?

In the states most roads and infrastructure projects are built by private companies that are contracted to do so by the government. There's offeten a public bid process between multiple companies to decide who gets the contract.

That seems like the wrong type of job to outsource. You outsource for expert help. This can be done in house with more control over when work occurs to sync with low traffic volume times.

I led cloud transformations at a Fortune 500 and we did make sure to look over contracts and have business agreements in place. YMMV though as we had some weight behind our needs as a government contract would. If you just sign up with a credit card I’m sure it’s different as mentioned however, a lot has changed since 2010.

They were warned repeatedly and ignored it.

Did you have any "disaster recovery" plans in place, what you would do if the cloud failed.

I think looking at this issue with the example of "Parler" may be missing the point. The issue worth debating is surely not whether Amazon made the right decision. Of course they did. The issue worth discussing is who gets to make this decision in light of what the decision implies. Due to how much businesses are choosing[1] to rely on companies like Google and Amazon as service providers, these "tech" companies can easily kill other businesses by denying service, without violating any laws in the process.

These "tech" company middlemen are generally not treated like utilities under existing laws but the question is whether they are being treated as such by customers, the public.

1. Perhaps not so much consciously choosing as being persuaded by tech company salespeople and, IMHO, biased "media coverage" (=hype).

Disaster recovery was a hot topic. Most transformations kept DR within the cloud. A few architectures wanted to fail over to another public cloud (AWS->Azure). The issue of “de-platforming” still exists. Most wanted to get rid of the on-prem costs without keeping a rack around for DR.

You’re absolutely right that the real meat of discussion needs to be how a cloud company like AWS makes these decisions? What criteria? What protections does a consumer have? What about data?

There’s a lot of unknowns there. AWS’ Business Agreement you can get their legal and your legal to agree upon, doesn’t really touch on all those points. Further, AWS’ has a canned agreement they use and don’t make concessions even if you’re a Fortune 500.

Enterprise customers, of signficant volume, do indeed get custom agreements. (Or at least, they used to be able to do so.)

Are the terms confidential. Can customers "compare notes" to see if they are getting less favourable terms than others.

They all include a NDA about the agreement itself. So yes, the terms are confidential. Even the fact that a particular that you may or may not have an agreement at all is confidential.

Are the terms of the NDA confidential.

With respect to the fact of the existence of an agreement being confidential information, usually an NDA will only apply to information that has not already become public through no fault of either party.

Every Master Service Agreement (MSA) I’ve seen has had a ton of legal back and forth. To say that they all include NDAs is a generalization which isn’t even mostly true in my experience.

Gov is pretty sure has a far stronger leverage than AWS. For example, if AWS cancels unilaterally Gov contracts, Gov will show that as Exhibit A of the need of Anti Trust.

Anyway, if Gov can procure cloud services, they should procure multiple providers and design their service as a cloud agnostic system.

The government can shut down Amazon.

Amazon can't shut down the government.

Stop delivering packages for two weeks, blame the govt...you'll find out very quickly that brown boxes are a real vector of power

Nope. Shut down every other website and blame the government, on the other hand...

Are there enterprise contracts that do not have provisions under which the supplier can end the relationship for cause?

I have a hard time believing the lawyers for a supplier would let them sign a contract that requires them to provide service to a customer under any and all circumstances with no way out.

Edit to add: if you stick servers in a colo data center instead of signing with AWS, doesn’t the data center contract give them the right to evict you under certain circumstances? If you lease office space, doesn’t the lease usually have terms under which the property owner can evict you for cause?

Analogies (such as leasing data center and commercial real estate) should probably account for the availability/lack of viable alternatives. For example, how much choice, what options does the consumer have in selecting its "service providers".

> If you lease office space, doesn’t the lease usually have terms under which the property owner can evict you for cause?

Indeed. But almost invariably you’ll have to go to a court to do it.

Minimum notice is pretty common by law, for one thing.

Building off-prem is one of the biggest boondoggles in the history of our industry.

Unless you stick to S3, EC2, or EKS, you wind up designing your infra for freaky weirdware architectures like Lambda that won't run anywhere else.

Your costs are double what they would be if you just had a platform team imaging and updating servers. Costs can grow unexpectedly, too.

Why anyone is using this trap is beyond me. I guess since folks spend their two year tours of duty at companies before rotating off to some other position elsewhere, they just don't care.

This is Oracle all over again.

s/cancel/terminate

https://www.acquisition.gov/print/11832

(I am not knowedgeable when it comes to FAR, maybe someone reading HN knows more.)

> When people debate the merits of "on-prem" versus AWS on HN they rarely focus on the contract terms.

if this had been electricity or water, instead of AWS compute resources, the tune would've been different.

I think the gov't just need to catch up with regulation - that infrastructure/utility providers cannot terminate without cause in the same way that an electrical utility cannot terminate a premise of electricity abruptly and without cause.