How we migrated Gov.uk notify to AWS elastic container service

You may be missing a point. Their PaaS was already hosted on AWS. [0] “with GOV.UK PaaS this was predominantly the AWS eu-west-2 (London) region, but some tenancies were hosted in the AWS eu-west-1 (Ireland) region”

The lack of “rapid growth” means few gov areas used it. Intended consumers just used AWS and competitors directly. Maybe in part for sovereignty reasons.

The blog post is saying they moved one of their services off their lightly used PaaS.

[0] https://www.cloud.service.gov.uk/migration-guidance/

It's pretty common, all the biggest clouds are USA or China owned.

In the UK government services go through information security classification to determine what level of security is needed, with the most confidential stuff still being self-hosted.

I assume most countries operate that way.

In the Netherlands, critical infrastructure is required to be hosted in government cloud data centers.

An exception is possible if after a risk assessment and the determination that no state secrets may be exposed, a government body decided to use a commercial cloud provider.

The private cloud providers list is then filtered by whether or not their country of origin / incorporation, or effective control, has an effective cyber-control program it runs against the Netherlands or against Dutch interests. This arguably includes corporate espionage programs.

It's pretty normal for ordinary government workloads in the UK, or at least it was at GDS. Using niche suppliers who cater to government paranoia is expensive, and they're usually much less mature than hyperscaler platforms. It's also open for debate whether those niche, inflexible suppliers result in a genuinely more hardened target or not.

You have to understand that buying computers comes out of the capital budget, and is several times more expensive than just leasing them for this year; and that hiring staff runs into severe civil service pay issues. Once "buy some computers and hire staff to manage them" has been ruled out by politics, buying hosting on the open market becomes the remaining reasonable choice, and nobody got fired for choosing AWS.

Usual: very

Good: Not so much

Unfortunately, cloud provision isn't very competitive and is very US/China centric.

I was at a talk recently around how one of the UKs major infrastructure providers was building their architecturrle, and I was pretty freaked by the level if vendor lock in.

Would love to see more governments viewing this as the security risk it is, but I'm not holding my breath.

Notice that each major cloud vendor has dedicated gov regions...

So I guess the tie is larger than it seems at first sight.

Ecs isn't exactly tying, because ultimately it's still docker containers, so moving out wouldn't be a tricky prospect. A cloud agnostic solution though would likely mean k8s and bring with it much more complexity and overhead (and is also a form of lock in).

I can’t speak for .gov.uk but in .gov one of the big factors is staffing and support costs. If I need to store some stuff, maybe I am looking at S3 and Minio. If I decide that I want a full open source stack running on my own Linux boxes, that’s a super valid reason but it immediately increases my ops staffing requirements considerably: not just to keep the service running (minio is hardly needy) but now I have to do all of the mandatory security and compliance stuff which AWS handles – failover and backup testing, hardware security (building access, shredding disks, firmware updates, etc.), infrastructure and software security setup and auditing, capacity planning, etc. I have to do some of those things with S3, but it’s multiple orders of magnitude less work and regulated environments often have policy challenges (e.g. slow change management processes, mandatory use of tools not intended for your situation) which AWS’ much larger team doesn’t even have. Again, there are reasons to consider it anyway but if your goal is to show a new capability you probably don’t want to first spend a year or two on infrastructure nobody can even see.

The other thing is the ease of replacement versus the cost of abstraction. ECS is a very lightweight service (as are many things like SQS, RDS, etc.) and if you have to migrate to something else that part of it will be the least of your worries, but unless it’s highly likely that you’re going to switch cloud providers you are likely to have many years of not needing to spend time running an internal cloud. Similarly, unless you anticipate switching on a hard, near deadline it’s likely that the effort of porting later will be less than the time you’ll save in the years before the mandate comes.

That’s a room full of people who can be working on the things your users actually ask for rather than treading water, especially because one big challenge for an internal PaaS is that you have to build a ton of services to even be an option: if a project needs one feature your PaaS doesn’t offer, they’re going into AWS and you get none of their business. I suspect they hit that trap of just not having enough resources to stay competitive with what their users needed.

Given that hyperscaler cloud providers can be in the 5x-11x cost increase territory; and AWS/Azure are definitely guilty of lobbying governments.... is your comment sarcastic?

EDIT: downvotes? What did I say that's untrue? Are so many people really employed by these hyperscalers that they go around downvoting things against them? Calm down- you'll have a job.

Oh, we can still outsource like the best of them, just look up the NHS Track and Trace app to see how badly we can do it (or line the pockets of certain people, whichever you prefer).

A foreign hosting provider like AWS is the opposite oft self sufficent

Unfortunately this very application, Gov.uk Notify, is currently being used by Councils to send emails to residents directing them to an outsourced company's website, https://www.householdresponse.com, to input sensitive details about where they live.

The emails are phishy to the extreme and there's no indication or way to verify that it's an official service. See for example https://www.bleepingcomputer.com/news/security/uk-gov-keeps-...

While some parts of Gov.uk are done well, there are still terrible practices everywhere due to cheapness and ignorance and presumably because the Gov UK people can't do everything, unfortunately, even though it would be cheaper and better if they did.

Anyone know if there is any way to find out what PAS cost the government?

This is a notification service. They sent 2.6M emails, 2.8M text messages, and 60K letters yesterday [0]. That’s about 30 emails per second, 32 SMS per second, and less than one letter per second. That’s not nothing, but it doesn’t need crazy processing efficiency either. Most of the work will be I/O bound, just messages sitting in a queue waiting for the receiving service to be able to accept them. Python is fine for this. You don’t need email to be sent ASAP; in fact a lot of work goes into making sure you don’t send high volume too quickly in case you look like spam.

[0] https://www.notifications.service.gov.uk/features/performanc...

Why do you think it’s great?

And how do you know its high-throughput?

The throughput is described as "thousands of requests per minute" – with modern hardware that's likely not something you'd even have to think twice about. It would probably run happily on a laptop!

It used to run GOV.UK PaaS which was decommissioned last year, all government services had to find new homes - mostly in Azure or AWS.

https://gds.blog.gov.uk/2022/07/12/why-weve-decided-to-decom...

facts.

anyone intentionally migrating infrastructure to the cloud in 2024 hasnt seen the bill, or is spending the taxpayers pound.

"Gov.uk notify" isnt a critical business service. it doesnt need multi-az or multi region failover. running a docker container itself isnt a feat of achievement.

you could save money over the longterm by running a pair of pizzaboxes in Cardiff and Edenborough running orchestrator. hell, you could arguably run notify in a pensioners basement off a pi powered by a solar panel.

We see this argument anytime a cloud migration comes up and it’s a severe simplification.

Having your own datacenter requires someone to manage that hardware. There are costs for the space you are in.

But you also have to have the hardware for your peak utilization. If you are either very bursty or have significant times of little to no activity you should be able to scale up and down.

Yes a simple 1:1 is likely going to cost more (depending on your scale, if your Cloud bill is under $200 not sure you could really do much better) but then your not utilizing the cloud to its advantage.

That is before going into the savings with things like spot inferences for processes and don’t have to be real time.

Yes this doesn’t take into account what else they might be running to share resources with. But not every organization is going to be running a bunch of different workloads that can fully utilize their hardware at different times