Gov.uk site broke Google realtime analytics during lockdown announcement | Hacker News Reader

Gov.uk site broke Google realtime analytics during lockdown announcement

twitter.com

101 pointswhyleyc75 comments

There are a lot of things to complain about in the UK but the digital services bit of the UK government is really well run.

Congratulations on riding out the thundering herd.

Indeed. They also maintain a site dedicated to the gov.uk web design system, with free resources such as HTML components. https://design-system.service.gov.uk/

Also their infra (or at least part of it) is available in public on github https://github.com/alphagov/govuk-puppet

Wow! The team in charge of digital at UK Govt are rockstars! Simply amazing for a govt agency to be this good!

They surely make some EU countries look like amateurs. It's not usual to run digital services in-house like this in southern Europe, instead all this stuff is outsourced to consultancy firms that deliver unresponsive, buggy sites for absurdly high prices.

That was the usual pattern in the UK too, until GDS. (We used to have world class computing facilities in the past before outsourcing them, but that was pre-Web). Credit to those involved (and to Francis Maude, the Conservative minister who believed in the project and gave them sustained political cover - I am decidedly not a fan of the Tories but that was very good work).

Italian digital team has a similar website. https://designers.italia.it/

But I believe believe most Italian agencies or local governments (around 22,000) don't follow those guidelines yet.

"all this stuff is outsourced to consultancy firms that deliver unresponsive, buggy sites for absurdly high prices." it's a good description for MOST (not all) Italian Government web presence.

That's a cool site. I'm afraid Spain is not even close to having something like that, there's a consultancy firm epidemic here due to our flexible contract laws.

Never give up

Don't remind me... The german COVID vaccinations sign-up page being down every time someone even looks at it the wrong way has been a constant news topic over the last few days.

Also goes for stuff like election software, getting the basics of the basics wrong - not even checking user privilege on admin pages.

Government form pages often just fill out PDFs for you that you have to print, finalize and send off yourself.

Best not mention the COVID app.

That got farmed out to a consultancy with the ear of one of our glorious politicians..

I will shit on gov IT any day of the week but I'm not blaming GDS for that particular fire.

That didn't have anything to do with the GDS.

Plus the search tool: https://www.theregister.com/2020/11/26/govuk_tier_postcode_l...

(Worse of all, Auntie Beeb (aka the BBC) has operated a similar tool without issue.)

It's good, but in terms of content, it's impossible to keep up with the government's ever-changing position, lack of communication, and lateness to act.

The gov.uk "Brexit checker" is full of out-of-date information and plain broken links.

If Boris had given out at least a basic summary of what the new rules mean I would have stayed off the website.

A better speechwriter could have saved a huge amount of bandwidth

TLDR for below. Several people argue that Boris was totally clear, and then demonstrate their complete ignorance of the rules he so clearly set out.

The rules are: stay home, which is the take-away message and what he said.

Some people will always go "yes, but..." and look for ways around the rules, as we've painfully seen in the UK since March.

In fairness, some people have very good reasons not not stay home and the PM did also mention these reasons.

Overall I found the message and level of details well calibrated for a short, straight to the point, speech. Going into an exhaustive list of all the rules and exceptions would have clouded the message and would not have been a good use of the format and of time.

The only thing he announced, the only change, is that some (not all) schools are closing. The fact he spent 10 minutes and didn't quite manage to communicate which schools/kids were affected says it all really...

All schools are to close as he very clearly said.

This is not the only change. Overall this is back to full lockdown as it was in March, which is quite stricter that the current situation (aka tier 4 in most of England)

Edit: yes, children of key workers are still accepted although schools are closed, like in March. He said that, you guys clearly understood that. Let's not be disingenuous here...

I don't think that's correct.

They're remaining open for vulnerable pupils (not defined) and children of critical workers.

Also, nurseries will remain open.

They're also staying open for exams happening this month. But the summer exams are cancelled.

Also, schools remain open for "elite athletes" to train.

And the building will continue to be used for religious activities.

https://www.gov.uk/guidance/national-lockdown-stay-at-home#g...

All he had to say was "school is cancelled tomorrow, except if you went when we closed them last year or for kids under 4. Exams are still on this month."

I don't see any other changes at all actually. Weddings are still on, there are 101 exemptions for visiting people. The main one of course is anyone doing business. Workers keep working so that 30mil people many of whom will be moving around and breathing on each other tomorrow.

Let me know if I missed something? It seems next week will be the same as today for me (no kids).

All schools...except nursery schools. And this is only England, Scotland, Wales and NI all have separate rules.

Full lockdown like in March? I read that I should not go out more than once per day to exercise. In March I read that I must not.

> All schools...except nursery schools. And this is only England, Scotland, Wales and NI all have separate rules.

See, you perfectly understood the message. Nurseries stay open, all schools close, he did say as much.

Our local school is providing lunch tomorrow, although didn’t come out until after 10pm. However they are only taking key worker kids if both parents are key workers - unlike last time. We are, but a lot of unhappy parents on whatsapp tonight.

It’s not as simple as “schools are closed”.

I travel in to work sometimes from teir 3 to teir 4, but I was teir 2 last week. Am I still allowed to do that? I tend to go in to make sure things keep working for people at home. Is that OK? Only I have three people straight on to Teams asking if they should go in tomorrow. All have things to do that are hard but not impossible to do remotely. What should I tell them? They all saw the speech, perhaps they were not listening hard enough?

Come on, the speech was as clear as possible: there are no more tiers, all of England is in full lockdown, stay home unless you positively have to leave (including if it is absolutely impossible for you to work from home).

He said that in rather clear terms.

You answered your own question when you conceded that it was possible to carry out the work remotely and that your trips were not essential.

>(including if it is absolutely impossible for you to work from home).

If that is what you got from there speech then it was very misleading indeed.

> from .gov..you may leave home to...go to work, or provide voluntary or charitable services, if you cannot reasonably do so from home

That is a world of difference from 'absolutely impossible.'

I also did not say that my trips were not essential, that is just as big a misunderstanding as you made above.

Estimated that >1% of the UK population (769K) were concurrently on the site before GA topped out.

That doesn’t sound like that much compared with the normal loads that GA handles every day, though? Gold star to the UK gov team for their bit holding up so well though.

Its not. In a previous job I used to run fantasy sports games and one football World Cup we saw a spike sharper and faster than the one described on GOV.UK today due to two successive goals from a top player just as the half time whistle went. GA handled that spike just fine but if it wasn't for Akamai our site would have been toast.

I'm not taking anything away from the engineers behind GOV.UK though. They do an excellent job.

If folks think this load broke analytics please think again. Is there no critical thinking left out there in journalism? Do folks understand the scale analytics operates at?

Some quick questions

Did they actually pay for analytics 360?

If not, did they do 10 million hits in a month for a property? if so - they may be over the limit of 10 million / month.

More on differences here: https://blog.littledata.io/2018/02/28/google-analytics-360-v...

Analytics 360 runs 12K+/month from memory, but someone probably has more recent info.

What journalism might you be referring to?

I'm starting with whoever wrote the headline here.

Gov.uk site did not break google analytics. This is standard click-bait / shock value type headlines.

Well, it actually broke Google Analytics Realtime, since basically the report stopped working.

False - they may just be using the FREE version for "individual or small businesses". This has limits.

Google will stop analytics if you are on just the free account and start going over their free limits. This does not mean google analytics has broken.

Is gov.uk an "individual or small business"? Or are they paying for 360?

As I said, the report broke. Stop. As the headline says. And yes you're right that they "may be just using the FREE version", but it's a "may be".

It was a tweet written by the engineer working on their server.

No clickbait, not journalism. Just people sharing their experience, and a HN submitter who got noverexcited.

This might be awkward for some but can governments be compelled to use Foss software and where a Foss software is lacking features, fund for development of that feature so that everyone benefits instead of lacing more and more reliance on proprietary tech which inturn forces citizens to use that proprietary junk because "compatibility"?

“Out of the 17M page requests made between 8–9pm (111M if you include assets: stylesheets, images, etc), we served 21 5xx errors.” [1]

[1]: https://twitter.com/therealnooshu/status/1346419468151488512

Am I the only one who thinks it's creepy that the largest advertising company in the world, and particularly one based outside the country, has so much data on what UK citizens access on their own government websites for health, tax, social welfare information etc?

This having Google track your users has impacts internally when your own tooling becomes even less accurate because you have to start asking for consent: https://github.com/alphagov/content-data-admin/pull/707

In the case of things like page views, this is something addressable by just gathering the data yourself. If you put in the effort up front, it'll be more accurate than Google (as you don't miss anyone who Google misses).

Governmental sites should have everything self-hosted.

No Google Analytics, no CDN, no nothing.

Just like they shouldn't have presence in Private Social Media Networks. But that's my opinion.

>no CDN

Then the website would have burned to the ground during this announcement.

Not if they had enough servers and bandwidth.

This is an inefficient use of government money to in-source everything. If private industry can do something cheaper and more effective, use common sense.

At what point do you think that sovereignty outweighs price? I think that is the point that GF is on about.

There is a good case for some sort of in-sourcing when it comes to national systems. For example if, say, a Chinese diplomat toddled on over to you (let's say you are a US citizen and an important one) and asked you to use his country's foremost VPN solution for secure communications. Would you?

Now let's muddy the waters somewhat. Zoom? sounds a bit Anglo/Sax ... ish. It isn't of course. It also seems that Zoom forgot to encrypt their encrypted channels. lol.

There's more to security and stuff than cash. I do the boring old security thing for a living. It is boring but I like boring.

Of course sovereignty and overall data security should be considered. My point is that a blanket "no outsourcing" rule is daft and counterproductive.

Private industry by definition can't do something cheaper and more effectively because its goal is to make money, not to make a quality or cheap product.

If existing companies utilize cloud providers and CDNs rather than investing in their own in-house infra, doesn’t that signal to you that perhaps your analysis of the situation is flawed?

It’s possible the government is a big enough customer that they could get the necessary economy of scale to make it cost effective, but it also means that they’re going to need to develop a bespoke solution that all government services build on which itself is an expensive endeavor (and politically a liability since such a project can fail to deliver and just balloon in cost).

The government could rent our spare compute as a cloud provider, of course, to recapture expense. Then we can ask how dare the government spy on private citizen web traffic like that.

Let's imagine that I had to make a website that could host a million concurrent users once per year. What do you think is cheaper, me acquiring all of the hardware to scale to a million users and then leaving it idle 99.9% of the time, or outsource handling that peak traffic to someone who's already operating at the scale for it?

The UK government does not remotely need the same year-round capacity as a top-tier CDN.

They don't. The data is anonymous, and no user tracking is used

It should all be public information in the first place.

Some analytics information is public, if you know where to look. You can add /info to the start of the URL path, and that might reveal some information about the page.

For example: https://www.gov.uk/info/find-coronavirus-local-restrictions

That's very cool, I had no idea! How did you find this?

A while back, I worked on the code https://github.com/alphagov/info-frontend

Nice! Do you know if historic data is available somewhere too? Or is it always just the last 6-weeks?

General statistics on site usage would be interesting. But what about all the other data that could be extracted, both on a personal and national level?

You could, for example, determine which areas are more likely to access HIV information on the NHS website. Insurance companies would like that but would it benefit the public or be in line with people's basic privacy expectations for government services?

(I'm sure Google claim that they won't add a user looking at cancer information on the NHS site to their data profile but I wonder what other insights could be generated.)

> determine which areas are more likely to access HIV information on the NHS website

We can already do this. Have a look at the Public Health England "Fingertips" website.

https://fingertips.phe.org.uk/

https://fingertips.phe.org.uk/profile/sexualhealth

https://fingertips.phe.org.uk/profile/sexualhealth/data#page...

Not really, it will include who is accessing this data, which can be used to infer information on UK citizens.

USA has a dedicated website : https://analytics.usa.gov/

I don't know if the data is from GA or a custom made solution

Why did it break?

Interesting question. I wonder if Google has a scale speed limit for an individual tenant's sites, or perhaps each URL? It seems that the underlying gov.uk page remained up throughout though.

Usage quota for the service they pay for, presumably.

Is that much?

He said they got around 88k RPS.

I did a quick benchmark another day, I'm hosting my blog at the Digital Ocean App Plataform, in the free tier, used Locust with some VMs in Azure to stress test it, managed to hit 20k RPS and the only issue I had was that the my VMs at Azure were hitting memory limits...

Doesn't look impressive at all, weird that GA broke.

Reply on news.ycombinator.com