Hacker News Reader

Comment by goodmachine | Hacker News Reader

>To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time.

Something of an understatement. The project was red-flagged as'undeliverable' in 2019, after spending £154m

https://www.theregister.com/2019/07/18/verify_to_be_flagged_...

By 2020, and now at £200m sunk:

https://www.computerweekly.com/blog/Computer-Weekly-Editors-...

So now in 2021, we're back to informal tests and Post-It notes again?

View on HN

> To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time.

This is a misleading paraphrasing of the article. With the full context, it becomes clear that it's specifically inclusiveness / accessibility that they're saying they didn't get right:

> Inclusion is a hugely important part of our work, because anyone should be able to prove their identity to access government services, and it’s often the most vulnerable people who are at most risk of being excluded. To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time.

This means that the rest of your comment doesn't follow:

> Something of an understatement. The project was red-flagged as'undeliverable' in 2019, after spending £154m

The second sentence could be right or wrong, I don't know. But the first half doesn't make any sense in the (actual) context of what you're quoting. If it's understatement then that means that they got accessibility very wrong.

Only £200m is impressively low for this type of failure. Canada's federal government is on track to waste more than 2.2 billion dollars on a pay system that does not work and needs to be replaced with another billion dollar system ASAP.

https://en.wikipedia.org/wiki/Phoenix_pay_system

> provided by IBM in June 2011

In Czechia we have a system where all sellers have to log bills to prevent tax evasion. IBM also provided the system. It cost ~ 15m$ to set up and ~15m$/y to run.

Given the rate of ~millions of bills submitted per day, of which the government only gets the total price and taxes, you have like ~hundred bills per second.

I could scale a gunicorn+sqlite on my 2014 MacBook Pro to be able to receive this rate of requests with a payload which could fit inside a single TCP packet. Sure, there is auth/backups/analytics... etc. Yet, I still just do not see how could IBM charge the extra 14m$ to set that up...

Based on my limited experience in govt projects, there is a lot more to such implementations than what meets the eye. The number of backend systems this application would need to integrate with, the long list of compliance requirements that adds complexity to development process and setup, redundancies that need to be setup at different levels, odd technical requirements mentioned in the RFP to name a few are things that can bloat up the cost and implementation timeline. Now add dependency on any parallel development or upgrades to the mix and you have a very tough project to execute.

I can guarantee you that there’s like 1000 different idiotic rules that are applied to each of those items. They probably need to be stored on servers stored behind balistic glass, and cooled with helium.

I’m not saying IBM is not still charging the government 14x more than necessary, but at least some of the expense is likely their own fault for having ridiculous requirements.

For one, it pays an army of "analysts" creating all kind of "documentation" nobody ever reads.

The German motorway toll system which was planned to take less than a year to roll out and ended up taking more than two years longer than planned is also a good example.

I don't know how it worked out in terms of budget, but the company ended up in a dispute over billions of euros in damages, mostly from lost revenue.

German Wikipedia source: https://de.wikipedia.org/wiki/Vergabeverfahren_zur_Lkw-Maut_...

Here's a thought experiment: suppose that you have a fledgling government somewhere that needs to digitize and can start fresh with the modern day technologies, as opposed to having their hands tied with millions of man hours that have been previously spent creating legacy code.

Suppose that they decide that every single governmental system will use the same tech stack:

  - front end: Vue and JavaScript, or whatever's popular and easy to use
  - back end: Java with Dropwizard, or whatever has decent performance, decent maintainability (static types) and isn't too slow to write code for by the average developer (as opposed to something like Rust or C++)
  - database: PostgreSQL, or whatever is suitable for its needs (though open source, so otherwise MariaDB could be considered)
  - infrastructure: x86 servers, all running something like Ubuntu LTS or another popular, relatively stable distro
  - communication: REST everywhere with OpenAPI, due to their abundance

In addition, perhaps there are some demands for the processes themselves:

  - all of the code must be open source
  - all of the discussions about the code and requirements must be public (a la GitLab's approach)
  - all of the services must follow 12 Factor App principles, have fully automated CI and must run in containers (say, Docker or anything OCI compatible, with a simple orchestrator, like Nomad or K3s)
  - all of the non-trivial services (e.g. files, reports etc.) must be separate, so not exactly monoliths, but not necessarily Netflix level of microservices
  - all of the services must have at least 75% method code coverage, separation of concerns, all methods longer than X lines must have comments explaining what they're doing and *why*
  - all of the services must use dependencies that are not older than X months, checked weekly
  - all of the above can be checked by a bunch of shell scripts and the CI pipelines will fail if the goals are not met

So, with all of that present, how could any of that be better than our current approach of outsourcing, developing closed source projects poorly, not knowing who to know accountable and not learning anything from these due to a lack of post mortems? Would a strictly defined and consistent tech stack be a good choice or a bad choice? What about the job market and running a single stack in prod, across numerous systems? What about mandating low level technological decisions, as opposed to trying to sell everyone on some abstract business framework that has nothing to do with the end product?

Short of powers that be wanting to line their pockets, for what other reasons would the above fail? Why wouldn't open source and strict limitations of what can be developed and how it must be developed work?

Context:

In Latvia, we have a system called "E-Health", there have been around 14-15 millions spent on developing it so far, however the project has been widely regarded as a failure: https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e...

Now, they're planning on writing a new system instead, however it feels like they'll probably make the same mistakes, due to numerous companies having had their hand in the previous system's development, with no good leadership and bad technical implementation: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...

Surely there are both social and technical decisions that can be made make projects more successful?

I think if you have a look at the service manual[0] you'll be surprised. New code is open by default (unless there's a security risk) and while the stack beneath the surface isn't predefined the frontend is[1]. There's also the GOV.UK PaaS[2] that is geared towards making a fully automatic pipeline an easy process - it allows anything from static sites to Docker containers to be deployed with just a few lines of config.

All services need to go through an assessment where their openness, usability, accessibility and security are reviewed.

It's a modern, sensible approach that's constantly being improved.

[0] https://www.gov.uk/service-manual/service-standard

[1] https://design-system.service.gov.uk/

[2] https://www.cloud.service.gov.uk/

Yes it probably is only the beginning. The finnish health digitalization project Apotti which currently covers only 10 municipalities or so is already said to have cost over 700 million euros. On here we always point to baltic countries and say look what they can achieve with only few tens of millions.

Um ridiculous government waste!

We could have done it all in $300K and so could have many other startups

You could deliver a single sign on system for 300k based on your requirements. Unfortunately you have to deliver it to the customer's requirements.

If you go around quoting for projects without seeing the specifications outside of HN comments then your business is doomed.

That’s not how it works.

If you make a system that is secure and flexible enough, you then work with other stakeholders to integrate with it over time. They each come up with a roadmap to migrate their systems to support your protocol.

The key is to skillfully negotiate the budgets and politics between teams, so as not to get stuck building the next healthcare.gov.

Government projects always go to the lowest bidder! You’d just fail faster.

Sounds like quite a naive statement. The scope of this project is huge. there are so many different systems at play here. Some are so legacy they go back to Tudor times.

Still, I could have attained an “undeliverable” state for far less money.

I don't know what they were covering, but there is a multitude of different systems from local councils up to central Gov.

GDS is full of mostly django based systems, some legacy ruby and that's just the stuff from the last few years.

Going back further there are various C# systems, the amount of separate systems is staggering, it's not too surprising costs add up.

The problem in these environments is often not technical, but human, full of egos, people without competence wanting to be involved in every discussion and generally people talking and evaluating systems based on specs for a year without actually testing things.

For every one of the systems you mentioned there is a fairly straightforward solution. And even if you can't solve them all you could gradually migrate everything.

Oftentimes because of management wanting to push liability to a private party, they don't want to invest into local dev teams, but instead want to buy a big enterprise solution.

The VA, NHS and the German systems all have these issues, but it's not because they're public institutions, it's because they work like enterprise.

Definitely not mostly-Django. There are probably more new Rails than Django projects these days.

$300K doesn't get you much. Perhaps $10M though..

Hmm, it gets you me for 3 years or so. I can do quite a lot in 3 years if I have zero communications overhead.

Reply on news.ycombinator.com