Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.
Sure, we techy types prefer a proper database with a web frontend and an API, but that requires significantly more skill to build than an excel file.
No database has the flexibility and flat learning curve that excel has.
Until a database manages to meet these goals, it remains a good tool for certain usecases.
We as engineers should be working towards making a real database thats just as easy to use as excel, and only then can we complain that people are using excel sheets for this kind of stuff.
I know it's a pretty shitty program to use - excel is much more user friendly. If MS had made excel, but have all the features of access in it...
In PowerQuery & PowerPivot there are also no practical limits on the number of rows of data you can have (other than the impact on processing time, clearly hundreds of millions of records might start to be a problem). It's not quite access - it doesn't actually persist/store the data, just aggregates it from other sources which is probably what is required here.
Access is a little legacy and would have it's own issues. I think in reality it depends on how you are ingesting data (which is probably hospitals submitting excel templates or similar?). Maybe these could be automatically aggregated and put onto an analytics platform? (e.g. PowerBi). Hospitals could maybe report it on a portal, but I'm not convinced that the leadership would have wanted to change the established process (because of operational focus, change reduction & risk).
Weaknesses of Access in no particular order: very poor multi-user/multi-device-access features. No real concept of servers. Buggy, had a nasty habit of corrupting its custom file format causing data loss (Excel never does this). Does expect you to understand SQL, data normalisation, etc. VBA just about serviceable for a spreadsheet but not good enough for a 'real' database-backed app.
Probably someone could try and make a better Access. There are surely lots of startups doing that already, along with hosting. All the "no code" tools that are out there, etc. Access had the advantage of coming with Office so lots of people had it already, whereas today's subscription based services that host your data have incremental cost.
Base has very solid data storage foundations (including the ability to connect to real database servers) but the UI is very buggy and is painful to fight with. Note I said 'fight with' rather than 'use.'
Base seems to have a fair bit of potential, however, if someone where to pour some money and/or time into developing it further.
Test and Trace is a multi billion pound programme of work, involving huge companies (eg, Serco) interfacing with national governmental bodies (Public Health England).
We've had months to get something in place. One of the reasons organisations like Serco is used is their apparent expertise with IT.
(my one-step programme for improving UK public contracting would be that after a significant failed contract the contracting company and its directors and any other companies they are directors of would be banned from tendering for a period; maybe a couple of years?)
It also has a required certification for companies that work with the public that they weren't condemned for certain crimes (trafficking in illegal waste, extorsion, organized crime etc).
So, it doesn't seem an EU/Single Market rule that you can't check this.
Reasons vary but the ones I've seen are generally:
1. Complex procurement regulations that take a lot of effort to comply with by contractors.
2. Low willingness to pay (both price-wise and time-wise).
3. Civil servants are nightmare customers who don't know what they want and are under no pressure to figure it out, so there's a lot of timewasting involved.
4. PR nightmare if/when things go wrong even if it's not your fault, as government is relatively open compared to other types of customers so easier to find out about problems, and lots of people automatically blame the private sector in preference to blaming government employees due to (misguided) assumptions of moral superiority of the public sector.
Recently I was visiting hospital quite a bit (ante-natal) and witnessing the horror of the massive NHS form filling software the nurses and midwives have to use. It's clunky and sprawling but one of the main sins I think is it's lack of adaptability. In the old days of pen an paper, a new form could be written, or adapted, and photocopied easily, and on-site. Now if a new field needs to be added or a process is changed it needs to go (I imagine) to some centralised IT development office and fed into a ticketing system where it might get changed in a few weeks or months. There's no room to quickly adapt at a ground-level, so you end up with things like this excel sheet problem.
People working on Excel replacements need to remember about this aspect of spreadsheets too.
I'm not sure if you're being sarcastic or not, but in case you're not, this has meant c.16k people missed from having their contacts traced. That's time-critical. Delays of any number of days are very bad. People will likely die because of this error.
This is safety-critical software. It should be stress tested and have sanity checks. It should be impossible for this to happen.
For something like this system, part of a track and trace system costing billions, to fail because of a file size error is just unbelievable.
That’s always true though if the virus is spreading exponentially (dunno whether it currently does in the UK).
Yes, anything-sql or whatever would be better, but deciding between putting all the data into sql by hand, or "uhm... i'll think of something and do it tomorrow", me, being lazy, I'd pick the second option,...
...until i'd get shamed on here. Maybe even a few days after.
How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".
> The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.
You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.)
Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow:
- Transposed data (new entries in columns)
- Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by computer).
- US date formats mixed in with non-US date formats.
- Each day in a separate file, often with as little as 100 bytes per file. Thousands upon thousands of files.
- Random comments or annotations in numeric fields (preventing graphing the data).
We don't all need to be data scientists, or machine learning wizards, or quantum computing gods.
But come on. This is the most basic, data entry clerk level Excel 101 stuff. Trivial, basic stuff.
Put rows in rows.
Don't mix random notes into columns you might want to sum or graph.
Don't mix units.
Have headers.
Use a sane date format.
That is all.
</rant>
Or else it wouldn't explain why it took so many days to realize that they couldn't add more columns.
And then the same people wondered why things based on this file didn't work.
ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership.
(Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)
I still haven't seen an authoritative source, or one that predates that comment.
But it definitely sounds possible.
It's not surprising that this reaches into government.
Even if your canonical database is done properly, there will be a clash here as soon as you share (or give anyone the ability to generate) CSV files that are too large for Excel.
[1] they almost certainly are using fax somewhere.
[1] Japan's COVID-19 Reports - 140KBs of Unadulterated Incompetence https://stdio.sangwhan.com/wtf-japan-covid-19-report/
[2] The previous HN discussion https://news.ycombinator.com/item?id=22728674
Nope. They appointed Dido Harding to cough manage it, and Matt Hancock oversees it. That should tell you all you need to know. I wonder if they were running Excel 2007?
The article in question: https://www.bbc.co.uk/news/health-54387057
For me, I always feel like I'm living in some sort of parallel universe. Imagine living in a world where _you_, and a select group of others, know that there are better methods to drive a screw into a wall than punching it with your fist. However, 99% of people in the world punch screws into walls by hand, because they know how to use the only tool they have to hand (i.e. their hand). Their friends and neighbours do too. Their colleagues show them neat little tricks about how to fold your hand better to get the screw in with less pain, or more efficiently. The "proper" option -- drilling a pilot hole, using a self-tapping screw, understanding the difference between nails, bolts and screws, etc, or the concept of wall-plugs, is "too technical", "too complex", or requires "a large amount of infrastructure to fix a simple problem" [buying a drill & screwdriver, etc]. Sometimes people buy new walls with screws pre-installed because the thought of driving in those screws seems like an overwhelming technical burden.
You, and a group of similarly-minded nerds, understand just how stupid this is at times. You personally then spend the rest of your life watching people occasionally have horrific hand injuries from punching screws into walls and refusing the offer of screwdrivers. Others profit from selling vertical hard surfaces with a whole array of convenient screws pre-installed; or get employed teaching others the _very_ basics of screw-driving, or sell products containing all sorts of klunky hacks to cope with the fact that screws manually installed are at odd angles and tend to fall over randomly. The supplier of a popular brand of screwdrivers happens to be the world's largest screw manufacturer, but never seems to promote screwdrivers to people who currently punch them in with their fists. Instead, they sell them a subscription to screws, with the option for a very expensive pre-screwed wall-delivery service, built to your specification (that they call Dynamics for some reason…).
You, and a subset of your nerdy and educated friends recognise just how batshit insane the widespread use of punching-as-a-method-to-install-screws is, and occasionally laugh about it, mostly to distract yourself from the grim reality of buildings badly held together by punched-in-screws. Every so often a skyscraper-sized building collapses, [1] and it's realised in retrospect that the manually-driven-in-screws were not the right tool for the job. Still, people carry on banging in screws with their fists, and you increasingly realise that you have to -- somehow -- accept that in order to sleep at night without going insane.
[1] https://www.businessinsider.com/excel-partly-to-blame-for-tr...
https://www.youtube.com/watch?v=K_FrQnQv0Vw (probably an accurate depiction of what's going on in Whitehall right now, though)
You wouldn't (want to) believe how incredibly common that is.
Edit: oops, it seems they used columns for the cases and the ~16K column limit was hit when daily cases exceeded that.