Hacker News Reader

Comment by callamdelaney | Hacker News Reader

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

View on HN

The common case for stories like this is the answer is "nobody."

Time point 1: "We need a way to map the long-form epidemiology state. It's 2D data... Let's use Excel." "Will we hit limits?" "Pfft, no. Not unless we end up needing to track tens of thousands of patients, and that's not practical; we don't have enough people to do that tracking in the whole NHS."

Time point 2: "This pandemic is an URGENT problem. We need to hire more people than ever before to do tracing. And be sure to keep the epidemeology state mapper updated so it can drive the summary dashboards!"

Have you every worked in a large enterprise?

I have personally tested an Excel based credit rating tool, to be rolled-out globally by a major financial institution.

The fact that one should not do this, is by no means a reason not to do it.

Fun fact: Toshiba has software that generates entire CRUD app skeletons driven 100% by excel spreadsheets. You fill out object definitions in excel spreadsheets with property names and data types etc. It's like an extremely uncool long lost cousin of rails -g

Source: Got one of their engineers to show me after I heard about it and had to know if it was real.

Anyone who's ever worked with or in Japan knows just how far they are willing to torture Excel spreadsheets to get them to do anything and everything.

I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is.

It was one of my nicest testing gigs ever. A test session would take only minutes, all results documented to the t. And fiddling around with the test data was so easy. Would be interesting to know if this thing is still in use.

Not sure the customer liked it that much, I regression tested the 6 previous - still running - versions of the service, something no one had cared to do for years. We found bugs both in the Spec and in the Code for nearly all old versions...

>I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is.

That's an upgrade on how Toshiba did testing. They did basically the same thing except it was Excel+Humans. No joke. Never saw that one with my own eyes but a coworker did. He also said they had doctors there and each doctor was responsible for X number of staff and every so often made them fill out a questionnaire of which the final question was "have you thought about killing yourself lately?" and if you answered yes you got a day off. Apparently people would ask for transfers to different offices where the doctor responsible for them wasn't physically present, so they could avoid even being asked the question for fear of being made to take time off work. One of the more senior guys on the project kinda just disappeared too. Fun times.

to play devil's advocate, i had a desktop support job in college and worked with an executive assistance. She was _very_ attractive and so i was at her desk "working on the printer" a lot, she knew Excel better than anyone i've ever seen (before or since). It was like watching a magician.

In the right hands Excel is pretty amazing.

I once created a very modern mobile offline-capable web app for a large enterprise company that was powered by APIs backed by excel. Luckily we didn't have to deal with those directly, but it was a bit of a pain for the poor guy doing th backend.

Would be interested in hearing more about that Excel tool.

It started very innocent: Some consulting company apparently was tasked to create a prototype of the credit rating algorithm. Quite appropriately, I would say, they did so in Excel. As you can easily tweak the logic, show input, intermediate results and output. And all can be manipulated by the client who does not need to know a thing about programming to do so. Great.

From there it went downhill: Apparently someone up the hierarchy thought "Wow, that's 80% of what we need, let's just add a little UI and ship it"

They added a UI, but not as a VBA UI as you might hope. No, they did the whole UI in different worksheets. Storing any intermediate data ... also on worksheets. Long story short, it was a mess, and a slow one for that.

Oh, did I say this was a multi-lingual application?

Production rollout was on Jan 2nd, Dec 31st around 4pm I found a bug in the other language, on the one machine which had the this language configured. I believe this was the only time I ever saw a programmer literally run down the office to that machine to debug this issue.

Come work at a bank. Any bank.

Can attest to this. Some of the things I've seen excel spreadsheets contorted to do horrify me.

I once got spreadsheet dumped on me to debug because it wasn't working. A colleague used the spreadsheet to 'generate' interest rates that were then input into a mainframe. Dug into the VBA spaghetti mess, turns out this 15 year old script that took 15 minutes to run originally hit a bunch of Oracle/External APIs and performed calculations was now just copying the rates in from a csv file on a shared drive.

The error was caused by an excel formula ticking over a new year causing it to look for a directory it did not need to access that didn't exist. I thought it was pretty funny until I heard that the entire asset finance business had been unable to write any loans for days because of this.

In my org excel might have been the start, the intent being to move over to a DB once someone got it stood up. Then nobody would stand up the DB - because not urgent, we have the excel for now. The DB would be completely forgotten, and 3 years down the road everybody would be wondering wtf we have this huge excel file for. Then we would have six weeks of meetings to discuss the file until something else comes up and we forget it for another 3 years.

That's why Excel wins. Because it's right there, infinitely adaptable to any change in workflow you may need to make. No need to send official requests through half a dozen layers of management to get the IT people to maybe, in half a year, add a column to the database. No need to involve 20 people for a year to create new software for a new workflow. You just agree with your co-workers on a change, swap a column, add some color, File->Save, done. It wins because it puts power in the hands of the people doing the work.

I mean, yeah, but pretend Excel is, idk, sword arms. You get a lot of work done by whirling around the office, spinning your swords, cutting off a variety of limbs and heads in the process. You get to the end, meet the few survivors, staring dismayed at the trail of blood and dismemberment behind you, and you yell, "BUT I GOT THE JOB DONE!"

You're not wrong, but...

I wish there were some alternative where we instead fixed our ossified, byzantine processes gradually over time, so that we didn't need to break out the sword arm tornado in the name of getting work done and then, in hindsight, say, "well, shame about the negative externalities, but there really just was no other way. Ah well, let's move on, everyone who isn't a pile of flesh and blood and bits on the floor tidy up. Gotta get everything clean for the next sword arm tornado."

But the negative externalities don't seem all that big, considering. For every problem like this, you get countless millions of person-hours and dollars saved in work that wouldn't have happened at all, or would have been delayed, because of the necessity and cost of creating complex IT systems to support it.

If Excel is a sword tornado, it's one happening in an environment where everyone knows to be super vigilant about sharp objects. The alternative then would be a central processing factory that takes several years and millions of dollars to built, and which has to be turned off for a month several times a year, to change the shape of the blades used by the automated cutters.

> For every problem like this, you get countless millions of person-hours and dollars saved in work that wouldn't have happened at all,

I've found errors in every Excel spreadsheet I've ever looked at, and I'm not some master-excel user; I usually find them because - if I care about the results, I rewrite them as a Python script, so I get to go through everything.

The fact that Excel is effectively not auditable is a huge problem. There are a lot of dollars saved, like you said, but also many wasted or embezzled without anyone noticing in time because past a very low bar of complexity, it's really impossible to figure out what's happening without much, much work.

> If Excel is a sword tornado, it's one happening in an environment where everyone knows to be super vigilant about sharp objects.

... rather, no one is really careful, everyone gets a small cut every now and then, to which the apply a bandage and continue like nothing happened. And occasionally, they lose an eye or a limb or a head - and it is only those cases you read about in the newspapers.

I do not have a better suggestion, I'm afraid, but Excel is causing damage everywhere - e.g. [0], the subtitle - which I'm afraid is not an exaggeration, is "Sometimes it’s easier to rewrite genetics than update Excel"

[0] https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

I agree with what you wrote. It does cause a lot of small mess everywhere, and an occasional big mess. But there really isn't any better suite of tools available that hits all the non-database requirements.

Alternatively...

"We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it"

"We should maybe start tracking some of this" "Where does the data come from?" "Dunno, a handful of hospitals... maybe?" "Right. Let's just use Excel, not gonna waste time on that now"

... six months later ...

"Hey um..."

From the species that brought you "The Game of Life" comes "The Game of Technical Debt"

>"We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it"

you have no idea how right you are :)

Been there, done that, got the sqlite t-shirt...

Wasn't a bad choice, but once you load 20TB into it, you realize you done goofed.

I think you'd be shocked by how many of our financial and social institutions rely on emailing Excel spreadsheets to one another.

I am absolutely not shocked by that. Most people do not know how to insert into a db yet alone spin one up and add tables. Everyone can excel.

Don't forget healthcare! Excel and CSV for tons of data exchange.

I think the healthcare industry is probably the only thing keeping fax machines alive. Interestingly, fax machine are tagged as HIPPA compliant. Lots of spreadsheets get printed and then fed to the fax machine for transmission instead of email/whatever. On the other side, and i'm being serious, someone gets the fax and creates a new spreadsheet and enters all the rows manually. fun fun.

CSV is somewhat okay-ish. Excel, not so much (thanks Excel for discarding the important leading zeros in this field, and for lossily parsing the other as a date... oh, this Excel file is so big it got the "XLSB" treatment which is hard to undo)

Who? Almost anyone who wanted to get something done without going through the red tape of working with IT. Even in a private corporation that red tape is enough to make many a things end up in excel (or for the more tech savvy, access). Government is surely even worse (better?) at having red tape so the pressure was likely stronger.

Yes exactly. An endless battle: IT want to stamp out spreadsheets and centralise everything but it all becomes an ossified bottleneck of development projects. Meanwhile people need to get stuff done so they use Excel.

Yup. What also helps is that the data literally sits on your drive, or your network drive. No SaaS vendor is holding it hostage.

It doesn't feel like that. Even if I save the file (365 subscription and desktop apps) on my disk it steel feels 'cloudy'.

Of course Excel. What else non-technical users can use? They simply don't have any other data tool.

Excel is perfectly fit for that and is nowhere near the actual issue.

It's lightweight, fast, portable, compatible, and just about everyone can use it regardless of (technical) skill.

Except it wasn’t the government. They outsourced the test and trace system to private companies.

Public Health England was responsible for collating the data which came in the form of CSV's - PHE is a government agency.

Reply on news.ycombinator.com