Hacker News Reader

Comment by danso | Hacker News Reader

I wonder if this is legacy software shoehorned into a current-day emergency situation, or something that vendors recently built as Covid-19 ramped up. In practice, depending on the data workflow and expected use cases, it's not terrible to use software that records in an Excel spreadsheet, especially if the software is intended for scenarios of 1 to 100K records. But today, it seems like a lot more work to create such a rickety system than to, say, design around SQLite.

The Daily Mail had a concerning detail about the current "fix":

https://www.dailymail.co.uk/news/article-8805697/Furious-bla...

> The technical issue has now been resolved by splitting the Excel files into batches.

View on HN

No one in their right mind uses a spreadsheet for data analysis. It just too easy to make mistakes. As in that World Bank accountant who entered '25' under the impression he was entering a percentage (0.25). The spreadsheet duly carried the calculation which was off by a factor of a hundred. Not only that, as the size of the dataset grows, so does the time necessary to complete the calculations. Sometimes the whole weekend, as I recall reading somewhere.

Sadly, lots of people are not in their right mind about these things. Lots of people do use Excel to store and share data, leading to all sorts of problems. Strings inadvertently getting converted to dates, leading/trailing whitespace, data integrity not being enforced in any way, and now apparently missing data. And very important data at that. Some people love Excel just a bit too much.

In my opinion the biggest reason Excel is bad for data analysis is that it is just too easy to produce non-reproducible results.

I had a boss once who knew Excel and thought he was good at data analysis, but he wasn't. I did most of the data analysis, and I used R. That was something he just didn't understand although I did explain it a couple of times.

Things like version control, CI, R/Python for data analysis, all make sense to professionals but not necessarily to dabblers. Perhaps a dabbler in data analysis made this decision.

Reply on news.ycombinator.com