Comment by totaldude87 | Hacker News Reader

Comment by totaldude87 | Hacker News Reader

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :(

5 – London Olympics Oversells Swimming Event by 10,000 Tickets

4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contract instead of deleting them.

3-utsourcing specialists Mouchel had to endure a £4.3 million profits write down due to a spreadsheet error in a pension fund deficit caused by an outside firm of actuaries

2- Canadian power generator TransAlta suffered losses of $24 million as the result of a simple clerical error which meant they bought US contracts at higher prices than they should hav

and the Biggest one is. -

Basic Excel flaws and incorrect testing led to JP Morgan Chase losing more than $6 billion in their London Whale disaster.

https://floatapp.com/us/blog/5-greatest-spreadsheet-errors-o...

They are costing money. No doubt.

At the same time, the business world runs on Excel. How much money is Excel making?

I've done my share of cursing at Excel at various jobs. At the same time, I am grateful for the quick and easy way it allows me and many others to manipulate data. It's unfair to just cite the costs of using Excel without acknowledging the benefits it brings.

The real question should be would it be cheaper to replace it with a proper solution (probably something custom built application written by a software engineer).

Excel's ease of use is it's downfall. It is the worlds most popular database, despite not actually being a database. I have wasted countless hours dealing with Excel where something else should have been used. I built a database for a friend recently, I think 75% of the work was cleaning the existing data from the excel to get it into the database.

it will only be cheaper after the company loses money... no business person ever looked at something and said "it works, but it might not, some day... better spend $XXX,XXX on an engineer to fix the problem".

You have just identified the new market of Spreadsheet Insurance. I am thinking it could be quite lucrative if you get the right salespeople in on the ground floor. I'm willing to bet no funding source is unfamiliar with the demand for this product.

The problem with a database is that business users struggle to build them, they take longer, and sometimes can’t hold logic in the same way.

Plus in a database you can’t do the same sort of real-time analysis and also pass the document around for other non-technical folk to add to and modify.

In the real world in big companies, people often don’t want to talk to IT because they over-spec and quote what are perceived to be giant sums of money for something that can be created in an hour in excel.

To add to this point, the latest article says that they started building a replacement to the spreadsheet two months ago, so I think that reinforces the speed of a spreadsheet implementation vs anything else in the public sector!

You might not have realized this but they should have started work in January.

You might not have realized this, but a global pandemic was starting in January and they were probably collectively shitting bricks at that point trying to shoe-string together something to help capture reporting, while probably being in a change freeze, and also probably having to do a mass-migration to at home work with the inevitable struggles of that.

People start using excel when the requirements are unknown, they use excel to understand the requirements while still being functional. To build a "proper solution", you frontload all the requirement discovery to build a system which only you or another developer will be able to update/change.

We blame excel, but excel is really just being used for prototyping and nobody takes a decision at a certain point to move on from that prototype.

I would say that people use Excel because they don't understand how to set up a relational database. It's usually non-technical users that set up Excel, then at a later stage I get asked to create a proper database from it.

>I would say that people use Excel because they don't understand how to set up a relational database. It's usually non-technical users that set up Excel,

I disagree that lack of db knowledge is the primary reason. I'm a programmer and I usually use MS Excel because it's easier than relational databases. I prefer Excel even though my skillset includes:

+ Oracle DBA certification and working as a real db administrator for 2 years

+ MySQL and MS SQL Server programming with raw "INSERT/UPDATE/DELETE" or with ORMs

+ SQLite and programming with its API in C/C++/C#

+ MS Access databases and writing VB for enterprises

The problem is none of the above databases (except for MSAccess) come with a GUI datagridview for easy inputting data, sorting columns, coloring cells, printing reports, etc.

Yes, there are some GUI tools such as SQLyog, Jetbrains DataGrip, Navicat, etc... but none of those have the flexibility and power of Excel.

Yes, a GUI frontend to interact with backend databases can be built and to that point, I also have in my skillset: Qt with C++ and Windows Forms with C#.

But my GUI programming skills also don't matter because for most data analysis tasks, I just use Excel if it's less than a million rows. Databases have a higher level of friction and all of my advanced skills don't really change that. Starting MS Excel with a blank worksheet and start typing immediately into cell A1 is always faster than spinning up a db instance and entering SQL "CREATE TABLE xyz (...);" commands.

Of course, if it's a mission-critical enterprise program, I'll recommend and code a "real" app with a relational database. However, the threshold for that has to be really high. This is why no corporate IT department can develop "real database apps" as fast as Excel users can create adhoc spreadsheets. (My previous comment about that phenomenon: https://news.ycombinator.com/item?id=15756400)

I find people often use Excel (or Access if they're brave) when they have a known problem and can visualise the solution in something they know, like Excel. The crucial factor is implementation time. They can open an Excel and start work immediately, or wait weeks or months going through a lengthy and complex IT procurement process that will take from their budget. So in most cases starting with Excel is a no-brainer. Same reason why a lot of stealth IT gets implemented.

It's often only after years of a business using what has become sacred & business critical Excels, that somebody suggests formalizing it into software. In a business with an IT function, or a consultancy looking for business, it should always be somebody's job to find these Excels and replace them with something more robust.

> Same reason why a lot of stealth IT gets implemented.

Honestly, I miss the days of writing VBA macros which save hours of work a week and being sneered at by the 'official IT'.

I worked in a team in a large commercial bank handling reconciliations with various funds. Some of which had to be contacted by phone to confirm the current holdings. We had a system which would import our current positions and take imports in various formats from funds. Somewhere around 40% of the differences where due to trades which had been executed over the reconciliation date. I wrote a VBA script which pulled in all of the differences and identified trades which where open over the period and automatically closed the discrepancy with a reference to the trade IDs.

Another time I wrote a VBA script which would take a case ID and look it up in a diary system (at the time the only way I found to do this was to use the Win32 APIs and manually parse the fields in the HTML from the system), it would then enter this at the top of a spreadsheet which had to be completed. People liked it so much I had to rewrite it so that it would work on a list of case IDs and automatically print out the checklist.

Much more fun than figuring out why Kubernetes is doing something weird for the 3rd time this week.

No, because the proper solution written by a software engineer will contain more bugs.

>It is the worlds most popular database, despite not actually being a database.

"A database is an organized collection of data, generally stored and accessed electronically from a computer system."[1]

Excel is an organized collection of data, stored and accessed electronically from a computer system. So I would call it a database.

[1] https://en.wikipedia.org/wiki/Database

Excel is not a database, it is a spreadsheet. They are different beasts.

This link will explain the difference between two of them. https://365datascience.com/explainer-video/database-vs-sprea...

What i was getting at is that it fulfills the (very) basic definition of a database, even though it's very bad for this purpose.

and one more:

"Scientists rename human genes to stop Microsoft Excel from misreading them as dates. Sometimes it’s easier to rewrite genetics than update Excel" https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

Also the post-2008 financial crisis austerity programs based on Excel analysis.

https://mathbabe.org/2013/04/17/global-move-to-austerity-bas...

Austerity wasn't based on that study, and it didn't go away once that error was found.

That study was the intellectual justification for the policy.

To be fair, I'm not sure if any of the proponents actually believed (or had read) the study, but it was definitely wheeled out in debates against Keynesians.

Yeah, why there is no "scientific" mode to just turn of all "smartness" has baffled me for years and will probably continue to baffle me for years to come.

Or even an algorithm that can detect that you are using gene name from the cells around march1 and sept7.

I think that's a good way forward, Excel is a fantastically powerful, ubiquitous tool and isn't going away any time soon. If you're going to standardise on a technology, it makes sense to design your systems to work well with that technology. That includes making identifiers that you need to be treated like strings look like strings to your tools. A little bit of thought in advance and maybe some small compromises early on can save massive problems down the line.

A master crafter always adjusts their industry to match the currently popular tools. That's why we still use reins to steer automobiles and aircraft.

Well, not exactly like non-Excel systems are bulletproof either.

Entire Japanese stock market went down last week, and it's not like there was a flood of people on HN bemoaning that. At least with Reinhart and Rogoff et al you have a responsible party.

As opposed to 'nameless machine failed, and nameless backup machine also failed, and now it's in JIRA so don't worry about it'.

https://www.nytimes.com/2020/09/30/business/tokyo-stock-mark...

The glitch stemmed from a problem in the hardware that powers the exchange, said the Japan Exchange Group, the exchange’s operator, during a news conference. The system failed to switch to a backup in response to the problem, a representative said.

Reminds me of this - https://www.reddit.com/r/programminghorror/comments/fv7qi7/b...

That example is overblown as the system is supposed to be restarted more frequently than that for maintenance. A better example is cars have been frequently rebooting various internal computers while your driving for decades, that seems bad but it’s similarly irrelevant.

I think similar to “code smells” in software engineering, using excel for critical data pipeline work can be considered a “data smell”. It’s not bad in and of itself, but it’s a signal that something may have gone seriously wrong with the data engineering process.

Unfortunately excel is the lingua franca of data exchange in SME

I am sure there is a solution to this problem that could make someone a fair bit of money, but i am not quite sure what it is yet.

On the other hand, its more like "nobody ever got fired for using excel"

I'm willing to make an exception in this case.

I think in this case it’s more like “nobody who would know better than to use excel had the authority to make the decision not to use excel”. Given that this was a government project, my guess is that there are very few technical people involved at the decision making level.

And how about this one: an Excel error was the cause for much unnecessary austerity after the 2008 financial crisis, costing potentially tens (hundreds?) of billions in lost output.

https://www.nytimes.com/2013/04/19/opinion/krugman-the-excel...

Is it really that hard to spend 5 minutes setting up a decent database? Use sqlite if you really want something simple.

I guess you still have to understand locking (especially on distributed filesystems.) I've certainly seen people mess that up with spreadsheets.

The problem is people can't do this in a office. Nobody has admin rights on there computers. Everybody know how yo use Excel. We all know, if you have to make a ticket to your office IT for such things, you mostly have to wait a long long time and it never, never works the first time. I guess that is one of the main reasons, why the world runs on Excel (and MS Access). It just works, it's just there, no dealing with IT.

Sqlite is a bit of a special case, in my eyes. It's so ubiquitous that it's probably already installed somewhere on the average machine, just not as a top-level application.

Of course, there's a bit of a gap between "it's there on the machine" and "we can rely on it for useful work", but baby steps...

Whenever you try to do something properly, some person in your organization condescendingly says "perfect is the enemy of the good".

And because Excel is "good" for a vast number of use-cases then people use it for everything.

Hey come now... we both need each other /s

Sure, I'll just teach tell my wife to learn SQL and some devops while she's at it. After that I'll coach her on how to get buy in from management on this new system while she neglects her primary duties.

Throwing up a database and integrating it into a workflow/system isn't something anyone can just get up and do. I have to imagein you know that.

Yep! And then when your wife gets that promotion she wanted, she still has to spend time maintaining and making changes to the old database system because nobody else understands it or has the skills to change it.

Was it really a manual process? I read it as some system importing info automatically.

And if it was manual I am surprised that Excel did not complain about adding more than 65000 rows (or saving more than 65000 rows as XLS). If a user gets a warning about possible data loss they should investigate more.

Do note that anything other than Excel, UI/UX wise, failed.

When people wants a “database” they fire up Excel, start punching in numbers, solar calculators next to keyboard, and use eyeballs to search for strings.

Things are never 5 minutes. And it is just as easy to make mistakes with sql as it is with excel.

I have definitely just spent 5 minutes importing csv tables into sqlite so I can run a query against them.

Every time I see a problem like this I can't stop thinking that computer education is terrible everywhere.

We are able to teach almost everyone how to use complex software like Word and Excel. Why can't we teach people how to use a terminal, SQLite, or how to create a very simple Python script?

Well maybe it has something to do with the Windows monopoly and that they only want you to use their proprietary tools. If you learn too much then you will want to be free!!

No it isn't, but in any big org the plebs are only allowed to use Office and anything else is IT captured.

It takes a course to understand how to to use a relational database.

MS Access used to come as standard with Office and is actually the perfect solution to many of the problems that businesses use Excel for. It's very rarely that people actually used Access as Excel was far more intuitive and good enough for many projects especially in the early stages.

I can attest to this. I move job and my new work place used to manage everything in excel. Something's still are. Realising excel could not scale, and with no support for IT for a proper solution the "tech" guy (that is good with computers but not a developer) was tasked with creating a "database". What we now have is essentially 3 excel spreadsheet tables in access that you can run SQL on and input data using access forms. No normalisation, no well anything really. We're now coming to the point where access is creaking at the seams. I'm confident with properly designed tables and queries we wouldn't be, and I'm no expert either, but it will have to fail before there's a concerted effort to redesign or god forbid IT do there jobs.

Sure, Excel is instantly available. But also you get charting and formulas. Especially with formulas they've done something really right. You can transform a column in a few seconds while keeping the old one, tweaking the new one and having a graphical preview through the chart feature at the same time. Of course every Spreadsheet software can do this, but to me that's the ultimate killer feature

Reply on news.ycombinator.com