Hacker News Reader

Comment by GiorgioG | Hacker News Reader

When I consulted at a health insurance company (a Blue Cross Blue Shield licensee with 1000 employees, 200 IT staff) about 10 years ago, I was absolutely horrified that the system of record for health care plan data was Excel spreadsheets. They had staff who would update the claims processing system based on thousands of spreadsheets. When you called customer service, the CSR app would auto-open the member's plan information XLS file to the CSR rep. It got the job done - but it was the wrong tool for the job.

We recommended they implement a PDM (product data management) system - it would have saved them millions of dollars (considering they had a staff of 20 whose sole job was to update the claims processing system by hand based on these spreadsheets.) Unfortunately when the 2008 financial crisis happened, management balked at the initial outlay. It's quite possible they're still using Excel to this day.

View on HN

There are upsides to excel in cases like that…

The health insurance people of various sorts who use this setup are the ones with domain knowledge. With an excel program, they can (1) understand how it works (2) add to it, modify it (3) create a copy and break it (4) debug it (5) have ideas about how it could work differently…

I’m not saying that excel is the tool for the job (sounds like it is definitely not), and those advantages erode as the setup gets more complex and/or has more people using it. But, there are reasons excel gets used this way.

The alternative is often rough. Software projects like this have high failure rates. Getting “requirements” out of users’/clients’ heads and into programmer’s is difficult. It means the standard (natural?) gap between domain knowledge and programming knowledge is restored. That has costs.

I’m going to repeat again that I am not advocating health insurance companies run policies and claims in excel. It is a bad idea, almost certainly. I am saying that we should be curious at why they are doing it this way. I think it’s a “good” question in that it leads to interesting places.

If the excel team was blessed with a plague of inspiration tomorrow morning, Is there a way they could make it so that excel is the right way to manage an insurance business?

On top of what you're saying, the alternative is frequently a heap of under specified, under tested (and only manually tested!), copy-pasted and shoddy quality Cobol/Java/Oracle/C#/SQLServer/PHP/MySQL code.

Yes, the performance will generally be better (1 minute to run a report, not 1 hour), but I'm not sure things will generally be better. As you said, the business people can at least modify the Excel. These systems? Not so much, without an usually massive support contract (Yaaaay! for us IT people, I guess).

People sometimes make a false equivalent and imagine that the Excel jungle will be replaced by a Google-developed, 90%+ unit tested, refactored and cleaned periodically bunch of code.

> the business people can at least modify the Excel

I'm not convinced that's a good thing. If they can change the Excel then they can also fuck up the system, and if you fuck up a system like this then it's horrendously difficult to unfuck. I think systems like this look great when they're working, but the second they go tits up you realise why people are horrified by them.

Users able to modify cod is the opensource model. It doesn't lose its benefits and advantages just because the users are not us(professional devs)

It's not so much being able to view or modify code that's the problem, but the fact that any modifications to an excel spreadsheet are inherently running in production. Open source has many benefits, but while open source projects may accept pull requests there is very few which put contributions straight into the latest build without review.

That's not a realistic comparison. I have yet to see one user try to modify Firefox's code. Spreadsheets, on the other hand ...

You are still making a distinction between user and developer. I think the point is that we can all be both.

That's not the issue at all. The issue is that these systems invariably have no review process, no testing, and no versioning (at best they'll have each spreadsheet versioned with no real way to know what each version contains). And this makes them incredibly fragile.

Especially since the people who do this generally don't use version control.

They do use version control. They put each version in a folder named after the date

And these user-developed systems are often great...until someone accidently deletes or changes some data and it becomes a mess.

Unfortunately, many businesses then find they have no way to know the extent of the data lost or who did it. And there’s no proper backups. Happens over and over thoughout the world.

Excel’s a great tool and there are lots of terrific uses for it. But what the article describes isn’t one of them. If we’re being honest, this isn’t the first time the issue’s been discussed and the companies likely aren’t actually switching away from excel, but rather supplementing it.

In fact, the article seems to be more a “hit job” / “marketing piece” for Anaplan Inc., Workiva Inc. and Adaptive Insights.

I agree, the issue is not Excel as such but how it's being handled: I once wrote a blog post on how you can unit test Excel spreadsheets, too: https://www.zoomeranalytics.com/blog/unittests-for-microsoft...

> The alternative is often rough. Software projects like this have high failure rates. Getting “requirements” out of users’/clients’ heads and into programmer’s is difficult. It means the standard (natural?) gap between domain knowledge and programming knowledge is restored. That has costs.

The alternative doesn't have to have a high failure rate. I was brought in to help reboot their in-house implementation of a product data management system - the core issue was they didn't have any real competency in software development (though they believed they did.) Ultimately I recommended they abandon the in-house effort and instead proposed using an off-the-shelf PDM system, we evaluated quite a few decent options over the course of six months and found several that met the business's requirements. It would have been the right thing to do and I believe it would have been successful. Culturally the IT department staff was plagued with not-invented-here syndrome. It was a generally dysfunctional IT group and it's quite likely that Excel got it's start as the system of record because the business needed to get the job done, and the IT department was unable or unwilling to deliver.

Alternatively you misunderstood the business requirements and would have cost then a fortune on a failed project.

Alternatively you were brought in by management to make cost cutting recommendations - and your solution wasn't cheap enough.

Alternatively the company accounted for the enormous risk of the project succeeding but actually being worse than what they had in place.

You really don't know the politics of this.

Edit: Another alternative is that your plan would have resulted in sufficient job losses that the company was worried that it would demotivate other employees and damage the company more than the cost savings.

I upvoted your post because I think those alternatives are interesting to consider, but _you_ know far less about the politics of the situation than the OP.

It's entirely possible that the OP was exactly correct, or close to correct. It's also quite likely that at least one of the factors you mentioned (or others) was also at play.

OP here - the director that brought me in was (and still is) a straight shooter. There's no question as to why I was brought in. While she had the support of the CIO, the rest of the directors spent more time protecting their kingdoms than trying to modernize their systems. Fortunately for both myself and the director, we moved on from that organization.

Make it easy to interoperate with databases and/or revision control painlessly for the end user and the developer. The users are fine with excel, it's the attendant issues like centralizing data, running automated services off of the data, backing it up, and keeping an audit trail that created pain when I worked in healthcare.

If excel online was better, a lot of that would be pretty doable.good points.

But..I do think that if we're thinking in terms of "users" and "developers" then we're no longer talking about excel. The whole reason excel is used this way is the continuous spectrum of sophistication.

Also because Excel is both an IDE for developing spreadsheets, and a GUI container for operating them. Splitting dev time and run time environments would help with control. When a dev supplies a user with a traditional .Net GUI app, they don't enable the user to hack the app by supplying a copy of VisualStudio alongside. Effectively, this is what Excel does.

Yes, but.

It's no good if you fix excel by making it not excel. The whole reason excel is so popular is the intuitive, unintimidating but terribly wrong way it mixes data, code and GUI. Functions in the data, constants repeated in every row, hard coded variables all over the place. If you're just supplying a GUI and hiding the code, the user can't change, add or debug anymore. The point is that they can code, even if they don't know they are coding.

The trick is to think up a way of shipping VisualStudio, but making it less terrifying to engineers, managers and such.

Agreed. But a different way of working the same trick is to have an alternate run time that can execute .xls[x|m] as a headless server process.

"If the excel team was blessed with a plague of inspiration tomorrow morning, Is there a way they could make it so that excel is the right way to manage an insurance business?"

What a professional software developer's workflow has and Excel lacks:

* principled way of sharing code across a team with distributed source control (or similarly powerful tool),

* automated regression tests,

* input validation,

* continuous deployment tools to automatically deploy working code (default mode for Excel I guess), and a mechanism to roll back to a previous version (not so much),

* build and dependency management tools for bringing in libraries from a vast open source ecosystem,

* systems for reporting bugs and feature requests and tracking their progress,

* networked database that is the shared "source of truth" for all users,

* security and auditing,

* hardware capacity planning (run locally or in AWS? what are the cost trade offs?)

So sure, get the health insurance people doing all of those things, and have Microsoft figure out how to do all of them well in Excel, and congratulations, they are now software developers. None of these require using a "real programming language", at least not directly.

That's great (though I'd quibble about a few of them, eg I was doing input validation in Excel 25 years ago, and boy do I feel old now), but while you're learning to do all that great stuff you're not learning anything about the domain you want to work in.

OK, I don't know who wants to work in health insurance but the basic point is Excel is a good tool for people who have specialist knowledge in a particular domain and want to automate their work. Programmers know the best way of programming but frequently overestimate their understanding of other people's problems.

You underestimate how difficult it is to work with at least some software companies, how expensive it is, how difficult negotiation it takes to convince quite a few nerds that user actually has business case for that requirement. And many software companies produce poor quality and are very slower then necessary despite using all buzzwords and all nighters. Managing even outsourced projects takes learning and expensive failures are nor an exception. All in all, the software we all collectively produce is still very expensive and unreliable.

Even absent that, it is difficult to explain what users need to coders who have zero idea about the domain. Unless it turns out they are the kind that just know user is idiot the moment she has different opinion on her own job, in which case expect needed time to tripple.

The homegrown excel is awesome analysis once the company decides to move on. It is concrete, you can generate test cases from it, it has features.

Integrated version control is one of the biggest things I wish they included in Excel rather than relying on an external system to do it.

One other point is let's say you do create some fancy new system to handle the task. It's likely going to erode and be surpassed by new computing technology at which point you're looking at an expensive legacy migration which could prove to be costly and you have the same downsides you mentioned.

Excel upgrades don't seem to break previous files as severely and have the backing of Microsoft.

I can't believe I'm defending Excel.

>If the excel team was blessed with a plague of inspiration tomorrow morning, Is there a way they could make it so that excel is the right way to manage an insurance business?

No... but I think the converse might be possible. If those with the programming knowledge sought to replace the Excel system but in a way which actually preserved the useful features of the Excel solution. Dirty, filthy things that programmers don't like. No referential integrity, data duplication, deferring decisions to human beings rather than lists of rigid criteria, etc. If they could swallow their bile for just a bit, they could build a system just as useful and flexible and introduce tools to add in the things like data consistency and referential integrity - tools that the user would initiate, thus accepting responsibility for the restrictions such things impose on themselves and the data.

That is not, unfortunately, how it is usually done. (I am currently literally working on a program to replace a 40K row Excel sheet that crashes multiple times a day which has a workflow centered around it that involves slicing and dicing the sheet, passing it off to multiple different groups for separate editing, etc... not sure I will be able to sell the team on this approach, however I do believe it is sound.)

| I am saying that we should be curious at why they are doing it this way.

If you hand a caveman a machine gun, he will club you to death with it. IOW, the casual ability for domain experts to screw up a spreadsheet is enough of a reason to decide that excel is a terrible solution to this problem.

I would make a different caveman analogy. Hand a caveman a gun. He probably won't be a better rifleman than you (or someone). Really bad form and technique. But, he'll probably be a way better hunter than you, shoddy rifle technique or not.

Caveman were better than you at a lot of things. They knew how to find squirrels

The problem is switching would mean transitioning to a generalized system that they would have to work around. The Excel system looks like a nightmare to the HN crowd, but it was built around their business processes. It's basically a boutique app designed specifically for their job, it would be very hard to find a general product that does the job as well.

I've seen this a lot where a consultant will swoop in and tell them that they need to use Peoplesoft or something and the organization spends millions of dollars and years of retraining to get a system that has a lot more friction in their actual workflow and is never quite as good for getting the job done.

Of course there are often side benefits that can't be overlooked, like a much more sane backup solution, interoperability with other providers, etc... But in the end most of the employees will tell you straight up that the old system was better. Too much of the domain knowledge was baked into the old system and doesn't make the transition to the new system.

>When I consulted [...] We recommended they implement a PDM (product data management) system.

>We recommended they get a system that manages their product data

Yep, I believe you are a consultant.

They should be careful though: excel allows a maximum of 1,000,000 rows. I have seen more than one major crisis occur when a spreadsheet ran out of space, yet all of it was needed together to do daily business.

1,048,576 rows to be precise. ;) Though you'll run into performance issues in more complex spreadsheets once you get into the 100,000 row area, or even 10,000 or so. Then you'll see power user departments switch to the more opaque but more performant Access.

They switch to Access until they hit the data size limit of 2GB. Then they come back to IT who has to deal with the mess that you get when you allow domain experts to build databases.

I feel your pain but not everything built in Access becomes an unwieldy, klugey monstrosity. When Access solutions are born, their useful life, evolving reasons for use and expected storage requirements aren't always well-defined. It's only after users begin to embrace something that works, and request new functionality, that these things become clearer.

When faced with a decision b/w using a RAD tool to create something outside of IT, or spending $500,000 and waiting 6 months so real devs can deliver something Finance needed yesterday, that may not be needed tomorrow, the choice is an easy one.

Your generalization isn't entirely unfair; IT is often called on to fix a mess, but some Access solutions are built by people who understand database design/normalization and can put together a easily-maintainable application that never hits storage limits or requires upgrading to a SQL Server-managed back-end.

Corporate IT never lays eyes on many Access projects that aren't crap.

Domain experts are not the best people to build databases (says a domain expert who's failing miserably at that task right now), but database experts are sort of terrible at understanding domain problems.

It seems to me that many programmers don't appreciate that industry practices or regulatory requirements impose quirky requirements on domain specialists that everyone knows are far from optimal but which need to be complied with anyway if the business is to turn a profit or retain its licensure.

Someone mentioned a health insurance example above: there's 50 states, with 50 different sets of insurance regulations, and then some additional territories, federal regulations, and then wird business practices of healthcare providers who are subject to their own 57 varieties of regulation, and there are powerful economic and political interests that get in the way of burning it all down and replacing it with a single system designed by a committee of one patient, one doctor, one nurse, one pharmacist, one lawyer, and one economist.

And even that highly simplistic requirements committee went through 3 versions while I was writing it up.

The domain experts already have lots of complex knowledge and experience they need to automate. They don't want to learn about databases. Unfortunately, the reverse is equally true.

> It seems to me that many programmers don't appreciate that industry practices or regulatory requirements impose quirky requirements on domain specialists that everyone knows are far from optimal but which need to be complied with anyway if the business is to turn a profit or retain its licensure.

I think only those that turn into consulting understand it properly, because they aren't confined into a cubicle coding all day long without direct feedback how the real world actually looks like.

At most there is some feedback coming from ticket management system or the product manager.

Yeah, updating these sorts of insurance systems was a huge line of work for Rails devs in the late '00s.

Reply on news.ycombinator.com