Why we can’t quit Excel
bloomberg.com
bloomberg.com
There's an immediacy to a spreadsheet: the user can start with very literal actions on data, and slowly introduce abstractions like formulae. In conventional programming, the programmer has nothing but abstractions with which to work.
It's reminiscent of one of Fred Brooks' remarks about how showing the data structures makes the algorithms obvious, but showing the algorithms reveals little about the data structures.
Note all the audio programming tools getting embraced by music folks as well, it is coding, but made approachable.
I hate Microsoft but love Excel. I was hoping python in excel would be this but I don't think it quite is.
(I somehow though of the videogame Factorio and of that thing called Labview, but I cannot form a coherent thought about it)
It's very, very rarely useful.
There is no replacement for visualizing the data and intermediates. You have to see what is going on, it's the only way to catch bugs and issues and make good performance improvements.
Like actually looking at the gradients and weights and activations and predictions and stuff often shows you that something janky is going on, its great for adjusting architectures. Looking into specific high loss test samples and mispredictioms and stuff will show you that there are problems with your data or normalization and whatnot.
The issue is that there are basically an infinite number of intermediates you could potentially look at. So ignoring almost all of them is the only thing that scales and you have to be extremely deliberate.
If mispredictioms is not canonical yet, it should be!
The immediacy is you are looking at the data. What you’re going to do with it may not yet be known Is it the right data? What are the values? What are the features?
I write scripts at work and tell colleagues if they want my help they should first work it out in Excel and write an outline of the details. If they can’t do that, then the problem may not be a candidate for _script automation_.
Our pipeline is not fully automated, but that’s how I describe it. The rest is Excel Carpentry.
Another thing would be of course powerful REPL environments.
And there are also visual environments like Matlab's Simulink and Modelica. Can you write control algorithm or a bunch of differential equations in any language? Yes. But control engineer can just take a quick look at screenshot of a Simulink model and already have a good idea on what is going on there. This cannot be said about the code.
This may be rather unpopular among hackers, where TTY clones and vi emulation rules everything, but computers and software seem to me most powerful and enabling outside the usual coding context of putting streams of characters into the source file using a keyboard.
I love a quote from from some VB folks, that while others laught of them using VB, they were laughing all the way to the bank themselves.
My only complaint about visual environments, is that not all offer modular tools, the EEE IC version of a module or function, some are quite primitive only allowing one screen for the whole flow.
I wonder what place will be left for the TTY clones and vi emulation folks, when everything on the system is driven by systems programmed in agentic tools, command line wizardry replaced by MCP like tools and such.
I worked at a place that used Google Workspace and the most effective people simply supplied their own Excel, because it's just way more efficient than spending hours learning how to use Google's clunky pivot table function, when you already know how to do it well with Microsoft.
I'm sure there are many orgs that would love to ditch O365 for Libre or Google, but can't simply because there is no real alternative at this point. Excel is too entrenched.
Excel can take people from fairly naive "help me make a table of numbers" tasks all the way to actual programming. Via PowerQuery, Excel gives end users access to data warehouses directly. Sure, a tiny bit of SQL is helpful here, but it's not required. I've written no end of data sanitization / transformation tools using Excel. Sure, I would have RATHER done it in Perl or Python, but Excel can be assumed to be present on the target user desktops; not so with Perl.
It would be cool if there was a better off-ramp for advanced Excel users into more focused tools appropriate to their domains (e.g., R, or Tableau/PowerBI), but if these folks are solving their problems who are we to push?
Every single attempt to migrate to something else has been a comical failure. On the plus side they tend to be rapid failures rather than SAP style multi years
All those migration projects tend to fail on the details, there is always that one customisation point that is easily done in Excel, but requires a whole team with architects and such for getting the same capability on the replacement tool.
https://appdb.winehq.org/objectManager.php?sClass=applicatio...
Then came 1995, M/S released windows 95 and at the same time they had a full software suit running on W95. It took Lotus a while to get a W95 version out. I think the same happened to Word Perfect.
So here we are, too bad Libreoffice was unable to do the same with Linux. But now people are so entrenched I doubt they will only change if forced.
Also, years ago IBM tried to get people on LibreOffice and off M/S, it failed miserably. Many Orgs. in Europe is trying now, I hope they succeed.
BTW I wonder if we are heading down the same path with github ? I hope not but seems we could be.
People were aware that Microsoft and IBM were starting to disagree on OS/2 as early as 1990 and Microsoft released Win32s, the first preview of Win32 APIs running in Windows 3.1, in 1993 (when OS/2 released the first obviously "post-breakup" OS/2 2.1 and Microsoft also released the first version of Windows NT [3.1] to truly signify the breakup). The writing was on the wall in advance of 1995 and companies like Lotus had a couple years of opportunity to build Win32 applications in advance of Windows 95.
It seems hard to call it a bait and switch when the writing was on the wall as early as 1990 that Microsoft wasn't confident in OS/2.
I have seen it used for everything from configuring aircraft for sale to quoting bond prices for voice traders.
There's even that famous Japanese artist who uses it for painting.
It's sort of like the OG normies Jupyter notebook.
The fact that all these tools eventually fall apart at scale is basically irrelevant. You can only take away control from their cold, dead hands, and they will never learn the programming skills to build something more scalable themselves. All solutions almost invariably trend towards shifting scaling problems away from the desired frontend that keeps control (or at least the illusion of control) in the hands of business stakeholders (one of the reasons why companies love building all manner of integrations on top of Jira).
And let's be clear, it's not like the alternatives here (i.e. databases) are so easy. Finance folk are used to creating new spreadsheets on their local computer for free and at will; most databases require setting up a server, DNS, certificates, firewalls, most of which have real costs. SQLite naively sounds like a reasonable approach, but by default a table is limited to 2,000 columns while an Excel spreadsheet is by default limited to about 16,000 columns, and yes, stuff like this really matters when you're talking about trying to uproot a favored tool. At the end of the day, most of Excel's limitations are due to attempting to cram a spreadsheet into a single file (same as SQLite); if Microsoft were smart, they'd offer a cloud-only "spreadsheet" (really a database over a full filesystem, or maybe over object storage) without the limitations of ordinary Excel spreadsheets, where Excel-the-desktop-app only downloads to the local client the relevant cells that are actually in view, while adding more options to mirror/load data from external sources into other sheets attached to the same cloud-only "spreadsheet".
Yes, it's awful in many ways, but it is very accessible to people who don't consider themselves programmers. "I just want to do my thing" is very easily done in a mashup of the spreadsheet and VBA.
We also forget the pain of learning a new technology. People whose first experience is excel also go through this. Shit doesn't do what you wanted. After a while, they can build stuff in excel, but they don't want to learn python, because there's more pain coming.
Most senior devs have transcended particular languages or technologies, so they don't see it. They pay a very small cost for picking up a new tool, so they scratch their heads when they see a guy who wants to run a trading book on a spreadsheet.
I can't leave Excel for that. I can set up a "data integration" in 3 hours that has a highly customizable and (relatively) bug-free front-end, and maintain it myself. The amount of work and knowledge it takes to get the same thing spun up in a proper language with a proper server is 1-3 orders of magnitude more.
Other commenters have covered the major reasons Excel is so popular. But Excel really shines because it's designed for business. I use R and RStudio for a lot of my work, and while it's great, there are little things that it can't do that Excel can.
I even have done this first day's advent of code, and the first part of day two, in a Google sheet. Formulas only, so no scripting needed.
Yeah, I can't quit Excel. Every day there are spreadsheets open in my taskbar, even right now :)
LibreOffice and Google Sheets are still quite far from everything that Excel is capabale of.
I was part of a project to replace Excel, on a lifesciences department that was using it, as you might think about R, Pandas and similar.
Tableau was even a more appealing candidate to them.
It also chokes on sheets with only a few hundred thousand rows on my 16gb work laptop, something a real database can easily do.
I blame it on Microsoft making such a mess of access that people didn't understand it and started using excel as a database.