In my opinion this might because of xlsx compatibility overhead. Using ods format might speed up a bit.
One example that comes to mind was that spreadsheets are just memory mapped files and the layout of the file on disk is identical to the data structures in memory. This allows them eschew translation to a data interchange format. So they got performance at the cost of interoperability, which is probably what's hampering open office & friends.
Microsoft likely changed several in memory structures when Excel went 64-bit, if not earlier.
One thing that Execl does is multi-threaded recalculation (https://docs.microsoft.com/en-us/office/client-developer/exc...)
Somewhere, there is probably someone running hundreds of threads for excel (likely in a beefy VM/VDI). It is probably wired so deep into their business that they are afraid to move to other methods (that are more scalable). But such is the power of excel. What you see is what you get is not to be underestimated.
I would predict/expect that both LibreOffice and MS Office (with their modern XML-based formats) are actually mmap(2)ing some temp file and treating it as an "on-disk working-state heap", and then importing from interchange formats by allocating from that heap / exporting to interchange formats by chasing pointers that end up inside that heap. (This is, after all, what every RDBMS does for its working state. It's pretty optimal.)
Even if you have a memory-mapped interchange format, I'd still expect them to have a separate disk-backed working heap for all the stuff that doesn't belong in the file but is nevertheless very large (e.g. cached intermediary computation results of spreadsheet cells); and, if they have it, they may as well just use it for most things by default. Thus, I would expect that even in old versions of MS Office, the in-memory data structures were actually an interchange format of sorts—not the ones being updated with each keystroke, but rather ones that'd be memcpy(2)ed into on export. (This also prevents you from either having to add a page-table structure to your file, or else constantly "defragment" it as data structures change.)
It’s definitely possible to create a performant and portable document specification and others have.
I just strongly suspect that the performance issue that libre office and others have is more of a manpower issue and not having equivalent resources and knowledge of the excel formats rather than some shortcoming in ms’ own file formats.
That improves save/restore performance, but in and of itself doesn't do much about execution performance of macros.
I'm sure they could probably come up with something that is both portable and performant but it's probably not a big priority at Microsoft.
However, I am doubting it would help given how often LibreOffice was freezing with it (without crashing). My suspicion is that the formula recalculation is triggering more often that it should, and I could not find any way to prevent it. Excel provides options for manual triggers; if Calc does too, that might just solve the problem.
PS: I am not deeply familiar with LibreOffice. :-)
We are in a thread about rendering an image by zooming out enough on an excel spreadsheet and then manipulating it in near real-time by applying a formula on the cells.
It's pretty damn impressive! I get it's 'cool' to hate MS but seriously...
But yes, I'm a bit jaded. I think you'll understand why that is too. The vast fortune in revenue that doesn't fix bugs put me off excel in a big, big way. That was 2008 or so.
Here's Andrew Gelman in 2013 on the topic:
"Microsoft has lots of top researchers so it’s hard for me to understand how Excel can remain so crappy. I mean, sure, I understand in some general way that they have a large user base, it’s hard to maintain backward compatibility, there’s feature creep, and, besides all that, lots of people have different preferences in data analysis than I do. But still, it’s such a joke. Word has problems too, but I can see how these problems arise from its desirable features. The disaster that is Excel seems like more of a mystery."
https://statmodeling.stat.columbia.edu/2013/04/17/excel-bash...
We've heard from microsoft so many times they have people working on all those bugs too. I remember the sheer disappointment in testing the newly fixed rand() function after all the fanfare by filling a page with =rand(), conditionally formatting when negative to be red and seeing the page turn largely red after a couple of F9 re-calculates.
I simply don't believe that excel programmers are idiots and have been for 15 years.
That leaves "they say they're working on it but they are not, not really." At the top of my alternatives. Is there a better one you can suggest?
Enlighten me please, how does this epic cluster of fudge happen? You were probably there while it was ongoing.
Parts of excel work well, /all/ the stats should have been removed 15 years ago as unfit for purpose and zero will or ability or effectiveness in getting a fix and making good.
Excel, just don't. That is a pretty reasonable response, don't you think? Yeah it's sad. I don't relish it but let's not pretend it's all ok, yeah? But I guess the masses of revenue keep coming in so I guess it is all ok from microsoft's point of view. Are you ok with that yourself?
I'm an Excel and VBA guy. The IDE hasn't been updated since office 97. It's not great.
But the thing that Microsoft understands iS that people buy your software if it doesn't break their workflow. Backwards compatibility is the most compelling feature when you've got an install base in the millions.
Now, they have fucked up. A lot. There is a bug that counts 1900 as a leap year. The statistical functions don't work. They can't dump VBA no matter how much they want to. They tried with VSTO and officejs but nobody is buying. I get it. They are stuck and the only real way out is to break compatibility. But haven't we seen what happens when you go down that road. Python 3. Perl 6. Acrimony. Discord. And for what? Your spreadsheet to break and you have to debug it? What if you have a spreadsheet that's never been documented with a million formulas. You probably have a day job, you need your tools to work and Microsoft understands that.
These days it sticks around because of compatibility with older versions.
1. "The Ribbon" The most useless, workflow breaking, unwanted garbage change I've ever encountered. No exaggeration. A change microsoft dropped resources into instead of fixing the bugs, sucking time from users and upping their stress and frustration levels. It's the poster child for workflow breaking. Excel. A higher microsoft priority than fixing bugs. Contrary to your claim they clearly understood that they had enough market power to force it on users breaking their workflow and make the users pay for it.
2. Microsoft love breaking compatibility. Every damn upgrade of office somwhere else by some other customer meant you couldn't open a spreadsheet containing a single column of numbers because it was "incompatible." You had to request they convert it and re-send if they knew how and probably you or your employer would be forced to upgrade to avoid that hassle while get slugged with the ribbon you didn't want.
No upgrade treadmill anymore, now they can just charge you yearly without having to play that awful upgrade treadmill game. They don't have to fix bugs either, as we agree and they don't. They chose not to. So that needs to be pointed out every time it comes up to counter a little bit of the horrible stealth of it.
They did /try/ to fix rand() with much fanfare by breaking it different but no version issues. Was broken, is broken, never use it. The only stats function I'm aware of they tried to fix? Really? No? Like "sure, but maybe some people are dependent on getting wrong results and allocating resource based on error?" Is that really the excuse for not putting a tiny part of that mountain of money to stop kicking customers? It's objectively awful. Compatibility with utterly wrong that anybody relying on that wrong has a massive issue.
1900 isn't a leap year and dates are stored as number of days since Jan 1 1900 so fix it and every single date rolls back a day, hilarious and everyone forgives that including me. Stats functions are not like that at all and do not require version changes. Just the will and resource to fix them. What is needed is to actually care.
So that leaves your quote:
"They have fucked up. A lot."
That one stands up. But I don't think you've really embraced the depths of the disaster that is excel and why we should encourage everyone to avoid it. Because (to paraphrase) the error is quite deliberate. It's not worth it to them to fix. They know the bugs are there. They no the bugs are material. They no the bugs stop the software for being fit for purpose but they'd rather not spend the resource, which they could do quite easily. They give software a bad name with that attitude. Are you really happy with it? Really? How much harm do those bugs do every day, in your opinion?
Excel, just say no. Really. I'm sorry if you hate hearing that and you were proud of work you did there or whatever. I was pretty bummed when I came to that realisation myself.
Microsoft earned a reputation with regards excel and they maintain it, even if it seems they don't maintain the actual software.
Considering the "every(wo)man" approach it ability to be useful is near genius, but just because your volkswagon won't get you to mars, it doens't make it crap.