Kale: A Transformation-Safe Spreadsheet System | Hacker News Reader

Kale: A Transformation-Safe Spreadsheet System

arxiv.org

57 pointszdw21 comments

> Spreadsheet formulas can refer to rectangular ranges of arbitrary size. When a user changes the structure of a referenced table, the spreadsheet system updates the references to refer to a new range. Unfortunately, this new range may differ from the user's expectations, introducing bugs in spreadsheets. We describe a user study showing that standard reference semantics are error-prone

Yeah. It's a pet-peeve of mine: after having been tasked to port a spreadsheet to a dedicated app, I've lost all illusions about spreadsheets.

Or, put it like this: spreadsheets are for programmers. When a non-programmer uses a spreadsheet, it's full, full, full of broken assumptions.

The famous Excel vid by Joel Spolsky shows just that: it's ultra-powerful stuff, but you basically have to be a programmer to understand what's doable and how to use that power wisely (and Joel Spolsky is a programmer himself).

Users don't understand dates and the relation between values and the values at a certain date or the values at the date of encoding or the values when they're automatically fetched, for a start.

Then they have absolutely zero clue as how to model things crossing boundaries: for example something crossing a year boundary.

I could go on and on but it's just mind-boggling. Reunions discussing the spreadsheet with "the users" and asking questions and they're just lost: they don't get it. They don't understand what they've done. It's confusion everywhere (in their minds and in their spreadsheets).

The corporate world runs on Powerpoint presentation using bogus numbers from broken spreadsheets.

Now it's nice to see a system that tries to add some sanity to spreadsheets but I fear it's like "SVN trying to be CVS done right, but it's impossible to do CVS right" (paraphrasing Torvalds).

P.S: do I use spreadsheets? You bet I do, I'm a programmer.

Everything you said is why I don’t agree with the idea that “everyone will be writing their own apps.”

Can’t wait for a world full of apps as janky as most spreadsheets (or Access or FileMaker databases)

My cynical mind think link you but also comes to the opposite conclusion.

Everyone will be writing apps. Almost every app will be as janky as possible and people will just take the jank as the normal thing and get used to it and working around it, just like they did when every company started using electron and fake mobile apps that are wrapped buggy webpages.

With fast updates that changes every feature across updates, breaking everything and people accepting as the norm.

It's already happening:

- you can see Show HN post making it to the front page full vibecoded with as much jank as possible and the all the comments praising it and not questionning all the jank.

- I've seen it in some popular open source projects when after updates existing feature started to break constantly. The whole thing is ai coded up to the handling of bug reports and feature request and have Claude as co-commiter. I've looked in the internet to see the perception no one is complaining about it

- same at work, managers accepting slop AI solutions that are wrong at so many levels simply because it's fast and complain about the symptoms months down the line without questionning the AI part.

In a way if people can’t yet use spreadsheets correctly (but they think they do) then it’s impossible to move them away.

Also the OP has much about the programmatic components, but Excel is also an interface. The front end of an unofficial corporate app. And people know (they think) how to use it.

I just listened to a demo of an insurance industry SaaS that isn’t yet public. Goal to move many things out of spreadsheets. Had AI agents built in and all sorts of bells and whistles. But the screens were chaos. I immediately thought it was just too “complicated” to switch a person off the busy/messy system they know - Excel.

In fact, everyone will ask their llm to write their own app, and it better be correct…

> Can’t wait for a world full of apps as janky as most spreadsheets (or Access or FileMaker databases)

This was already the case even before generative AI

I think reality says otherwise. People do use spreadsheets effectively, if perhaps not efficiently, every single day.

The brittle nature of spreadsheets is also its most powerful feature.

I couldn’t agree more. There are many brilliant business users who can build highly sophisticated spreadsheet models but have little or no coding experience. In almost every business and every department, you’ll find those spreadsheet gurus who everyone turns to when they have a complex spreadsheet problem or need help figuring something out.

I work at a small business where probably 95% of the work gets done through spreadsheets. Spreadsheets are incredibly valuable, but yea, like you said it requires a certain mindset or discipline. Otherwise you end up with a disorganized mess which almost always has errors. The errors are bad, but the distributed arrangement of the formulas is what gets abused the most in my opinion. Modularity isn't a thing and there isn't formula re-use like in the software world. Something might be correct in one cell, but wrong in another. It could be due to bad assumptions or simply because there is so much formula repetition and the user just wasn't exercising care.

Programmers pervasively underestimate the sheer amount of manual brunt work that business users routinely do by hand and with their own eyes, mouse, and keyboard to make spreadsheet-based processes "work out."

I work in Finance. It's routine for the folks I work adjacent to to spend literal hours performing a task using a spreadsheet. The same logic could be executed by a query engine or scripting language in seconds. But... I know from trying... it would take months if not years to accurately identify and account for all the weird exceptions and edge cases that the human operator is accounting for intuitively because of their undocumented tacit knowledge of the domain.

So while the spreadsheet itself as an artifact is a mess, and the process as a whole is gallingly inefficient... the outputs they produce are often preternaturally correct.

I mostly agree. There are certain classes of tasks where the spreadsheet is more efficient. It's "more efficient" because it 1) requires less set up time (like you said, translating a computation in a more traditional system could take months), but also because 2) the spreadsheet is a highly interactive and accessible environment.

We have a programmer on our team who keeps trying to convert our excel calculations to more structured systems (database store). The issue is that the spreadsheet-work almost always requires navigating and making decisions based on unstructured data. The format isn't formally specified, so in the end our programmer makes more mistakes because he doesn't engage with the data - the processing gets treated as a routine algorithm, and it's not. The interactivity (and the fact that we're not spending energy on formal specification) is what allows us to more effectively engage with the data.

Another important point is that spreadsheets more easily allows you to consider data points individually without hand cuffing yourself to one kind of processing algorithm or software design. I can easily edit one cell to make a correction in a data set (which is a double-edged sword). The programmer has to create an additional handler to handle the exception. Depending on the design/quality of the code, this could just be adding undesirable complexity in the software. So we can either spend 30 seconds and edit a cell, or spend half an hour adding to code (which by the way may never get used again).

Yes. And yet an amazing amount of work gets done with spreadsheets. Honestly, spreadsheets and databases are probably the app categories most responsible for business productivity over the last 50 years. PowerPoint is probably third.

Hm it's interesting perspective. The paper adds sorting into the mix. Like sorting moves cells and thus prone to issues with references. For me sorting always moves values in cells and I even can't imagine how you could sort formulas. It doesn't make sense.

Since free relational DBs like sqlite & mysql came along, only optimal use I see of spreadsheets "in the wild" now is psychological (i.e. real-time "massaging" of the numbers to overcome objections). For things that are more objective, we have so many superior options. Not just databases either. Jupyter & various other REPLs are also great.

this isn't how it works. Most of the people doing tech work are not tech-savvy That's why excel is the most used software in the world

> only optimal use

> Not tech savvy.

Excel is tech , too.

Anecdotal evidence but when I used duckdb and Python and Pycharm the Pycharm table viewer was not working as well as Excel. The rendering was messed up. Could be related to the remote desktop rendering but still not as flawless as Excel. The patchwork of multiple software from different vendors needed to replace Excel with database plus viewer plus Code has downsides compared to the single app single file solution that is Excel.

Personal experience is opposite. Doing stats on millions of rows in any SQL is a one-liner. Working with more than a few screenfuls of data in Excel is a misery.

Ok, but I don't even want spreadsheets. SQL relational DB is way better.

At least I won't be tempted to eat it.

Reply on news.ycombinator.com