Hacker News Reader

Interesting and agreed. What if excel was very basic in functionality, and then as soon as you get into complex modeling or advanced functions you pull from a user generated (either private or public) template library. The trick would be in making the implementation of templating simple and intuitive.

The goal behind templating would be to prevent complex formulas from being entered in tiny cells, and to leverage a userbase that has "been there, done that." Do you think that would shorten new hire ramp up in using existing complex models etc?

I think Data Nitro is really cool, and as a data geek the idea of having a real language available inside the Excel environment gets me very excited.

Unfortunately, by increasing the power and capability of what a careless user can do, while expanding the learning curve and moving Python code to a different place than any cell-level functions or global VBA functions are stored... all of this just makes the potential for issues of human failure far greater.

Is there a reason you don't use one of the current data analysis languages like R, Stata, SAS, etc?

I use those languages frequently, when they are the right tool for the job. Well, R and Stata, at least — but also Processing and D3 and Google Maps...

Not sure how this is relevant to my concern, though.

I worked in a similar environment and came to similar conclusions. In my view, your three requirements (usable without a big learning curve, intuitive to non-programmers, equally or more efficient than Excel) point directly back to the spreadsheet UI. Spreadsheet users get spreadsheets in a way they don't get anything else. It's their mental model and they love it and there's no way they're going to give it up. (In particular, there's no way they're going to migrate to even the simplest of general-purpose scripting languages.) Therefore the only thing that will get them off Excel is a new kind of spreadsheet. The question is, what would that look like? If it's too different from Excel, it won't meet your three requirements. But if it's too like Excel, there won't be any reason to use it, nor will it address Excel's limitations which are the subject of the OP. I've been thinking about and working on this ever since—it's a deeper problem than it appears to be!

Edit: by the way, you're right (in my opinion) to emphasize barrier-to-entry as a great danger in this area. If you look at the history of attempts to innovate in the spreadsheet space (most famously Lotus Improv), that is the reef on which they tend to run aground.

I don't think scripting languages, of themselves, are the real barrier, after all, excel formulas are not too far off from what one would use in a language like R. I think the barrier is the data import/manipulation phase. Scripting languages loose out to the ease of copy/paste and data inspection that spreadsheets offer. It's not that using scripting languages for these tasks is very difficult, it's just that spreadsheets are, comparatively, much easier to use.

I worked in a place that used both, and the programmers were successful in preaching the benefits and converting some of the excel users, where it made sense. However, I imagine a large majority of analysts using excel in the corporate world do not have that exposure and don't realize there are other alternatives out there, or if they do know about the alternatives, they get quickly turned off from them when they try to import/manipulate the data. I don't think any tool, by itself, is going to push individuals to change. People need to be exposed to the benefits of approaching data analysis as a small software/programming project over the long term. It is short term thinking that pushes people to use excel in the first place.

I respectfully but strongly disagree. There is a chasm between Excel formulas and general-purpose programming languages. Very many (even very sophisticated) spreadsheet users have no interest in crossing it. These are the users I was talking about. They're not programmers-in-waiting. They think in spreadsheets.

The difference between the two paradigms is much deeper than data import/export, and exposing most spreadsheet users to scripting languages (even with good import/export) won't convert them, it will just turn them off. I know this from hard experience, having built a domain-specific scripting language on a previous project that was trying to make software for petroleum economic modelers (i.e., sophisticated but non-programmer business experts). Every feature request we got boiled down to, "make it be like Excel, complete with an interactive grid UI".

It isn't short-term thinking that drives this kind of user to build complex models in Excel. It's that Excel fits how their minds work. Partly that's because spreadsheets are intuitive for many people, and partly it's because spreadsheets been mass-adopted for so long that they're utterly entrenched. Trying to get spreadsheet users to think more like programmers is a sure way to keep them stuck on Excel forever.

Here's an example of how different the thinking is. In a programming language one first builds a computation abstractly (code), then instantiates it on concrete data (input) and looks at the results (output). Spreadsheet users don't do that. They start with a concrete example, build the formulas they need to get that particular example working, and then (if they need a similar example) copy what they did and tweak it. In other words, a spreadsheet user is never looking at a pure abstraction; they are always looking at a concrete example, even when the computations they're building up are complex. If you showed them all the formulas at once, they would recoil. I think this low tolerance for abstraction is one reason why spreadsheets fit the minds of non-programmers better, and also why it is so difficult for those of us who are comfortable with abstraction (and thus with programming languages) to appreciate how different spreadsheets are.

I agree about your point on abstraction, but I also think there are different methods to deal with it, and that those methods often get overlooked.

My experience with SAS and R suggests it is possible to somewhat re-create the typical excel workflow. The SAS IDE allows you to open the data sets created in your code, which means you can write code one line (well, more accurately, one data step or procedure, as it is called in SAS) at a time and inspect the intermediate results at each point. Similarly, RStudio allows you to inspect the values of data frames and variables, once again allowing the user to step through the code writing process and get relatively immediate feedback.

Regardless, there are going to be a large portion of excel users that won't or can't switch, and there are also a number of use cases where excel makes perfect sense (smaller data, simpler summaries), but I have personally 'converted' a few analysts, and think that, with more exposure, a not insignificant portion of excel users could be convinced to make the leap.