The world needs better data manipulation tools than Excel and Access

Besides Dabble (which can kinda cut it for certain non-tech companies, but is primarily used by tech companies) Excel is the only real option and is probably Microsoft's best product.

Sure it may have a few strange date bugs. Its column/row order shortcomings in vlookup() and hlookup() may be annoying. But you can really, really, really DO stuff with excel.

The learn curve for the software is basically Y = x^.7 (0<x<1) so getting started is as easy as you can get for a tool that can do all of the below:

Math functions (I could have individually listed them, but they are many, including correlations for many statistical distributions) Financial functions (Mortgages, Black-Scholes, Variance) Multi-Dimensional Linear Optimization - seriously, this can (with a bit of tweaking) also give you all non-inferior solution sets. VBA - I hate programming Microsoft languages, but VBA was designed to be easy, and with excel it is SO easy. Relative cell references. Stunningly beautiful graphs - Next time you are at excel and you need to show data really dig into how much you can change. You can make these things look extremely pretty if you take some time to learn how.

The nice part about all that stuff above is that the new/uninformed users don't every really have to worry about it. I've met tons of young people who got into programming because they started with macros (and later VBA) for Excel.

I'm don't agree with the article's premise, that "the world needs better data manipulation tools than Excel and Access".

Most of my present colleagues are not programmers, and they manage to create relatively complex applications with Excel and a bit of Access. There's no gap between what they can do with their tools, and what I can do with a real programming toolset. Also, I can't see one tool fitting everyone's needs.

I have also observed that most non-programmers have an allergic reaction to anything that looks like code, even something as basic as VBA. They'll do acrobatics with all sorts of weird macros, but that's OK because it takes place in a GUI; there's very little syntax to get wrong, or structure to create.

I think programmers underestimate the learning/confidence people need to code even a simple program.

See, I was going to agree with the headline until I read the article. I was thinking along the lines of a data warehouse version of Mathematica with distributed processing and web front end and RPC/REST and all of that.

No, the author wants to "integrate a (not yet developed) visual programming language." On the JVM. What will this accomplish? Well, it would allow users to "create fairly sophisticated apps by dragging, dropping and plugging things together." But fear not, programmers: "Programmers wishing to make more sophisticated apps would have at least one real programming language with a DSL or DSL-like library for interacting with the system."

Wonderful, another do-everything GUI app development environment where users can do more and programmers can do even more than that. Ugh.

"Does anyone who understands the needs of the corporate market have some input?"

Sure, make sure you check out the billions that have been spent on this very problem:

http://en.wikipedia.org/wiki/4GL

http://synthmaker.co.uk/ http://synthmaker.co.uk/about.html < pictures

This tool is designed for audio DSP, but is not limited to that. If you're interested in this topic, I recommend downloading the trial, putting it in advanced mode, and just futzing about for a few hours.

It's very accessible and fast to work with, and could be adapted to quite a lot of non-DSP tasks. I would love to see it generalized: the modules within are how I feel libraries should present. Draw graphs for data flow or complex functions, open up a code box for tight procedural stuff. Perhaps not what the author had in mind, but audio synthesis of this kind is event-driven, asynchronous and features a high degree of interactivity, so it may be worth further exploration.

Please email if you know about this kind of cool stuff in other contexts.

R is now at the top of my programming languages to learn list. So, obviously, I don't know R yet, but is it something that could work well for this? Is there a way to skin it with a GUI/web front end that would make it more powerful than Excel, but still make mathematically astute end users not feel like they are "programming?"

In the theme of "ideas are worthless," I have a vague idea that a "Heroku for R" could be a winner. Collaborate on datasets, pay for extra capacity for long jobs on large amounts of data, good UI for quickly generating graphs from datasets, integrate with existing data stores like Simple DB, Couch DB, etc.

Does this make any sense at all?

This idea has a lot of potential, especially if you were to focus on a few situations that I've seen over and over in the corporate world. The most common scenario I've seen is data coming in an extract format like CSV, containing information required for another business process. You've got data coming from an expense reporting web app that needs to processed before entries can be made into an accounting system. A spreadsheet or database is created by a computer-savvy manager, and a clerk uses it on a weekly basis to do their work of posting entries. If the data format changes, the manager tweaks macros and formulas to adjust, and life goes on. Any new clerks coming on board learn how to use "the spreadsheet" and the business process keeps chugging along. Change managers, and 9 times out of 10, a lack of documentation and knowledge transfer will cause things to blow up in a short period of time. If you were to introduce a tool that exposes the funky macros and formulas in a more visual, understandable fashion, I think it becomes more easily maintainable. Providing some data validation, error handling and notifications would be valuable. Finally, provide some version control to prevent accidental deletion or modification of important functions. The key would be a much better user experience than the Excel function editor.

There have been a number of competing products. Two that come to my mind quickly are Lotus Notes and DabbleDB.

I wasn't too interested in the linked blog post, but the title made me think of Resolver.

http://www.resolverhacks.net/resolver.html

http://code.google.com/p/resolver/

I have no experience with it (I'd be interested in comments from those who do), but the premise is intriguing and it appears to be non-vaporware, if still beta. (N.B. It appears to be going in the opposite direction, in some ways, from what the posterous page is proposing.)

----

Separately, I'm reminded of a recent page I encountered where someone describes (including how-to steps) using SQLite as a backend for Excel, for handling large datasets that benefit from the Excel interface (without getting into the whole Access thing/disputes).

Googling, it looks like the below is the page. I haven't read through it, yet, but I'd made a mental note to do so sometime. I'll still mention it in case it intrigues anyone else as it did me.

http://blog.gobansaor.com/2009/03/14/sqlite-as-the-mp3-of-da...

The world certainly needs better data manipulation tools for programmers and non-programmers alike BUT ... neither group feels like strongly enough to make the market for spreadsheets and user-level database tool competative again. It's sad.

It shows that individual users drive markets rather than "the world" driving markets - I don't mean this snidely. There are many things that people would benefit from if they thought from the point of view of "the world" But that thinking is not common.