The big data successor of the spreadsheet | Hacker News Reader

The big data successor of the spreadsheet

medium.com

61 pointsmisterdata31 comments

I think the problem now is not a better excel but a better access; spreadsheets are pretty good at what they do but biz people use them when they should use a database because they find databases less intuitive.

Yeah - my reaction to the idea of "spreadsheets for big data" is that it kind of misses the point of why spreadsheets are so popular, namely that most data is small data and a sheet of rows and columns is a super easy way to wrap your mind around a set of small data. So the big challenge is coming up with a way to work with small data that gives you the benefits a database brings over a spreadsheet (relations, querying, etc.) without losing that immediate "oh, I get this" feeling.

Excel already has relations, albeit simple ones, as well as simplistic forms of querying, though filters and pivot tables.

I would argue that the major benefits that a database offers over Excel is data integrity and consistency. I think the lack of that sort of constraint is a major aspect of what drives 'biz' people to spreadsheets.

The core problem of spreadsheets is that they should have been organized around tables with fixed data meanings from the beginning of the history of spreadsheets.

Instead the sheet is both layout tool and data "meaning" in one glob of garbage.

Apple tried to solve the issue with "Numbers" but ymmv on how well.

I disagree. The SUCCESS of spreadsheets is that they are general-purpose tools that deal with information laid out on a grid and they would be nowhere near as useful or powerful or indeed popular if it were otherwise (c.f. Numbers).

"Data laid out on a grid" is a paradigm that applies to a wide range of problems and spreadsheets are pretty much the only tool around that can deal with that entire paradigm. The user isn't stupid. The user has a use case that isn't being catered to.

If users had a clean and straightforward way to define the permanent header row, and viewed it as just what you did for a spreadsheet, the whole industry would be better off.

This is probably a glass half full/empty at the end of the day.

Fact is spreadsheets don't scale because they are sloppy and they don't provide a clean coherent path to getting the user to scale themselves into a sql like solution.

Oh well.

But doing that would limit spreadsheets to a tiny subset of what they're used for in the real world. If you have a permanent header row then you limit yourself to one table of data per page, which you often don't want.

Excel's tables are probably similar what you mean, but Microsoft rightly allow them as elements on each sheet rather than basing the sheet around one table.

"""Apple tried to solve the issue with "Numbers" but ymmv on how well."""

Your "tiny subset" is your confusion at not seeing numbers I suspect.

Wikipedia actually does a decent job of trying to describe the philosophy.

Numbers just doesn't do a good job of representing data grid information that isn't rows and columns of numerical/textual data. Its paradigm is that takes that subset of use cases and cleanly separates the presentation layer.

Excel has Tables, which sort serve a similar purpose, but the sheet itself is a flexible grid with no pre-conceived purpose, which makes it much more flexible than Numbers.

For example, it's relatively easy to knock up a simple Gantt chart by directly coloring cells, something that I've seen often. And you could argue that it's not the best tool for the job, and you'd be right, but it's easy and does the job well enough in many cases.

And that's why it's popular and iWork isn't. Numbers concentrated on making charts and tables beautiful; Excel is a general-purpose programmable platform.

There was a really cool app on NeXT called Quantrix (Also Lotus improv had the same idea). It was a spreadsheet where rules and data were kept separate.

The modern java version can access databases. A spreadsheet user could get up and running with it over the course of a week or two, then switch to a database behind the scene.

I feel the same way. In the 90s you'd find small businesses (and some larger ones!) built entirely on a single Access database on a network share. At the current small business where I work, we use google sheets instead, and it leaves a lot to be desired.

For the past few years an easy-to-use database interface for small businesses has been my primary idea of what I'd like to build if I founded a startup.

QuickBase(.com) may be something close to what you want: while mostly aimed at > 10 people businesses. It's not great, but it does have a reasonable amount of power... especially for apps with say less than 10 tables.

DabbleDB was a good one for individual users, until Twitter bought it and shut it down. Google says grovesite.com is a competitor... and I also think Dabble's founders were debating rewriting it.

There's always the old FileMaker Pro, or databases of that age / time. (4D??)

@jewel, @kikidrew, @rpwilcox: We're building this: https://fieldbookapp.com. We're still in private beta, but message me (jason@fieldbookapp.com) and I can get you an invite.

i still work at a business that is built around access and am desperately looking to move away into something html5 but as easy to use. it's insane that no one has done something that just replaces access on the web.

you should definitely try to give it a go.

Agree 100%. In the span of a decade or so, the goalposts for what a "small business app" have moved dramatically: multiuser by default, offline editing/sync, usable on mobile devices, modern-looking UI/UX... but the Access-like tools never made the leap. They're still stuck in the ancient past, and effectively unusable now. Why hasn't anyone been able to build a modern equivalent to Access yet?

Google Sheets is laughably primitive when you try to use it as a multiuser database. It's impossible to validate input and preserve structure (giving a user edit permission, so they can insert new data, also allows them to remove cell validation and destroy the structure of the spreadsheet). Something as simple as copy-and-paste (Ctrl+C and Ctrl+V) will destroy all of your carefully placed conditional formatting and validation constraints. And sometimes a formula will just "break" for no reason at all, only to start working again a few days later...

I'm honestly surprised that this is what we consider to be the start of the art. Where's my easy-to-use cloud database?

There's Libre Base.

Agree. I wok with FoxPro (like Acces, but better and powerfull) and I miss it. I enjoy python, but the DBase family is missing.

I dream in build a moder relational language, but then I don't have the resources for doing it... However I still do some code/meditation about it each week...

Soulver is the best evolution of a calculator I've ever seen, think of it like a dynamic paper tape of calculations you can work with and alter and it keeps the results updated.

"""The interfaces require the end-user to know about the concept of a table, rows and columns, the concept of logical expressions, as well as aggregation, even though none of those are directly visible nor have a ‘real life’ analog."""

This sentence expresses the problem I see with designers.

Instead of making that disappear into magic why not focus on actually purposely and clearly educating people about these CORE concepts?

edit: Think of how you would do this if you were Khan academy...

Instead of making that disappear into magic why not focus on actually purposely and clearly educating people about these CORE concepts?

Because "people are dumb"? Well, people aren't dumb-dumb but just as you aren't going to get everyone to code, you aren't going to get everyone to understand abstract logical relations (not generally, throughout society, though anything is possible in a single enterprise).

Spreadsheet maintain their dominance in places that they logically don't belong because you will always have a person who got their position, in say a brokerage firm, through social intelligence, language intelligence or "sheer guts". And they will want to be able to manipulate data without training and spreadsheets let them. Heck, we can't replace the Excell interface with a better spreadsheet interface.

The goal isn't teaching everyone.

It's to teach the very limited subset of business analysts that CARE about the answer to learn their tools well enough to ask the question properly.

Generally I think we are seeing some of these things should probably be part of much earlier math education.

Sets, predicate logic in 7-8th grade?

The pragmatic reason is that it takes a lot of time. As a product designer at a company that actually tackles exactly this problem, we generally have approximately 0ms to stop a user's workflow in order to teach them a cool concept.

Building intuition DURING their workflow is where things get really exciting for designers though.

Edit: Should point out that I'm not quite sure if you're suggesting we teach this in school, or...? But I read this as a designer's job to teach this in-product.

0ms is a good goal for calling lyft or uber.

It's probably not a great goal for coherently querying data in a flexible and powerful way.

Having very little friction when your user has attained some fluency is a great goal however.

Author here. I agree with your remark but it simply isn't reality. Another issue is that while users may be aware of the core concepts, they are applied differently between applications (e.g. a relationship in Excel between tables is based on a VLOOKUP formula whereas in SQL it is a JOIN or carthesian product).

I do think data tools can educate users, just like a spreadsheet interface teaches its users about the concept of formulas and relationships between cells.

It "isn't reality" because it's a very hard problem that hasn't been well solved. That should intimidate you enough to take it seriously but not to keep you from trying.

While the concepts are similar the brittleness and bad design of spreadsheets (as I say elsewhere in the thread they should have had tables as first class object from day one) limits the ability to really make them a sane solution.

That being said, spreadsheets (some running hedge funds) are a testament that the core concepts are learnable.

http://en.wikipedia.org/wiki/Powerpivot

I was about to post a comment on power pivot myself.

We were building a lot of tools in R but eventually switched most of it to power pivot. My onl complaint would be that it isn't practical to refresh it with new data, it requires more manipulations than I would like.

I work in a big, dumb enterprise with too many spreadsheets and Access databases.

What made me move away from Excel spreadsheets:

1) As in the infamous case, its too easy to miss extending a range when you add new data[1] 2) There's no good way to manage concurrent access 3) Doing aggregates of aggregates is really hard

What made me then move away from Access databases:

1) Excel and Access attempt to guess the "type" of cell contents automatically, and this creates issues (Is 060E2 a product, or a scientific number? Of course, its 6000. Is 3/6/2015 a US date? Access will figure it out without you.) 2) There's still no good way to manage concurrent access 3) Storage limits still aren't that great if you want to deal with more than a few GB of data

For me the ideal spreadsheet would solve all these problems :) I moved to Postgres, instead.

[1]http://www.bloomberg.com/bw/articles/2013-04-18/economists-s...

At my workplace we use SAS Enterprise Guide for some of this stuff. I like it because it has a drag and drop GUI that non technical users can utilise to extract data (It executes SQL queries in background) and do simple operations (like aggregations, statistics, graphs, merging, filtering etc) At the same time it has a decent programming language underpinning it and pretty advance statistical capabilities.

On the down side it still suffers from some of the problems the article references. Some non programmers, even quite technical people simple don't grasp concept of querying and joins.

It is also horrendously easy for the auto generated SQL to be horribly inefficient the default appearance of the plots and other graphical output looks ugly etc, which leads to people doing things like extracting subsets of data from SAS and then pasting it into excel to manipulate it. Which is kind of self defeating.

The "pipeline of blocks" idea goes back a long way (Visual Smalltalk anyone?), I'm excited to see it applied in this way.

Datameer - big data spreadsheets

Scratch - best thought out block language

Reply on news.ycombinator.com