Hacker News Reader

Comment by Gustomaximus | Hacker News Reader

The core complaint of not having a single source of truth seems more about how the company handles data/filing rather than Excel itself.

Maybe I'm a bit bias having used Excel my entire career and very dependent on it, but everytime I've seen companies steer away from the desktop application with 0365/sheets/tableau its simply not as useful for adhoc data analysis and reporting.

View on HN

I agree. Those are great products, if you have the staff to support and use them, but this is rarely the case. Every time I see a company implement them, they think it's going to be a silver bullet to reporting and analytics, and most of the time it just causes a mess, because no one really knows how to properly use them, or if someone does, that person doesn't have the bandwidth to keep up with the entire organization's needs. Excel is amazing, it's relatively easy to use, has immense power for dealing with calculations and graphs, and it is cheap. An organization's problem isn't excel, its the fact that they keep 20 copies of the same data and no one is keeping track of which is the source of truth.

Every email reply becomes an independent fork of the data inside the spreadsheet. The issue is storing and sending the file locally, and the solution is storing it in a centralized location. Cloud storage is often used towards this end, but on-premises file shares work very well too.

This is one of those solved problems that companies inexplicably keep working to solve again.

It's such a bizarrely difficult uphill battle. We have Office 365, with a damned good web editor, and the ability to save directly to OneDrive/Sharepoint from the desktop client, with the versioning that that supports, and we still end up with shit emailed all over and files with version-12.xlsx and -$DATE.xlsx saved here, there, and everywhere.

The web editor might be good enough today but it used to choke and wouldn’t open/render all kinds of stuff that’s normal in financial spreadsheets. Thats not even getting into what a shit show SharePoint has been over the years. Microsoft has slowly eroded trust in anything but email sharing and (if there haven’t been missteps within an org with them) file shares.

PlanSpreadsheetFinal2ReallyFinal2016_2(1).xslx

This is a key point. It's easy to branch data, but painful to merge it.

How do you even HAVE a source of truth? You email a spreadsheet to two people, and the both want to work on it... which one is the source of truth? They each have modified the original, and we want to keep both changes.

I mean, this is the problem that distributed version control was designed to solve. Emailing a spreadsheet around and calling one a source of truth doesn’t solve the simultaneous edits problem.

>its simply not as useful for adhoc data analysis and reporting.

<rant>

You wanna know what's even more useful? Code notebooks.

I store everything from my sql queries to my processing pipeline to my models, and my vizualizations. Most notebooks have slide-show modes now so your presentation to business stakeholders is simple to prepare as well, and with the right plotting packages you can draw interactive charts that do all the fancy stuff. And when it's time to update the analysis, you just re-run the notebook.

But business analysts are too freaking lazy to just learn to write some python or R, and they're perfectly happy to get paid to do the same task over and over again when it comes time to making updates.

And don't get me started on source control.

</rant>

Do you have any way you'd recommend of getting multiple users working together with code notebooks? I've messed around with Jupyter, but that's about it.

We usually just send .ipynb files around because of the stuff we do, we all usually have the same libraries.

Installing a notebook server is an option though. IBM will be happy to let you pay to have them help, or you can do it the sane way with continuum analytics.

Oh yes. I have tried to introduce jupyter notebooks with pandas to data analysts so many times, but they always want to do it in excel.

Most business analysts who are pulling this kind of data have a finance background. To say that they're "too lazy" because they don't want to do their work using Python seems a bit harsh.

In practice what I've seen is that IT and finance departments don't even let business analysts write their own SQL queries for fear that they'll write bad ones that bring the database server to its knees.

>In practice what I've seen is that IT and finance departments don't even let business analysts write their own SQL queries for fear that they'll write bad ones that bring the database server to its knees

I'll take data warehouses for 600, Alex.

I use jupyter a good amount, but even in a low user environment there's a big worry about version control and code sharing/review

I know there's " automatically check stuff into git" sort of models but would really like an integrated Jupyter + GitHub + Google Drive system so that it's easier to track what's happening and to find older versions of stuff.

This!

I happen to use and advocate for R/RStudio for ad-hoc work (I spin it as "excel on steroids" when I am trying to convince someone). But it is hard to pull people away from something that has worked for them.

The alternatives to excel in the past have been really, really unpleasant to use. A now EOL product from Oracle comes to mind. It is called "Discoverer"-- that's right, "Discover" + "er". It is a turgid POS requiring extreme suffering and handholding/permission granting from your local friendly DBA-- all this to produce 90's era bar charts that are all but impossible to tweak.

IMO if you really want non-tech users to become more technical and use DBs as single source of truth, use KNIME. It still needs proper reporting addons though, but you can export data, even to tableau.

I saw a presentation about KNIME but was unfamiliar with many of the processes that it allows one to string together to build a workflow, so the notion of building a workflow myself seemed very daunting. Do you think it would allow one to build workflows for basic stuff and gradually grow in their confidence with it?

The presentation mentioned that the company is/was working on reporting add-ons, I think.

Oh yeah, I'm pretty dumb and I managed to do very complicated stuff. It seems a bit tricky to "get" at the beginning, but it's pretty easy.

The trick is to learn what node does what, because there are lots of them. You can also download community nodes for additional behaviors. Once you learn the basics, it's pretty straightforward.

I learnt by trial and error, looking for examples and info about nodes, and now I have some workflows with data from DBs, files, APIs and all at once.

Totally agreed. Especially when the alternative is abusing an ERP for ad hoc reports or using a journey map approach that captures the top X use cases, but misses what is actually needed.

Agreed. Excel users should learn more about importing data from other sources and processing that. Excel itself should make it even more obvious.

I'm not sure of the origin, but I was once told a truism that any speccing process, if left long enough, will eventually spec out Excel.

Reply on news.ycombinator.com