Hacker News Reader

Knime. Desktop app is free, is not as polished as alteryx but with some community love it will improve massively. It already is WAY better than Excel.

Stuff is organized in a manner someone unfamiliar with data handling or programming can understand it.

It takes a little bit to master, an some stuff (like looks and variable passing) is still unintuitive.

Is it just a better "Excel" or does it have some actual feature to prevent people making mistakes?

It isn't a spreadsheet.

It uses a visual data flow approach to transform data step by step from one form to another. This approach is much better for data transformation. It is also less error prone as operations take place on columns or rows of data, not cells. But you wouldn't try to create a set of company accounts with it.

Take a look and you'll understand it right away. It's organized with nodes and links. Nodes perform operations, and links transport data between nodes. You can arrange them visually as you please. There's even stuff like "metanodes" that allow you to put nodes inside, so you don't have to have a messy playground.

Most nodes (there are official and 3rd party nodes) do have error output, and there's a console to inspect what's going on.

You can also inspect data visually in most nodes.

Even someone who is not used to play with data, who doesn't understand consistency and yada yada will understand the advantages.

This is amazing. Thanks for pointing out knime. There were so many occasion since and including uni where I wish I could've used that.

It's pretty useful for data manipulation. I use it for very heavy lifting (millions of rows with A LOT of text manipulation, regex, and even some ML) with a Dual Xeon + 32 GB Ram and it's a beast. It takes some time to master, but I encourage you to play with it.

And how does knime/everything else compare with PowerBI, given how aggressively it is being pushed by Microsoft, and the fact that it would come bundled as part of most organizations Office 365 deployment.

When you see it, you understand it. Knime interface is designed around "data pipelines" if you wish, completely different from PowerBI, which seems more oriented to dashboards.

Instead of only considering a consultant, the company should have a developer on staff to work on small tasks like this. The cost gradually decreases as they learn the business and don't need everything explained from scratch.

See. You've got answers to something that you haven't asked a single question about. But in your judgement the company should have had the right developer on staff.

That's what's frustrating about these anti-Excel discussions.

Ok. What happens when I say that the company bought another company and it had a product line that the new company wanted to pay sales people commissions on? And the compensation plan around it was complex.

For whatever reasons, that's not a report the DBA was up for developing from scratch (or, someone decided his time wasn't to be used that way). That's just the reality.

We can talk about what should have been or what ought to have been, but that doesn't change the reality.

The reality is that I ended up doing most of it ACCURATELY in Excel.

> But in your judgement the company should have had the right developer on staff.

Yes, like a janitor for example.

> The reality is that I ended up doing most of it ACCURATELY in Excel.

Yes it's the hammer for all your nails. You could also use Python+Jupyter. There are selfhosted and managed versions.

Because of guys like you, municipial installations run their sewage system with an excel sheet because some guy implemented a vba macro to talk to the sewage api endpoint.

let it go.

I think the company would quickly realize that they need a different solution if they have no Excel wizard on staff, and the team spends weeks just calculating sales commissions.

Or maybe not, and they’d just hire more people solely to calculate commission :/

I would say that you probably do not know what you did wrong having done it in Excel. Stuff gets complicated really fast in Excel thus it’s much easier to do something that you can’t possibly debug and thus live in blissful ignorance.

It's just unbelievable ... how people KNOW about situations that they weren't in and don't ask any questions. But since you brought it up I can tell you about my accuracy in Excel:

- The process involved 2 people checking my work before it was sent to payroll. And we did catch and correct mistakes. We were able to trace back where and how the mistakes were make, and build in ways to prevent or flag future mistakes.

- The job was necessary because of all the garbage in the database. Excel's data-checking and data-cleansing and flexibility was superior to the database.

- After 1 year all of my work was audited. For several dozen sales people, there was ONE person I wasn't 100% accurate with. And that one was off by 0.5%. SUCCESS!

- If we'd gone with the report coming out of the database, things would have been such a confused mess.

It has nothing to do with your particular situation. This is a reasonable assumption to make if somebody comes and wants a solution in Excel.

But ok. So what you are saying your Excel was reviewed by somebody else. How is this any different from reviewing somebody else’s code?

I work in a company where even the least technical analysts are capable of using numpy. We have libraries to use Google Sheets as input-output data. So you get the best of the two worlds.

What does Excel resolve here?

Also I don’t believe that you could not use Excel to cleanse the data and then put it back in the database.

And I also do not believe that a python script would not do the same cleansing of the data in a much more maintainable way.

You've got all the answers about a company in Chicago in 2005-2008 and how it should have been run.

Why didn't you jump in and help us?

I am not giving you advice on how to run a company, I am saying that your anecdotal opinion will not affect my decision-making process. And I am clearly stating that it should not affect the decision-making of other people here either.

Also I am going to underline that what you did was relevant 16 years ago; which you did not state in your initial message about the usefullness of Excel.

Which is kinda funny, if I didn’t ask you would have otherwise omitted a lot of otherwise relevant information which makes your opinion even less important.

It only makes me question why do you take this whole thread so personally? Probably because by now your skills are way less relevant than back in 2005.

Not just a developer. A developer that's clued up in business needs, can communicate (listen, clarify, translate technical concepts), has a smattering in finance to promote cost, time and/or quality savings they make. And the company needs a succession plan should such a person not turn up.

Like a Business Analyst, sorta?

What is the first tool a BA would reach for for most of these problems?

Excel is the proverbial hammer that makes everything look like a nail. It is very flexible, powerful and beginner-friendly. And while you can solve a lot of problems with it, some of those should be handled by other programs (or other people). The problem is that the average Excel user doesn't have the right knowledge to judge what the best solution is to their problem. And sometimes it only becomes clear after long use that they have created an unwieldy tool which causes many unforeseen problems.

Again. What's the comparable alternative?

This is so frustrating because I've worked in small departments in an international company. Our problems were too small to get consistent, needed help from IT and we weren't important enough to get one of the limited licenses to the fancy reporting system.

Ultimately, there's all this bad talk about evil Excel, but we get no help or other resources.

And what about all the junk reports that were coming out of the company database that no one wanted to spend the money to fix?

THE ONLY VIABLE ALTERNATIVE WAS EXCEL.

No. it's not the only alternative. it's the only thing you and many other accept. Python+Jupyter is a real alternative.

You go tell the Director of Customer Service that his people who are handling critical tasks need to learn Python. Good luck.

exactly. because he does not see a problem with this because people like you tell him it's the only solution since dawn of excel.

But when developers complain they have to learn domain specific items, people say the shall keep up with the times. but suddenly the analysts and the others are exempt from this?

Real life doesn't always provide ideal solutions. In your case, using Excel is probably the easiest or even only way to do what's needed. That doesn't mean it's the best, or even a good solution. If it causes organization-wide problems, whoever runs your organization should make sure better processes are instilled. The only thing you can do is make it clear to the person above you and hope things improve.

We're closer in agreement. And I'd never suggest that Excel was the best solution or even a good solution at times. (I hated the tedious error-prone things that I used to do in Excel.)

I'm just offering the context from which weird stuff happens. And the answer isn't always as easy as: put it in a database.

The context, the processes (or lack of processes), the decision-makers, budgets, timelines, they all play a part in the outcomes.

The alternative depends on the people involved. In many instances it's a relational database, perhaps in combination with Excel for charts.

One thing Excel is extremely bad at is reliably importing data from CSV. It's unbelievable what errors you can get if you're not extremely careful.

I think the question is what a viable alternative is if the company does not want to spend any more effort than the effort of the singular employee that wants the report.

They can do it by themselves with Excel. Can they do that with any other software?

Yes, some can, but depends on the task.

Not many people would be able to do complex calculations in an RDBMS, especially if it requires joins or grouping. But many more are able to use a GUI tool to create a single database table and import some CSV data.

The recent UK disaster wouldn't have happened if the data had been imported into Access or SQLite instead of an Excel sheet.

Of course it's not a panacea. Badly designed databases can be just as error prone as Excel, but they have different error modes.

R, or Python.

"The right solution to their problem" being a developer, at least as far as developers are concerned. Hammers and nails...

Sometimes that database feels more like a jetplane when all I need is a hammer for my nail. Excel (or libreoffice calc) is perfect for a lot of stuff, especially those small things that you won't ever do again.

100%

Excel is a UI over a database table. With a bunch of other UI driven math/stats functions built in. It really comes down to a difference between UI driven folks and folks that are code literate.

Your position is more reasonably explained than most of the Excel Haters' (I am a card carrying member of the group, btw) but there is an additional problem.

Even if/when the "average Excel user" finally sees the Light ... s/he still has to work in an environment where excel files (including .csv) are the lingua franca.

It's a bit like working for the German (or French or Japanese etc.) branch of a US large company. Of course the official language is English, and all documents should be written in English and emails should be in English and so on. Even if everybody (down to the Janitor?) speak acceptable English and are perfectly able to talk business in English... everyone will still use German unless they have to communicate with someone like me, whose German fluency is rated in Imaginary Numbers.

To me it seems Python or Julia notebook stuff like ipython or jupyter could (and did in some circles) displace Excel in almost all cases. The self-sustained inertia to change is real though.

Who's going to pay to train a Customer Service Rep, Admin Assistant, Sales Assistant or Warehouse Manager to do Python when there's Excel?

Get into the world of the Excel user. Some of us do learn coding. But generally speaking? No. If it happens, it's on our own.

And we can agree that spreadsheets have problems, and Excel users typically are not data people.

BUT!

When we agree on that, it becomes evident that a lot of suggested solutions are non-starters. Not all Excel users are in an enterprise that's throwing money around for training and dedicated software.

I'd like to see the Warehouse Manager who goes in Monday morning and says to the company vice-president, "I've been looking into this Python class ... "

The problem isn't businesses using Excel, the problem is businesses using Excel for critical processes.

The Warehouse Manager or Sales Assistant doesn't need to learn an alternative, but e.g. the Finance people should. PowerBI/Tableau for reporting and R/Python for modelling should be a thing in those worlds (even if Excel continues to be an amazing tool for entry level stuff and some quick number crunching).

> PowerBI/Tableau for reporting

In my experience these take ages to load data that is near instant on Excel. I can totally see how people prefer excel.

It may take longer to load, but it allows you to automate the data processing. That's where the effort (and errors) usually lie...

How do you have so many answers to situations you haven't been in?

Do I need to say it again? My Excel skills were necessary because of garbage in databases.

I could sit at my desk; extract a data-dump of 50k rows of data; locate the fuzzy matches, identify which records needed to be kept and which ones deleted; merge it with the client's spreadsheet that they emailed to me and compare them; identify problems and report back to the angry client.

Why was the client angry? Because of inflexible garbage reports coming out of the database that people were blindly trusting.

Why was there garbage in the database? Not because of Excel.

With jupyter, you get the problem that editting cells does not automatically make later cells get recalculated. That alone, in my opinion, is worse than any of excel's failings -- imagine a spreadsheet which could get out of sync like that?

Charting in excel is a million times faster and easier than having to learn an entire language.

Can’t we simply accept this for what it is? Code nerds preferring a code solution while normal people simply use the modern equivalent of graph paper?

BRAVO

Try deploying python apps in corporate environments with fairly locked down machines to non-programmers and get them to perform simple operations, and then compare that with just emailing someone an excel spreadsheet.

Libre Office Calc covers 99% of what Excel use cases, has a very similar UI, and doesn't do secret magic guessing of whether a value is a date or not.

>, and doesn't do secret magic guessing of whether a value is a date or not.

This Libre Office question&answer[1] in 2020 says it guesses that the users want leading zeros removed from numbers. This behavior is similar to other software tools I looked at.[2]

This means that if a csv has "02142021" as a date for Valentine's Day with a leading zero, it will be imported as number "2142021" -- unless one takes extra steps to prevent the default behavior.

[1] https://ask.libreoffice.org/en/question/246295/how-can-i-imp...

[2] https://news.ycombinator.com/item?id=24696732

Ok, it might try to deduce the data type, but you can set it in the import options that appear when you open a CSV file. Excel will automatically convert a string like "2-4" to "02-Feb" upon opening a CSV file. No import options appear when you open the file. If you then save the file it then uses that new string. As far as I can tell there's no way to stop it doing this.

Gnumeric or Tex combined with R can help also...

There is ages since I opened an excel and I don't miss it.

BRAVO for you. And I mean that. You've got the tool YOU need and you haven't pushed YOUR tool on other people. Thank you.

Metabase (for reports rather than transformations) was useful to fill all the "cant we just have like a spreadsheet" requests while maintaining some quality

- hypothetical Excel as a DB client app with MySQL as backend