Hacker News Reader

Comment by 998244353 | Hacker News Reader

Non-technical users don't want to do that, and won't understand any of that. That's the unfortunate reality of developing software for people.

If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.

View on HN

> then my "import data from CSV" function needs to handle that, in one way or another.

It doesn't. Well, maybe "another".

Your function or even app doesn't need to handle it. Here's what we did on a bookkeeping app: remove all the heuristics, edge cases, broken-csv-handling and validation from all the CVS ingress points.

Add one service that did one thing: receive a CSV, and normalize it. All ingress points could now assume a clean, valid, UTF8 input. It removed thousands of LOCs, hundreds of issues from the backlog. It made debugging easy and it greatly helped the clients even.

At some point, we offered their import runs for download, we added the [original name]_clean.csv our normalized versions. Got praise for that. Clients loved that, as they were often well aware of their internal mess of broken CVSs.

The point is to separate the cleaning stage from the import stage. Having a clean utf-8 csv makes debugging import issues so much easier. And there are several well working csv tools such as the Python one, that can not only detect character encodings but also record separators and various quotation idiosyncrasies that you also need to be aware of when dealing with Microsoft Office files. Other people have already thought long and hard about that stuff so you don't have to.

OTOH if no one ever pushes back against entropy, your great-grandkids will still be dealing with Windows-1234 problems in 2100.

Hot to push back against entropy?

Could this work? Implement handling of ancient excel files in your SaaS product, but charge extra dollar for parsing legacy formats and provide information how to export correct files from excel next time :)

Plus, Excel really likes to use semicolons instead of commas for comma-separated-files. That's another idiosyncrasy that programmers need to take into account.

I think that happens mainly in those locales where a comma is the traditional decimal separator and the dot gets used for 1000s grouping.

I'm German and I hate this. The icing on top: Excel separates function arguments with a semicolon in German locale, too. Got me some head scratching. Examples are separated with comma, iirc.

>A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding...

What if you just gave them instructions?

I think most people would rather get in their car, drive to the ocean, board a ship and join the navy, and come back after a few years abroad - than following some instructions on how to use the computer.

Joining the navy and then be obliged to follow instructions on so many things you'd normally never even be ordered to do (because now you're in the Navy, sailor! and there are three ways to do any thing: the right way, the wrong way, and the Navy way, and guess which way we use here in the Navy?), just to not follow some instructions on how to use the computer... is something I can actually imagine some people doing.

Pretty funny, and probably some truth to that being a user sentiment.

OTOH, they have to follow some process to use the software. For just the CSV export, they already have to ensure column orders, values, formats, maybe headers, etc. Selecting UTF-8 from a dropdown seems like the easy part.

> they already have to ensure column orders, values, formats, maybe headers, etc.

My accrued compatibility shims disagree.

Sorry to hear that. You have my sympathies.

They’ll probably switch to the application that does it for them and just works, instead of the one telling them to do something they don’t really understand.

Selecting UTF-8 from a dropdown on the export doesn't seem too onerous an ask. If that's the differentiator between you and your competitors, then you might have bigger problems.

I have sat in on and observed many in-person usability sessions with various applications and websites.

In can tell you right now anyone reading this site is in a completely different universe of tech competency than the general public, and even professionals who aren’t tech-focused.

You would have lost many of them simply with then jumble of letters “UTF-8”.

I hope someday, with improved public education, we can have better users. Unlikely with Gen Z apparently having worse computer skills than the previous generation..

Everyone knowing about character encodings sounds like a lot of time wasted.

Gen Z has skills in what is available and popular today, just like the previous generation had.

Yeah, I started writing software a "while" ago. I've encountered a handful of users (and charsets) in my time.

With CSV exports, there's already a good bit of training you have to do WRT the column layout, file format, headers (or not), cell value formats, etc. There's far more lift involved in training users there than ensuring they select "UTF-8" when they select CSV.

And, there's really nothing technical about it, as they don't need to understand what "UTF-8" actually means any more than they need to understand what "CSV" stand for. It's a simple ask in a list of asks.

I weigh this against having every developer now believe they have to check/convert every character set, which can be unreliable and produce garbage, BTW. And, speaking of garbage, there are some encodings that you won't be able to convert in any case, so it would be technically impossible to preserve data integrity without pushing some requirement on the source up to the user.

So, it's about tradeoffs. And, again, if asking users to choose UTF-8 is the difference between customers choosing your app or a competitor's, then you probably need to be more worried about that than your charset encoding.

Hell we just accept xlsx and xls natively. It sucks but it solves many issues. Also causes some, but you'll never get it out of users' heads that Excel files are not a data exchange format.

Definitely helps. Should be able to detect the charset from the Excel metadata, at least.

Of course, if that charset uses code points not available in your app's native charset, then you're kind of back to square one (unless your use case tolerates garbled or missing data).

Having worked quite a bit in b2b applications, that is exactly the kind of thing that makes customers consider other options.

Interesting. My experience is that business users are much more teachable and mindful than consumers, especially when working with data.

Reply on news.ycombinator.com