Hacker News Reader

> why is locale stuff even in sprintf? What things get localized

Decimal places in floating point numbers is the primary standard thing (some locales use `.` as a decimal seperator and `,` as a thousands seperator... some locales swap interpretations!) Nonstandard extensions may also do things like accept unicode strings, wide unicode strings, etc. which may need to be re-encoded to whatever the ambient locale-specified narrow encoding is (UTF16 => windows-1251?).

In the modern era of sprintf being exiled to some low level internal thing, I agree it makes little sense - causing more bugs than it fixes - but in the days of using it for a lot of heavy lifting of user-facing data, it was an understandable target for this kind of treatment.

Yeah, as a french I often have to set my LANG to C because of how many buggy apps that use sprintf without fixing a locale exist (making save files at best unshareable with the world at worse unreadable again)

The overhead is completely unnecessary in far too many cases. So what they should have done is kept sprintf to its traditional behavior and invented a new, slower alternative for those fairly unusual cases that need locale sensitivity. Locales tend to be all or nothing and extremely inefficient to make general use of anyway because too many things are controlled by the same setting.

Standards committees should not carelessly sit down and devise things that destroy performance by a factor of ten or more, especially when it is entirely unnecessary and where they do not keep or provide simple, fast alternatives for the most common cases where the format is known and has nothing to do with a preferred locale. What they did instead was a massive energy and time wasting imposition of unhelpful semantics on functions that did not have or need any such thing instead of a tailored let's only do this in special cases with special functions approach.

C++ has thankfully locale- independent functions now, std::from_chars, std::to_chars.. but it took way too many years

I am very curious to know why you guys are forgetting std::to_string. It is available since c++11. This and then str.c_ctr() should make the trick in most cases.

to_string is just a wrapper around sprintf... here's the (de-__uglyfied) code on libstdc++:

    inline string
    to_string(float val)
    {
      const int n = gnu_cxx::numeric_traits<float>::max_exponent10 + 20;
      return gnu_cxx::to_xstring<string>(&std::vsnprintf, n, "%f", val);
    }

it will be worse in pretty much every metric than fmt::format("{}", val);

to_string uses the locale.

sprintf/setlocale date back to at least C89, and probably even earlier - over three decades ago, this wasn't the performance and multithreading hazard it is today, because people simply weren't multithreading. Global state was the practice de jure - even reasonably so, compared to drowning everything in a sea of redundant parameterization on monochrome 80x25 terminals, in an era of smaller programs, when global state was easier to reason about.

The C89 standards committee wasn't "careless" here - it was a product of the times. Of course, when modern APIs and standards committees duplicate sprintf's issues, it's far more fustrating - even when justified in the light of "backwards compatability".

The worst part is that there is no way to turn off the locale sensitivity.

This is bad for not only performance, but correctness. If you are writing a JSON serializer and you use sprintf() to format float/double values, someone can make you produce incorrect output by setting the locale to something that uses "," as a decimal separator.

And there is no way to turn this off!

This is one moment where the difference between an application language and a system language seems clear. An application language could maybe assume that string formatting is for showing to a user. A systems language should assume that you are implementing protocols, and that mere user preference should not change your output.

The old way to turn it off would be setlocale() in your library before and after sprintf, but this is a disaster for threads, and so now there's the _l functions.

glibc only supports a few _l functions. It doesn't support snprintf_l, for example. The solution here is uselocale.

Well, you can replace setlocale() with your own setlocale that uses dlsym() to look up the "real" set locale, and your setlocale can just call the real setlocale with params (LC_ALL, "C").

That turns it off pretty well. Even if other libs call setlocale (looking at you, gtk), they'll get your version.

like so: https://github.com/smcameron/space-nerds-in-space/blob/maste...

Kind of like a bug in PHP where identifiers with i in them didn't work properly if the locale was Turkish since PHP managed its case insensitivity by uppercasing the identifiers, but used a locale-dependent format in which, if the locale specified Turkish, would uppercase i to İ rather than I.

Chrome had a bug in how css animation values would get serialized, where you'd end up with "5,43" instead of "5.43" in many European locales.

It took surprisingly long to track down, partially due to this all being dependent on various ambient Information (env vars, browser config, arguments passed, what exact function is called, etc).

My joke would be there's a difference between a programmer's laptop, and an application user's laptop.

Install linux on your laptop, seeing the source code is worth it.

> bugs me with various libraries is when someone has made the decision to do something high level at a low level. For example, localizing inside sprintf

I'm more surprised that sprintf is seen as low level. It formats and outputs strings, just by that token the problem space is enormous. Localization is just a small part.

People are fast to forget the good old days where sending specific user strings to people would crash their browser or os. "Plain" text is not simple.

The worst libraries _only_ accept a file path or URL for unzip.

I've seen audio libraries where you can only provide a file path for playback. Guess you're screwed if you wanted to get a Vorbis file out of a tarball, decode it silently faster than real-time, and send it over the network...

Selective untar into a FIFO pipe. Not optimal in this case but you can work around the limitation with some extra work. I'd rather do this than expose some 0-day due to odd combination of rarely used libraries.

Or memfd + mmap + /proc/<pid>/fd for those libraries that unconditionally want to seek.

This also works for annoying APIs like execve ;)

That's likely the use case for 99% of people who zip things though or play audio. Your corner case is exactly what simple libraries should avoid.

The only thing they're going to do is open it as an ifstream. Zero value is gained over taking an istream instead; initializing it is one bit of extra code for the user, in exchange for being able to pull in data from anywhere. Embedding it in the binary might be rare, but how about stdin? Or for that matter the Internet?

Recently I tried to make a Zig program that parses and formats some format containing numbers quickly, and I discovered that (after getting rid of all the places where one read function would repeatedly call another function that handles arbitrary-sized reads asking for 1 byte) I was spending a lot of time parsing UTF-8 strings to determine their length. This was surprising, because I didn't write any code that's concerned with that sort of thing! It turns out that any time you ask for a formatted value to be padded, the stdlib fmt routines will helpfully format the value, then determine its number of UTF-8 codepoints, then pad it with whatever padding byte you supplied until the number of padding bytes plus the number of UTF-8 codepoints in the formatted value is at least the width you asked for. It will do this even if you were trying to pad a number with zeroes. So now we're compiling against our own stdlib with one line changed to not parse UTF-8 strings when you ask it to format numbers.

Luckily none of this stuff cares about locales.

Lucky you, in Europe I have plenty of fun working in projects where the files being parsed for ETL are written in the local language, or not, while the host OS might be English, or not.

I get your point about bloated NPM libraries but I think it's also ironic that NPM is the only registry with some of the _smallest_ packages which are equally as annoying. left-pad, ansi-yellow, ansi-red (yes, packages for a single console color with over 200k monthly downloads), is-odd, is-even (which of course, depends on is-odd and inverts it), etc.

To be fair a lot of those super tiny packages like the ones you listed were created by the same person. They have over 800 repositories on GitHub and hundreds of (probably too) small published npm packages. There's been discussion of that person and their possible motivations on HN before.

Don't forget about "is-thirteen" and its inverse lol

https://www.npmjs.com/package/is-thirteen https://www.npmjs.com/package/is-not-thirteen

These are known as "kitchen sink" interfaces.

Unfortunately, it takes experience for people to realize that they're not such a good idea. A better design is the component approach. As you suggest, a zip library should not be doing file I/O.

For all the memes Java’s IO BufferedReader nonsense is really really composable because of this design.

Maybe I'm dumb, but locales make no sense to me.

Why should the country I'm in and the language that I'm currently speaking determine if I want periods or commas as thousands separators? Or if I want dates to be presented in one of dozens of insane formats instead of RFC 3339/ISO 8061?

Because that's what you grew up, learned, and have used forever? Let's ask it another way - why would the country you are in and the language you are currently speaking NOT decide those things (which are defined by said languages.)

What are we doing right now to each other besides banging out some symbols that have a shared meaning?

>why would the country you are in and the language you are currently speaking NOT decide those things (which are defined by said languages.)

Because these have no relation to how a particular piece of content should be formatted. The fundamental mistake made by many of these legacy APIs with respect to localization is the assumption that the locale should be determined based on some property of the user, which is reflected in some OS or application-wide setting that applies to all content. This only works as a rough approximation in the small minority of cases where the user is a member of a cultural sphere / bubble where exposure to multiple languages is a rare exception, like the united states. In the rest of the world, it is an everyday occurrance for multiple languages to exist side by side, including within individual web pages, documents, spreadsheets, and other content, which is why the only reasonable solution is the one where a locale is a property of a piece of content in its most atomic form, not of the user.

I grew up with dd/mm/yy, comma as decimal separators, dot as thousands separator, but:

1. The fact that I grew up with those formats holds little weight for me. At this point, most of the literature I read and content I consume doesn't come from my own country. And why should it? when I have access to books, movies, websites from all over the world, and my country makes up for a tiny fraction of that. With the massification of international remote work this tendency will only increase.

2. Even when working with people from my own country, I can agree with them to use formats different from the ones we grew up with. In fact, everytime we type a floating point literal in any programming language, we do it without following the rules we grew up with.

3. In the 21st century, in the context of globalization, massification of the Internet and widespread access to computing, to keep doing these things differently by country/language makes no sense, specially when there are already good international standards that we can follow. It's not that hard to learn, either. We even accept English as the de facto language of programming and software development, and that's a full natural language that we have to spend years learning.

4. In my opinion, yyyy-mm-dd and dot as decimal separator are simply better for practical reasons. However, if we collectively decide that other formats are the "international standard" then I'd follow whatever rule we decide, as long as it's not too bad.

Can you imagine if instead of using SI units, each country used their own special units of measurement? We are definitely better off with SI, standarization makes everything so much easier. I can talk to someone from Japan about meters and kilograms and they will understand without any ambiguity.

The trouble with this is it essentially reduces all differences to "biggest player wins". Because it almost never matters what symbol or representation you use for these things, your personal preference is largely based on what you grew up with.

I could resign myself to Americans* being allowed to pick the global date format to save a few cycles on an operation, but frankly I don't want to. It's not that important to me, and not having aspects, albeit minor aspects, of my culture steamrolled in the name of pointless efficiently is at least a little bit important to me.

Differences create friction yes, but I'm alright with that. I would prefer a little friction to grey uniformity. You're free to think differently, but you're not the speaker for everybody else.

* I should clarify that most Americans don't seem to want this either, this isn't a jab, it's just that if we did standardise everything their choices would probably be the ones which won out.

I'm not OP but a related point is that locale-aware formatting can be offputting if your application is not actually translated into the language. At a previous job we delivered an English-language product that used locale-aware number formatting. Our German customers asked us specifically to use English number formatting, both because it was a technical programming product and because the application was entirely in English. German number formatting with English text was undesirable to them, and they didn't expect us to translate into German. They saw it as an American product and that carried into their expectations about formatting. We went into the code and explicitly specified the American English locale for all string formatting, and they were happy.

Out of sheer idiosyncrasy, I've developed the habit of formatting dates as dd.mm.yyyy rather than the more typical US mm/dd/yy (and long dates as dd mmm yyyy). Setting that as my date format in OS X for a long time caused the paper size to default to A4 even though everything else was set with US settings. Somewhere along the line, that particular "feature" was removed, although I couldn't say exactly when.

I speak English and live in the US.

So dates in the US are not RFC 3339, they're MM/DD/YYYY drunk-endian. And units are all US Customary, not SI. If you want something else, you need a locale other than en.US, and if you want other things (like spell check) to match US conventions you end up needing to create a custom locale.

Having locales by itself makes sense, you after all want to map real world information into the digital realm and for that to work you have to stick with whatever real world conventions are already established. Keep in mind that computers are often just used as better typewriters and a lot of information exchange still happens on paper. Moving from paper to fully digital is a process that takes decades, so you can't just decide that digital uses '.' instead of ',' as all your printouts will be wrong.

The part that doesn't make sense in C and other languages is that locales are forced on you as global state. And not just by default, they don't even provide locale-free alternatives, you have to modify the global state and reset it after every use, which is cumbersome, slow and error prone. C++ added the locale-free `std::to_chars()`, so things are slightly improving at least, but it's still an ugly and largely unnecessary mess.

I find this annoying as well because none of the premade locales match my preference. Why can't I set my prefences, just load the defaults from a local.

Maybe I'll have to write my own locale file. That doesn't sound fun.

Ditto this. One person, one device, one language, one locale is a terrible assumption.

My Android phone is set to a language that doesn't match the country I'm in, and it hyphenates all the local phone numbers wrong. This is beyond stupid. The phone knows which country the numbers belong to. I would expect it to format each number according to the conventions of the country they belong to, not according to the language I've selected. I'm not doing anything fancy like RTL, either.

Can you output Eastern-Arabic numerals (e.g. ٠١٢٣٤٥٦٧٨٩١٠) with sprintf? My gf sometimes uses them and its confusing as fuck for me. Especially as the numbers are then written RtL.

I believe the numbers are the one thing in Arabic that are NOT written RTL.

They are written RTL, it's simply that in Arabic numbers are written little endian.

In case this comes off as sarcastic, it is not intended so.

In fact, the fact that arabic is written rtl is why our numbers are big endian. Because the European number system is based on the arabic system.

I don’t think that is true.

Europe got decimal digits with their own characters via the arabs, true, but we use shapes derived from the original source, the Devanagari digits, not the shapes the Arabs developed due to their writing system/technology. South Asian languages other than Urdu are LTR.

You can see the parallels here (clipped from Wikipedia):

० 0 १ 1 २ 2 ३ 3 ४ 4 ५ 5 ६ 6 ७ 7 ८ 8 ९ 9

Ah, I get your point. They are _pronounced_ little endian, and consequently written that way RTL, which works out to big endian LTR, of course.

locale stuff is even in strtod, so you can read numbers like 3,14159 in locales where comma is the decimal point.