Hacker News Reader

Comment by wzdd | Hacker News Reader

Alternative sane approach:

- Forget about long paths unless you really need to care. You need to care if you're accepting a file path from another app, for example. You don't need to care if you're using your own files in your install directory; too much of Windows doesn't support long files (including explorer, as mentioned, not to mention third party apps), so it's very unlikely that you'll need to deal with them if you're just handling your own data.

- Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. It's unsurprising that you need to call the conversion function twice in order to find out how much memory to allocate, that's just how that works.

Don't change random registry settings or code pages on the user's computer, that's mad.

View on HN

> Don't change random registry settings or code pages on the user's computer, that's mad.

A million times this.

Multiple Valve games set the microphone volume in Windows to 100% whenever you launch the game, which, for my setup, makes me sound unbearably loud. Unless your app has a valid reason to change my settings, like if I've set it up to do that on a schedule or with some condition, sure, otherwise don't mess with my settings.

Story time

When the USB audio device class was created [1], it had support for AVR-like audio controls (volume, balance, tone etc.), and, like would be reasonable, made all that work with decibels. But, as far as I know, that part of the standard got ignored by basically everyone, including some of whom wrote it (Microsoft), by presenting e.g. the on-device volume control as a 0-100 scale, like Windows and ALSA still do. Except... some hardware devices actually implemented the spec, including the PCM29xx family of audio codecs. These, and clones of them, have been used in quite a few USB audio cards and some very low-end audio interface over the decades. And for those, "100" in Windows means something like "+30 dBfs" [2] - so you'll get extreme distortion with basically any signal.

[1] 1.0 is actually impressively complex and basically models the entire stack of audio devices someone could have possibly had in their TV cabinet in the 1995, including stuff like dolby and surround decoders.

[2] This is defined around 5.2.2.2.3 and/or 5.2.2.4.3.2 in https://www.usb.org/sites/default/files/audio10.pdf

> Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app.

But that's not compatible with other OSes, so you can't write native multiplatform C++ code just because windows keeps insisting on their worst-of-both-worlds UTF-16

(worst of both worlds because it's not single value per unicode character like UTF-32, and not ASCII compatible like UTF-8)

Store it as std::path

Have a few functions taking UTF-8 paths for the filesystem operations you need, have separate versions for Windows and for other systems, and enable one or the other version with the preprocessor?

linux uses utf8 for display of filenames but, the paths themselves allow non-utf8 byte sequences.

linux don't display them, the shell (emulator) do. Linux just send the bytes back to userland and let shell interpret them with a proper format to human. And even then. Tons of distro default the global LANG to c for some reason. So utf8 display isn't even working by default.

Even better: Each user can have his own locale and charset, and may even change that per program/shell/session. One may save filenames as UTF-8, one as ASCII, one as ISO8859-13, one as EBCDIC.

However, the common denominator nowadays is UTF-8, which has been a blessing overall getting rid of most of the aforementioned mess for international multi-user systems. And there is the C.UTF-8 locale which is slowly gaining traction.

It's an oversimplification to say Linux uses UTF-8 for display. Linux just stores bags of bytes and leaves interpretation to userspace. You could store paths in ISO-8859-1 if you wanted. The only special bytes are '\0' and '/'.

Not only could you, this actually happens in practice. Not necessarily ISO-8859-1, but specifically SHIFT-JIS, a Japanese encoding that you will run into if you run old Japanese software. To make things even worse, SHIFT-JIS is almost entirely incompatible with any form of UTF based encoding, and depending on the attempted normalisation you can quickly end up with paths that have been messed up multiple times in a row.

I forgot what Japanese emulator I tried to run when I found all of this out, ut sufficed to say I didn't enjoy the experience.

I buy digital Japanese Doujin music on sites like booth.pm, and their provided zip files extracts "beautifully" on Linux if you simply `unzip` them.

Don't you mean `unzip -O shift-jis` them?

Lots of Japanese products are switching or have switched to UTF-8, so I have no doubt that modern ZIP files will extract without a problem.

Except that Linux does support several filesystems that do claim to store the filenames in a specific encoding and therefore the kernel must do conversion. Mostly Windows FSes, but nowadays case-insensitive ext4 also applies.

These are exceptions, not the norm. The VFS layer does not care.

This is how 99% of cross-platform C++ works. Very few C++ is actually "cross platform" in that it compiles the same functions for different platforms

Not to be that guy, but utf-32 is fixed length code point, not fixed length character.

Characters can consist of multiple code points.

I agree that utf-8 is preferable to utf-16 though in most cases where the root script is English/Latin.

Makes me wonder how bad utf-8 in off-latin environments really is: what's the fraction of strings that is "language content", and how much is colons, quotes, decimal numbers and the like? What fraction of coding cultures embraces home locale script for identifiers, and how many continue on the trajectory set in motion by pre-unicode programming languages? My guess would be that utf-8 is still quite bad, but I wouldn't be entirely surprised if actual numbers turned out considerably less bad than one might expect. And how much of the "utf-8 tax" remains if the actual format of most data at rest and in transit isn't json but gzipped json?

Experiments I recall from a while back from browsers and JDK indicated the vast majority of memory is consumed by latin1 strings (even for apps in Asia). Of course, those experiments I recall are something like 20 years old and likely biased (websites were smaller, content was still latin1 dominated). It would be interesting to know the results from Chinese apps and websites today. For websites, the markup used the most space and that stuff remains latin1. For apps I don’t know. But also, text size in memory is typically a joke because the expensive bit is all the multimedia that accompanies text these days. Remember, 100mib is 25million characters even in utf32. That’s 1 hour or so of high quality audio or a few seconds of video. In terms of compression, the smaller initial file will typically compress better so not sure what you mean by “utf-8” tax. UTF8 is not slower to process than utf32 as far as I’m aware except for one small corner case of random codepoint access (but then in UTF text processing, you really shouldn’t be doing that I think but I’m not a utf expert ).

> Characters can consist of multiple code points.

I enjoy being "that guy", and so, to be unfathomably pedantic, I will point out that the Unicode standard actually uses the word "character" to refer to _encoded characters_, i.e. the mapping from a code point to an abstract character [0]. So 1 code point = 1 character for Unicode. Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway.

[0]: See section 3.4 of https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda...

Your position is obviously wrong both with regard to the Unicode standard and with regard to the meaning of the word "character."

Unicode does not define "character" because that term is far too imprecise for a deterministic standard. And the Unicode standard is extremely clear that an abstract character may be represented by multiple code points.

"A single abstract character may also be represented by a sequence of code points—for example, latin capital letter g with acute may be represented by the sequence <U+0047 latin capital letter g, U+0301 combining acute accent>, rather than being mapped to a single code point."

> Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway.

Human language and writing are the defining inventions of the known universe. Unicode just happens to be one way of representing them. Referring to them as "silly and made up" in comparison to Unicode is nothing short of extreme mental illness.

Unicode Standard section 3.4:

> Unless specified otherwise for clarity, in the text of the Unicode Standard the term character alone designates an encoded character.

Reading is free. I even gave you the section number.

> A single abstract character may also be represented by a sequence of code points

An "abstract character" is not the same thing as a character. The whole point of my comment was that the general word "character" is a general class of definitions but means nothing by itself. Much like the word "number" - you will notice that mathematicians define many kinds of numbers, but never the word itself.

> Human language and writing are the defining inventions of the known universe. Unicode just happens to be one way of representing them. Referring to them as "silly and made up" in comparison to Unicode is nothing short of extreme mental illness.

All words are made up. I'm sorry you had to find out this way.

> All words are made up. I'm sorry you had to find out this way.

It's so, so interesting that you believe this to be true.

Do you believe all things made by humans are made up? Is HN "made up"? Are other people "made up"?

If you're calling Win32 functions it's already not compatible with other OSes... If you want to use cross platform code with files, use the C or C++ standard library's file handling functions.

C++ also supports wstring, but its implementation is dependent on the platform for god knows what reason.

utf-16 is also supported in C11:

https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Chan...

> Improved Unicode support based on the C Unicode Technical Report ISO/IEC TR 19769:2004 (char16_t and char32_t types for storing UTF-16/UTF-32 encoded data, including conversion functions in <uchar.h> and the corresponding u and U string literal prefixes, as well as the u8 prefix for UTF-8 encoded literals).

You can use uchar.h for converting instead of the Win32 functions.

Here's a helper I wrote to wrap these string types in std::string and std::wstring:

    bool mbStrToWideChar(string const& in, wstring& out)
    {
        bool status = true;
        i32 bufferSize = MultiByteToWideChar(
            CP_ACP, 0, in.c_str(), -1, nullptr, 0
        );

        WCHAR* buffer = new WCHAR[bufferSize];
        i32 conversionResult = MultiByteToWideChar(
            CP_ACP, 0,
            in.c_str(), -1,
            buffer, bufferSize
        );

        if (!conversionResult)
        {
            cerr << "Failed to convert ANSI string [" << in << "] to Wide string!" << endl;
            status = false;
            goto Delete;
        }

        out = wstring(buffer);

    Delete:
        if (buffer)
        {
            delete[] buffer;
        }
        return status;
    }

and

    bool wideStrToMbStr(wstring const& in, string& out)
    {
        // Get buffer size
        i32 bufferSize = WideCharToMultiByte(
            CP_UTF8, 0,
            in.c_str(), (i32)in.size(),
            nullptr, 0,
            nullptr, nullptr
        );

        // Set outstring to new buffer size
        out = string(bufferSize, '\0');

        //Convert
        i32 res = WideCharToMultiByte(
            CP_UTF8, 0,
            in.c_str(), (i32)in.size(),
            &out[0], (i32)out.size(),
            nullptr, nullptr
        );
        if (!res)
        {
            wcerr << L"Failed to convert Wide string [" << in << L"] to ANSI string!" << endl;
            return false;
        }
        return true;
    }

Our native multiplatform c++ code uses wstring (win, mac, android, wasm). It’s not the worst option. AFAIK There is no 100% method to have single codebase tackle the differences in platforms. My recipe for keeping sane is to have single implementation for storage (wstring), and then very explicit points of converting paths from ‘platform’ to program internal.

Platform differences unfortunately go way deeper than just using UTF-16. Lots of gnarly details. No platform is worse than another - they just have different kinds of good and bad in different places.

Speaking as someone who did what GP suggested, it can be cross-platform; I just abstracted the string to path and path to string conversions, as well as any path operations that are different across platforms.

The first thing to realize to make it work is that paths are not strings, so you actually don't want to treat them like they are.

How do you write code that you can't write one function that opens a file?

If you need to consider slightly older Windows versions, it's WTF-16, not UTF-16.

Isn't it UCS-2 rather than any form of UTF-16?

Not after Windows 2000.

There’s no such thing as WTF-16—or if you want to use that label for “potentially-ill-formed UTF-16”, then everything that says it does UTF-16 is actually WTF-16, because no one validates UTF-16.

Yes and no - some programs or systems of programs regularly transform between UTF-8 and UTF-16. A file path in Windows can be "valid" (or whatever we wanna call it) and still not convert cleanly to UTF-8.

Edit to clarify: many systems which deal with non-path UTF-16 strings, nowadays are so intertwined with the UTF-8 world, that they get the validation "for free" so to speak, here and there, wherever they interact with the rest of the world.

Paths though, remain a common little corner case, working fine until it suddenly doesn't, for some weird combination.

That’s just potentially-ill-formed UTF-16, which is what all UTF-16 is in practice. The novel thing about UTF-8 environments is that they (or most of ’em, anyway) finally actually validate stuff.

This is exactly the approach I took in my own cross-platform "standard" library. It works.

In particular, paths must be a different type than strings.

As a bonus: it becomes easier to implement niceties like path concatenation and realpath.

You can also just over-allocate to avoid having to call it twice, or do some really basic heuristics to fast path the call (it's very specific UTF-8 sequences that need more than 2 bytes in UTF-16).

Reply on news.ycombinator.com