Hacker News Reader

Comment by ormax3 | Hacker News Reader

Like explained in https://utf8everywhere.org/#windows , you can write simple wrapper functions `narrow`/`widen` used when you are about to call window api functions.

    ::SetWindowTextW(widen(someStdString).c_str());

Implementation is straightforward, relying on `WideCharToMultiByte`/`MultiByteToWideChar` to do the conversion:

https://github.com/neacsum/utf8/blob/master/src/utf8.cpp

View on HN

I have good experiences with this approach. It isolates the weirdness of windows and let's the rest of your code do things idiomatically

Microsoft added a setting in Windows 10 to switch the code page over to utf-8 and then in Windows 11 they made it on by default. Individual applications can turn it on for themselves, so they don't need to rely on the system setting being checked.

I haven't tried it yet but with that you can just use the -A variants of the winapi with utf-8 strings. No conversion necessary.

Do you have any references about it being enabled by default in Windows 11? I've seen conflicting reports and often seems to vary depending on the system locale whether it gets enabled or disabled by default.

You are missing OP's point - this still costs you 2 extra calls.

If this cost really matters (and practically speaking it never does), then, as the other commenter said, the correct solution is to just use OS-native encoding for all file system paths and names used by the program, hidden behind an abstraction layer if needs be. UTF16 for Windows, UTF8 elsewhere.

The above manifesto makes the argument to use UTF-8 *everywhere*, even on windows where the internal representation is not native utf8.

The conversion overhead is really negligible: https://utf8everywhere.org/#faq.cvt.perf

(note: the two api calls per conversion is because how those specific functions work, first call to get the size to allocate, second to do the actual conversion, but you can always use another library in the implementation for the utf8<->utf16 conversion that might be more optimized than those windows api functions)

Especially negligible versus the trip to the file system you are setting up for.

Not all API calls are for filesystem access.

Sure, but basically everything having to do with file paths on Windows, the topic here, relates to the file system.

And AVX-512 can help:

https://lemire.me/blog/2023/09/13/transcoding-unicode-string...

I thought the OP's point is there are too many considerations when doing this?

Someone is suggesting a way of making it less tedious, and your response is "performance?!" even though in both scenarios you're running the same code and it is likely the compiler in release would remove the intermediary.

> your response is "performance?!"

No, that's not what I said.

"2 extra calls" is a weird metric here. Some calls are vastly more expensive than others. Syscalls come with a significant cost, encoding conversion of short strings (esp. filenames) does not. Hiding just the syscalls behind an abstraction layer is vastly simpler than doing that and additionally hiding the string representation, so "UTF-8 everywhere" is IMHO the right solution.

Reply on news.ycombinator.com