I would call that Windows pain at this point, not locales.
In the Linux/Postgres world, everything is UTF-8. Which is the default for all internet protocols. Do that and the pain is gone.
Of course Windows doesn't do that...
In the Linux/Postgres world, everything is UTF-8. Which is the default for all internet protocols. Do that and the pain is gone.
Of course Windows doesn't do that...
Windows, the OS, is UTF-16 (or UCS-2 - I forget the details between the two), SQL Server has, for historical reasons, defaulted to CP1252, probably for compatibility with Office components.
But, it's not really a Windows problem, per se, because you have to deal with this issue if you deal with data originated from numerous Windows apps, even if on Linux. Yeah, you can insert a byte order mark (BOM) to indicate UTF-8, but most tools expecting UTF-8 actually dont check for the BOM and blow up in interesting ways if present. Ive seen this far too many times. Enough that anytime I see an encoding error from the likes of Python or Ruby, its an instant recognition (I do a lot ETL work from a number of vendors, so I see a lot of different files "types").