Unfortunately, Java and Windows both adopted 16-bit Unicode as their internal representation. So did the Python of the time. This led to a hokey scheme where two 16-bit Unicode characters are used to represent one longer character. Current language status is that Go and Rust are all 100% Unicode, Java still has the 16-bit hack, Python 3 is 100% Unicode but Python 2.7 has 16-bit or 32-bit Unicode builds, and C and C++ still require a lot of explicit work by the application programmer. PHP has some Unicode support, but doesn't really understand string encodings.
On the web, HTML is now almost entirely UTF-8. At least that's been straightened out. Mostly. There's still too much Latin-1 (ISO8859-1) around. Browsers still mostly support the old encodings, although Microsoft Edge apparently doesn't do so very well. Note that HTTP is not Unicode-aware; URLs, domain names, and HTTP data often have strings with other encodings.
The major databases understand Unicode, but it may be necessary to specify an encoding. MySQL went through a 24-bit Unicode period ("utf-8" is 24-bit) but now supports 32-bit Unicode. This requires specifying "utf8mb4" as the encoding. For new work, use that for everything. Postgres uses "utf-8" for 32-bit Unicode. MongoDB has no idea what character set it's using; it only knows raw bytes, and the application has to handle encoding and decoding.
The current big question is whether programs should use UTF-8 encoding internally, and whether the encoding should be visible to the programmer. Rust uses UTF-8 internally, it's totally visible to the programmer, and you have to use special libraries to iterate over a string properly. Go tries to hide it a little; if you iterate over a Go string using "range", you get 32-bit "runes". Python hides the internal representation completely; if you iterate over a Python string, you get strings which contain one code point, and strings are subscriptable by code point.
[1] http://unicode.org/emoji/charts-beta/full-emoji-list.html
It's often easier to use that for-loop with a break to look at the first character of such a string, as in...
package main
import "fmt"
func main(){
str:= "我是一个人"
for _, r:= range str {
fmt.Printf("%c\n", r)
break
}
}
Doing a str[0] prints the first utf8 byte of the string ("æ") instead of the whole character. The name of the official function is difficult to remember, and you have to put an import at the top... import "unicode/utf8"
...
r, _:= utf8.DecodeRuneInString(str)What's the reason for the quotes around utf-8? To my knowledge it's bog standard utf-8, with the exception that NULL is disallowed in many types.