It's not sufficient to use code points right? Some characters, for instance your example of the black thumbs up emoji, are grapheme clusters [1] composed of multiple code points. I think you have to iterate in grapheme clusters and convert that back to an offset in the original underlying encoding.

If you just rely on code points you risk splitting up a grapheme cluster into (in your example) two graphemes, one in each sub-string, the left representing "black" and the right representing "thumbs up." Further, you actually need to utilize one of the unicode normalization forms to perform meaningful operations like comparison or sorting.

This is one thing Rust's string API gets right, allowing you to iterate over a string as UTF-8 bytes in constant time -- and, by walking, your choice of codepoints and (currently unstable, or in the unicode-segmentation crate) grapheme clusters. Even that though is a partial solution. [2]

Definitely a tough problem!

[1] https://mathias.gaunard.com/unicode/doc/html/unicode/introdu...

[2] https://internals.rust-lang.org/t/support-for-grapheme-clust...