Hacker News Reader

Comment by brundolf | Hacker News Reader

Rust's string type doesn't have the mentioned problems: https://play.rust-lang.org/?version=stable&mode=debug&editio...

Though I would bet this mainly has to do with being a language born in the age of the web. It seems like something that would be very hard to back-port (as opposed to designing for it from the beginning), but proper handling of Unicode is practically table-stakes for a new language in today's world. I would guess (I don't know) that Swift, Go, etc also do a good job with this stuff.

View on HN

It does for multi-codepoint characters (such as the rainbow flag): https://play.rust-lang.org/?version=stable&mode=debug&editio...

Elixir gets it right: https://replit.com/@natanbc/reverse

Yeah, another commenter mentioned grapheme-clusters, though apparently there's a readily available crate that adds support: https://stackoverflow.com/questions/58770462/how-to-iterate-...

According to this they added it to the standard library at one point but then broke it back out because the lookup tables were too large

Here is a working rust version https://play.rust-lang.org/?version=stable&mode=debug&editio...

.chars() iterates over unicode scalar values, not actual grapheme clusters, so this code would also be incorrect in many circumstances. Ultimately it is incumbent on the developer to understand what their actual unicode-processing requirements are.

I'm not familiar with what exactly falls into each category, though apparently there's an (official?) crate which extends the core str type to support those as well: https://stackoverflow.com/questions/58770462/how-to-iterate-...

this isn't quite set up the way the article was mentioning. Here's with it set up correctly:

https://play.rust-lang.org/?version=stable&mode=debug&editio...

It looks like it still misses some of the cases (specifically with decomposed vs precomposed).

As a side note, it looks like the playground editor also doesn't correctly render some of the lengths (the cursor is 1 character off on the decomposed case)

Huh, I'd never heard of decomposed/precomposed before and I kind of skimmed over them in the original article. Looking at your example, I still can't figure out what the difference is between the two (I guess the Playground editor itself just can't represent the difference?). I'll have to go read up on that concept.

Edit: Having read about them, I guess I assumed that if my keyboard inputs an ë it's actually inputting those two underlying characters. Or to put it differently, I assumed there weren't two separate representations for the same character; that the "decomposed" and "precomposed" versions are one in the same, at the byte level. Learned something h̶o̶r̶r̶i̶f̶y̶i̶n̶g̶ new!

After a little bit of research, it looks like this is how you would get the expected outputs: https://play.rust-lang.org/?version=stable&mode=debug&editio...

What does "table stakes" mean? Never heard that before.

"Baseline expectation". I think it comes from poker or something

Table stakes are a required minimum bet in a game (often poker) - the meaning being if you can’t provide at least this you can’t play. So if a new language doesn’t support Unicode well it won’t get any adoption.

It's a gambling expression meaning the minimum [money] you need to even sit at the table. To be a really competitive player you need more.

Reply on news.ycombinator.com