Hacker News Reader

Comment by inigyou | Hacker News Reader

TLDR: they implemented case folding with a lot more SIMD via autovectorization.

> almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ)

Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered.

View on HN

+1 great idea. I get the impression that some of my sibling comments thought that you meant fold all code points in the opposite direction, but it it's clear you mean change the fold direction for only those two code points.

It does not look like the author posted this here since it was posted on the blog last week. So would probably be a better idea to suggest / recommend this at https://github.com/github/rust-gems/tree/main/crates/casefol...

Don't see why I should go out of my way to improve Microsoft shareholder value.

I wonder if making the index uppercase is strictly better in this sense, or if both upper and lowercase have chars that take more bytes.

> The search index won't only consist of lowercase characters any more

This isn't the case anyway. Unicode case-folding has a few lowercase-to-uppercase mappings, e.g. Cherokee

Curious if his isn’t some Over-optimization since how often do those characters ever come up.

If you can kill-off the extra buffer allocation code path that's got to be a win.

This is a nice follow up to the other SIMD article that was posted here a week or so ago hah.

Reply on news.ycombinator.com