Wait, how is that safe? Aren't Rust strings UTF-8? Even if you search for a ASCII character, couldn't it conflict with the second byte of a different unicode codepoint?
https://github.com/rust-lang/rust/blob/5cf55165fae5c8538db5c...
This does mean you have some somewhat pathological cases, but they're relatively rare because most strings come from only a couple unicode blocks, so all bytes other than the first will be from a really small set. A further optimization could be to fall back to the original find code if we realize we're hitting a pathological case.
(Also, FWIW, searching for an ASCII char can never cause this because of how UTF8's streaming property -- https://en.wikipedia.org/wiki/UTF-8#Description -- but given that we only search for the last byte, searching for a multibyte codepoint can lead to false positives with a shared last byte. We handle that.)
TL;DR, it's not a naive straight `memchr`. `memchr` will generate false positives, then they use knowledge of UTF-8 to weed out the false positives.
The pathological case is worse, but the overwhelming majority of cases are better.
Or, searching for multibyte character ABB (like U+A041 YI SYLLABLE PA) and finding the first "B" instead of the second "B".
Sadly there's no memchr for consecutive sequences of characters. memchr2/memchr3 let you search for multiple needles in the haystack, not a bigger needle.
Or maybe memmem does something else clever for shorter needles. Dunno. Best to benchmark it. But it's not an obvious win a priori.
That would be pretty neat.
You can memchr for the last byte (either 0xxxxxxx for only byte or 10xxxxxx), then in the latter case go backwards to the MSB to see if the entire codepoint matches.