Hacker News Reader

Comment by pabs3 | Hacker News Reader

I wonder if the authors plan to get this merged into commonly used implementations of UTF-8 validators so that folks can benefit from their work.

Anyone have any ideas about which open source codebases UTF-8 validators exist in?

View on HN

> Anyone have any ideas about which open source codebases UTF-8 validators exist in?

Rust's std library -- the canonical way to read a text file to a string is to (implicitly) use `std::str::from_utf8`. If I remember correctly, the current implementation doesn't use SIMD specificially but will of course contain vectored instructions if the compiler can select them on the platform you target.

I did a comparison with another SIMD based implementation last year. Maybe it's time to update it: https://github.com/killercup/simd-utf8-check

Eh, these types of algorithms aren't the ones that can really benefit from autovectorization of the scalar algorithm.

The whole algorithm basically has to be redesigned from scratch to introduce vectorization: something far beyond the capabilities of the compiler.

I don't have any particular plans for that, beyond getting a bit of publicity for it here :). Or really any plans at all, besides adding AVX-512 support. It was just something I hacked together in a couple days as I began thinking about adding UTF-8 support to the text editor I've been writing.

That said, I'm happy to help anyone get it integrated into their codebase if they need it. There's generally other concerns in large projects, though, like portability, that I don't care as much about.

There's not as big a niche as you might think. A lot of the software that needs valid UTF-8 don't validate upfront, but do it as part of a parsing process. An example would be something like an XML or JSON parser.

For more mundane uses like opening a big text file, you often want to tolerate invalid bytes and show a replacement character. You could use the technique, but you'd have to modify it to work for that usage.

In case anyone sees this: I initially agreed with you, thinking that pure UTF-8 validation is a bit more of a niche than might be expected. But if I'm not mistaken, I think at least for JSON validation, the UTF-8 validation can happen completely independently of the parsing. I don't think any control characters (braces, commas, quotes, escapes, etc) will change the validity of the UTF, or vice versa. So it might be beneficial (both for speed and security, as the sibling comment notes) to validate chunks of input as UTF-8, then pass them off to the parser, which now doesn't need to deal with validation.

Maybe this could be applied to XML, but that's quite a behemoth of a standard--I really wouldn't be surprised if there was a way to switch encodings mid-stream. I have no idea though...

Sure, you can validate and then parse the JSON, but parsing it is also effectively validating it as UTF-8, except for the string values in the JSON, so it's heavily going to be duplicated effort. And unfortunately, the strings can contain things like invalid surrogate pairs by using unicode escapes, so you may want to validate after escaping.

You can't normally blindly parse XML as UTF-8. The encoding has rules for detecting the character set.

[re: upfront parsing - a PSA about validating input]

> A lot of the software ... don't validate upfront

They almost always should be validating all input up front. Deferring validation tends to become many different pieces of code all informally parsing fragments of the input that are needed locally. Without upfront validation of the complete unit of input, the resulting fragmented parsers are just a weird machine waiting to be programmed by a malicious attacker.

For a much better explanation, I strongly recommend Meredith and Sergey's 28c3 talk[1] about The Science of Insecurity.

> you often want to tolerate invalid bytes and show a replacement character

While this validator wouldn't be useful in that situation, it is still important to validate the input upfront. When showing replacement characters, the "invalid bytes" that will be supported with a replacement character should be formally defined and added to the validation grammar, because they are no longer "invalid", but instead will be handled as a special case.

Postel's robustness principle shouldn't be used as an excuse to skip validation. Being "liberal in what you accept" should still be well-defined and validated.

[1] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit...

You trimmed my quote in a way that completly changes the meaning, so I'm not even going to bother responding to that.

How unicode replacement characters should work is already well defined. Just look it up.

Reply on news.ycombinator.com