Hacker News Reader

Comment by Ygg2 | Hacker News Reader

> The usability is not great, too opinionated. For example, they have decided their strings are guaranteed to contain valid UTF-8.

Wait, why is that a problem? Using UTF-8 only is the closest to programmer's ideal. Basically, you turn the UTF-8 validator for Rust's str into a no-op.

> Yes. Even very simple data structures like std::vec::Vec require unsafe code

I guess this is the sort of glass half-full vs glass half-empty. In C++ the std::vector::pop_back will forever cause undefined behavior. It's like putting razors on your screwdriver's handle, sure you can avoid them, but why not do the sane thing...

View on HN

> Wait, why is that a problem? Using UTF-8 only is the closest to programmer's ideal.

I have a lot of archived data with various unknown encodings. APIs that blindly try to force an encoding when all I have is "bytes" tend to be a major pain and I usually end up bruteforcing my way around them with things like base64.

If that's the case the winning move is get the bytes, and convert them to UTF8 or just process it as bytes. &str is a wrapper around &[u8] with strict UTF8 guarantees.

Modern systems should be able to convert various encodings at GiB/s into UTF8.

> If that's the case the winning move is get the bytes, and convert them to UTF8

That would require knowing the original encoding.

> or just process it as bytes

As long as the APIs you have to use take bytes and not u8 strings.

> Modern systems should be able to convert various encodings at GiB/s into UTF8.

They might even guess the correct source encoding a quarter of the time and it will break any legacy application that still has to interact with the data. I guess that would be a win-win situation for Rust.

>That would require knowing the original encoding.

Or just use a library that can detect the encoding, and spit out utf-8. There's several of those.

Yes, you can try to automatically guess the wrong encoding based on statistics that only work some of the time when given large amounts of free flowing text.

> That would require knowing the original encoding.

If you don't know that's one more reason to get bytes and try to figure out encoding. Usually using lib like encodings.rs or WTF8.rs

> As long as the APIs you have to use take bytes and not utf8 strings.

You can convert one into the other, albeit converting to str requires a check.

That's why byte strings exist. You don't know the encoding? Well, it's a binary chunk and nothing else.

Reply on news.ycombinator.com