Hacker News Reader

Comment by natemcintosh | Hacker News Reader

"The fastest and safest AV1 encoder" but it's also 62.1% assembly? Is the assembly as safe as rust? I'm confused

The assembly is not safe but it is used with safe wrappers. Likewise, there are some custom data structures (specifically for tiling) that are internally implemented with unsafe code (just like libstd's data structures). Fortunately the assembly bits have well defined inputs and outputs so they are testable on their own. I would love for comparably fast SIMD to be written entirely in safe Rust, but that is not possible yet.

I don't know if its the case with rav1e but in many video codecs most of the assembly is just a flat unrolled circuit with zero or minimal control flow, minimal address calculation-- essentially just a static circuit that you might otherwise implement as a fixed block of logic in an asic. As a result they aren't too much of an unsafety vector.

This is also often true for cryptographic software, particularly fixed-parameter optimized code.

The performance gain usually comes from some mixture of good scheduling or register allocation that the compiler fails at and the use of instructions that the compiler won't (reliably) emit.

For these kinds of straight-line codes, I haven't found mechanically validating them to be significantly more complicated than verifying C code... and as C code these functions tend to be the lowest risk type.

So, without any direct experience, I'd expect there to be more risk in rav1e from the unsafe-rust than the asm.

That said, already the 'attack surface' of an encoder is pretty small. I'd personally expect rav1e's gain from rust's safety would be less security and more in avoiding wasting time on blind alleys caused by memory corruption in the codebase. I've seen more than one poor design decision made in a multimedia codec which was ultimately due to a bug that a better language might have prevented.

I've not checked the actual numbers but I think in terms of line count, that assembly may be the same function done multiple times for different (sub-)architectures for different generations of Arm and Intel chips that have different vector instructions.

Yes, there are currently 4 copies of the assembly - for AVX2, SSSE3, 64-bit NEON, and 32-bit NEON. There will likely be more as future architectures get added.

It depends on the amount of third-party crates, unsafe{} and assembly being used to confirm this claim. It's possible that out of the other alternative AV1 encoders (not in Rust), this claim may or may not be true, but can be more true if they said "The fastest and safest AV1 encoder in Rust".

Most rust projects have these sort of safety claims to attract companies, devs to promote it in meetups or tech conferences. Maybe they should audit this encoder to see if it lives up to these claims.

Rust can compile to many different targets. The assembly is for x86 platforms, but this project also supports arm64 and wasm targets. If security is of utmost concern, I would actually suggest running the WASM build in a sandboxed JIT like lucet.

What blows my mind is that there's a Rust encoder, but no Rust decoder. There are many more orders of magnitude of people who will use a decoder than an encoder. One would think that the safe decoder would have come first.

Most people will use the decoder in their browser, while the encoding environment will be much more diverse.

Decoders are notoriously harder to support than encoders due to the various internal formats and encoding/decoding options. Until it supports all the different formats (a lot of effort), it will only be able to play back a few videos.

By focusing on an encoder (at least initially) you can just support a subset of features that work well for you. Then focus on making a decoder that can at least play back videos from your encoder.

More people will find it useful to have an encoder that works all the time, rather than a decoder that only works on a subset of videos.

I've added assembly to the title for now. If someone can suggest a better (more accurate and neutral) title, we can change it again.

The assembly appears to be optional.

Is it still the safest when it's the fastest, or do you have to pick?

Despite the tagline, I don't think its ever actually been the fastest AV1 encoder, it's quite possible its always the safest, even with the asm in place and even more so when using only rust.

Is Rust really safer? It's memory safe, but you can still have all the other fuck ups and vulnerabilities that other apps have.

How many times are we going to have this discussion?

> ~70% of the vulnerabilities Microsoft assigns a CVE each year continue to be memory safety issues

https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro...

There are similar reports from other organizations.

The logical structure of your statement is: Are five apples and two oranges really more food than two oranges? You have apples, but you still only have two oranges.

> safer

For any reasonable definition of "safer", yes. 100% safe? no.

Rust? Not really.

https://github.com/xiph/rav1e/search?q=unsafe&unscoped_q=uns...

Reply on news.ycombinator.com