Building better software with better tools: sanitizers vs. Valgrind | Hacker News Reader

Building better software with better tools: sanitizers vs. Valgrind

lemire.me

104 pointsingve21 comments

> Sanitizers are a game changer, they bring C++ closer to languages like Java and Rust.

This is a stretch, IMO. I suppose it is ever-so-slightly-closer.

> They do not bring the safety to production, since you probably do not want to use sanitizers in production or release

You're right, they're not intended for use in production -- until now! ARM dedicated some virtual address bits to tagging memory [1][2][3] for ARMv8 so that we can have sanitizers in hardware. Imagine: chrome on your smartphone being protected!

[1] https://developer.arm.com/docs/ddi0487/latest/arm-architectu...

[2] https://clang.llvm.org/docs/HardwareAssistedAddressSanitizer...

[3] https://arxiv.org/abs/1802.09517

> Sanitizers ... do not bring the safety to production, since you probably do not want to use sanitizers in production or release

My team runs ASAN in production. We run lots of servers, and we have load balancing that keeps CPU utilization equal across servers. ASAN binaries are slower (maybe 1/2 throughput / 2X "local" cpu-bound latency?) but not so much slower that running it on a small fraction of our servers means we can't meet our SLO. And it's been a huge help to us. Once in a while, we have some segfault that's pretty mysterious otherwise, as the stack trace where memory corruption is discovered can be totally unrelated to the stack trace where the problem was caused. If it happens on the ASAN servers, the root cause is instantly obvious.

TSAN is more problematic to run in production; it's a _lot_ slower.

Yes, thread-sanitized code is unbearably slow. Plus, you cannot use both ASAN and TSAN in the same build (but either can be combined with UBSAN).

The distinction between "sanitizers" and "Valgrind-like" drawn here is quite arbitrary. I think the core distinction being drawn is that "sanitizers" use compiler-inserted instrumentation and "Valgrind-like" tools use dynamic binary translation, though the author doesn't make this clear.

The conclusions drawn are questionable. The author states that "sanitizers" are fast and "Valgrind-like" tools are slow, but it actually depends on the tool. Some sanitizers are much faster than others. Some dynamic binary translation tools (e.g. DynamoRio's Dr Memory) are much faster than Valgrind. It wouldn't be hard to come up with examples where a compiler-based sanitizer is slower than a dynamic binary translation tool. It all depends on what the tool is actually checking.

Items 3 and 4 are also questionable. It's certainly possible to configure dynamic binary translation tools with customized function-specific behavior. Dynamic translation tools are generally even easier to use than sanitizers because you don't have to modify your build at all, you can just run tests under some kind of wrapper command.

He doesn't use the term "Valgrind-like". He only talks about Valgrind, which he calls a "great tool" (he's right!). But he is evangelizing sanitizers, in clang and gcc, which are relatively new. And, yes, they do work by adding instrumentation at compile-time, which allows them to be much faster than Valgrind (which does all its analysis at run-time). See: http://btorpey.github.io/blog/2014/03/27/using-clangs-addres...

Sanitizers are great tools, too. Maybe you can quibble about his item 3, but he's dead on with item 4. If you have tests, you add a couple compiler flags, and then you'll find memory problems automatically with your unit tests (without much slowdown).

I currently work in the DNA analysis industry. We have some pretty massive data sets and computational challenges. We use C++ for the heavy lifting. There's no way we'd be able to valgrind anything but toy input (valgrind's performance cost is prohibitive) -- and there's so many different possible inputs and code paths that it'd be effectively impossible to automatically generate all possible toy inputs. So we use sanitizers, as many as we can. They're a game changer in correctness validation.

I'd love for fuzzing to be a thing, but even that seems impractical on a large scale (please prove me wrong!) since it's a lot more work than just setting a compile target in cmake and ensuring that sanitizer flags are set for that compile target.

The right approach to start fuzzing with AFL, to me, is to recycle existing unit tests or integration tests. In fact if you already have tests that create a record (sorry, struct or class instance), inject them in your part of your application, you're almost. Read your input data from a file and 'there' you go... Paralellizing or distributing an AFL session on dozens of machines is a matter of rsyncing some files.

Sanitizers are also very useful there in C/C++ languages because they're mostly runtime checks. In Ada you had them already for a long time, and you can be very through with your checks (see for example validity checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.ht... , runtime stack size checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Check... , and the very very powerful Initialize_Scalars that has saved me many times https://www.adacore.com/papers/exposing-uninitialized-variab... ).

Shameless plug, but if anyone is interested in an introduction to fuzzing (in Ada) I wrote something some time ago that could be useful : https://blog.adacore.com/running-american-fuzzy-lop-on-your-... .

Fuzzing is fun !

I did a little bit of fuzzing and in general I agree with you about the right approach being to build off of existing unit/integration tests.

But in practical terms, that's still a lot more than "add this compiler flag" or "swap out this library".

Why "versus"? You use both, of course. And additionally, you use static analyzers like clang-tidy, cppcheck, PVS, you-name-it. And you're still gonna be miles away from Java and Rust w.r.t. safety. Undefined behavior is the bane of the C-language family.

Sanitizers and Valgrind are nice, but you need a comprehensive test suite to really benefit from them. They only detect problems that are triggered by your testsuite. This is a far cry from statically proving that your software is free from e.g. use-after-free mistakes.

No test suite, no problem. Just run in a production staging area or deploy to beta users.

That's better than nothing, but it would still be no proof that the are no issues, as uncommon behaviour/access patterns could take some time to appear (or will only show up after other services are introduced).

> or deploy to beta users.

And for God's sake: your users are NOT beta testers!

I mean, if you don’t have tests, write some. I know that sounds glib when confronted with a horrible old big ball of mud, but it’s probably the best place to start.

Yesterday a coworker had an issue where ASAN wasn't finding a std::vector out of bounds error. It turns out the code doing the deference was in libc++, which was not ASAN instrumented even though his program was. Valgrind found the error right away. They're all great tools with their own pros and cons. If you have the testing infrastructure, use all of them.

Note that valgrind can also detect uninitialized reads (or conditional considered uninitialized memory). This is possible with MSan but MSan is a little more challenging to use than ASan+UBSan or TSan.

In the years that I've used the sanitizers, I have yet to discover a false positive. It's always been a real bug.

I tried getting MSan to run but it never links properly for me.

Can someone explain how windbg is valgrind- like? I always considered it a debugger.

windbg has many powerful plugins, and some do automatic verification (or talk to tools that do so), so it can be a tool kit for verifiers like valgrind is.

Interestibg. Are there advanced blogs or so out there, demonstrating in the real world? Raymond chen seems to only touch it's surface.

The author might be referring to the recommendation to run a program that's instrumented with Application Verifier under a debugger, which WinDbg is one of many.

Reply on news.ycombinator.com