Hacker News Reader

The syscall writing the data is async, which requires the memory it uses to be available until until it completes. Since this was on the stack here, it means that the function that contains this memory on its stack must not return until IO completes

But the code that was supposed to wait until completion was interrupted by an exception, which unrolled the stack beyond the function owning the relevant memory. So the completing async operation writes into memory owned by somebody else now.

This is a general complication with completion based async IO. You could get the same problems without any interrupts, if you just don't wait for async IO to complete in some circumstances.

Thank you. Now I understand why Rust async is so complicated. It's about lifetimes. It's dangerous to write data asynchronously because... eh... that location might be already invalid.

I know this is not about Rust. Forgive me that I mentally connect this story with async Rust.

There are two ways how async IO can work:

* Readiness based: You use `select` (or preferably its successors) to figure out which file descriptor has data available. Then you synchronously read the data, which completes instantly because there is data available. This is the traditional way on Linux, and also the way Rust's async works. This approach doesn't run into any lifetime issues, because you only need to provide the memory (`&mut [u8]`) to a sync call once data is available. This approach works well from sequential streams, like sockets or pipes, but isn't a natural match for random access file IO.

* Completion based: You trigger an IO operation specifying the target memory. The system notifies you when it completes. This approach needs you to keep the memory available until the operation finishes. This is the traditional way on Windows (IO completion ports) and also used by the more recent io_uring on Linux. This one runs into the lifetime issues that caused the bug in the linked article. Thus in Rust it can't safely be used with a `&mut [u8]`, you have to transfer ownership of the buffer to the IO library, so it can keep it alive until the operation completes. The ringbahn crate is an example of this approach in Rust.

Good writeup. One thing to note is that Go, Crystal and other systems based on green threads tend to not run into the problem as they have separate stacks for different execution contexts.

Note that what you're referring to as "readiness-based I/O" is not asynchronous I/O in the strictiest sense--it's actually non-blocking, synchronous I/O.

I find this article helpful in clearing up confusion of asynchronous vs non-blocking I/O: http://blog.omega-prime.co.uk/?p=155

More like a general complication with SEH. One of several reasons why I don't allow it in any code I touch.

Anything that obscures the flow of control is a violation of literate programming principles, and exceptions are pretty high on that particular shit list.

> With that said, is there not a valgrind equivalent for Windows?

Yep, it's called Application Verifier and it will check all kinds of system calls and make the memory allocator incredibly aggro towards checking validity

> The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call?

When you call select(), your thread is in a special state called Alertable Wait; one way to get out of it is to have an APC queued to your thread, which has a similar vibe to a UNIX signal (with all of the similar caveats about how easy it is to fuck things up in it). Basically, OP's scenario is similar to corruption caused by a badly written signal handler

> The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call? Even if that is possible that sounds like a dangerous game to play, and this class of bugs seems natural.

I mean, signal handlers exist on Linux, too ...

> when you have a blocking syscall interrupted you will get a sigabort on linux

Doesn't the syscall just return EINTR?

The select() was not interrupted in a traditional sense. It looks like a special windows API was used (QueueUserAPC) which allows a callback to be called in user space while blocked in certain kernel calls. My guess is that when used normally, the callback returns and control goes back to select() being blocked, but in this case an exception was thrown which unrolled the stack and left the select() call holding onto an invalid stack address. Jumping out of a syscall using an exception like this will probably cause all kinds of problems and my guess this is completely unsupported.

https://docs.microsoft.com/en-us/windows/win32/api/processth...

Throwing from a signal handler on Linux is also UB. Or rather, it will blow up on Linux too, and that's why it's UB in the standard, because it'll blow up on any platform.

With the understanding that somebody had complex logic in the equivalent of a signal handler in linux, then i am not sure this bug is so fantastic after all. I have signal handlers so high on my danger zone that i do not recall ever coding any. If I am in a sigabrt or sigsegv i hope i die as cleanly as possible, so that a core dump is useful. No wonder he needed kernel level development.

No, absolutely not. It just illustrates the problems that obviously insane code can lead to and why we call it "insane" in the first place. But it was still a fantastic read following him on his journey from "huh, that's odd" to "I had to use kernel-level debugging to find out they were ~~throwing from an APC~~ using APC."

Imagine how much he could have achieved in those five days if he had just unfucked obvious bug magnets proactively without first painstakingly proving that they were really manifesting as bugs, and how.

There's IBM Rational PurifyPlus, which is kinda the same as valgrind, but with a Visual Studio plugin and fancy GUI. But when we bought it, we only got one license for the entire company because it cost around $10k per seat.

PurifyPlus was an old enterprise software similar to valgrind for Sun/Solaris. It eventually got Linux and Windows support, but for more than a decade has been one of those undeveloped systems milking legacy enterprise subscriptions as it slowly bitrots into uselessness. It was bought by IBM and then passed on to Unicom. These days it is not a reasonable choice for anyone not a legacy user.

Not sure if PurifyPlus existed for HP-UX, but I know Purify did (at a very previous job, we even managed to find a small memory leak in the standard libc, which we filed with HP; unfortunately, a repro required having a Purify license).

There’s address sanitizer on newer versions of visual studio now but in my experience getting it to actually work with all of your projects’ dependencies can be very hit and miss. Windows Debug build config also does a lot more memory checking (part of the reason it’s so slow) so you’re not totally screwed for automated tools.