Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)
marcan.st
marcan.st
Flashbacks to a job where I was asked to figure out why a newer kernel was crashing. This was a very frustrating time, because I had (have) basically zero real C/C++ experience but I'd helped out with Bitbake recipes and everyone else was busy or moved to other projects.
To cut a multiweek tale of dozens of recompilations short: The kernel was fine. The headless custom hardware was fine. The problem was a hypervisor misconfiguration, overwriting part of the kernel address space. All of our kernels have been corrupt, but this was the first one where the layout meant it mattered.
A month of frustration, two characters to fix, the highest ratio I've encountered so far.
My reward for struggling through a complex problem I was unqualified for? "Great, now we need to backport security patches from the main Linux kernel to the SoC vendor's custom fork..."
I've run into some of these where it's a lot more rare to hit, and so then it's reasonable to not do the thing that hurts, but watch out for it in the future. Sometimes you get lucky and it magically fixes itself forever; sometimes the weird case that you only hit with internal traffic ends up getting hit by public tratfic a lot.
Crashes like this where a wild write breaks something at a distance are always a PITA to debug (especially here, where the wild write is harmless if there's no data race)
As long as they're moderately competent, yes. They need to know how to debug normal things, but they don't need to handle every esoteric niche. The thing that matters is ability to put out tons of productive code.
Often that means a speciality where they're an expert, but it doesn't have to mean that.
> When I think of 1x (or less) devs they are usually the type that don't get things done because they can't (without a lot of help), not because they are slow. I.E. overall technical chops, not just speed.
It sounds like you're taking "1x" as a dismissal. Isn't it supposed to be a pretty ordinary dev?
But to my original point, I just don't buy the 10X story as being only how fast you write code. I've known developers who designed and wrote such that much time was saved working with that code in the future. These people are, IMO, much more deserving of the 10x moniker as maintaining code is much more important that just cranking it out... with the only possible exception being early stage startup (but not really as they will be crippled in the next stage).
That said, if a 10x developer does exist, Marcan is one of the few of them. What a ridiculous statement.
He works on Asahi Linux, a Linux port to arm64 Apple hardware.
Comparison is the thief of joy!
The explanations are also very clear. Thanks for posting.
But vDSOs don’t enter the kernel. They just run as userland code; and so they depend on the userland allocated stack to be arbitrarily deep in a way that kernel calls don’t.
As shown in the article, Golang seems to have code specifically for dealing with vDSO-type pseudo-syscalls — but this is likely a specialization of the pre-existing syscall-carrier-thread allocation code, and so started off with a bad assumption about how much stack should be allocated for the created threads.
(I should also point out that the OS stack size specified in the ELF executable headers, only guarantees the stack size of the initial thread of a process created by exec(2). All further threads get their stacks allocated explicitly in userland code by libpthreads or the like calling malloc(2). Normally these abstractions just reuse the same config params from the executable (unless you override them, using e.g. pthreads_attr_setstacksize). But, as the article says, Golang implements its own support for things like this, and so can implement special thread-allocation strategies per carrier thread type.)
On Linux the 8192K aren't reserved unless the are actually used, so what is the point?
Ok, Golang will allocate green threads from its own allocator, but 104 bytes?!
Applications should not be assuming a hard-coded stack size, that's a bug. They should be checking the value of PTHREAD_STACK_MIN. That is what it's there for.