The act of stackwalking breaks. You often end up in completely nonsensical stacks halfway through and then all is lost.
That makes sense, as you're fucked by any false positive.
If it's a graph of potential stacks, though, wouldn't you eventually find the one that unwinds, if one exists?
GP mentioned in another comment that they don't actually follow the algorithm described by userbinator -- checking the return address for a preceding call -- which would increase the number of false positives a lot.
The thing that makes this tricky is that x86 is a variable length instruction set: you don't know where the preceding call instruction might begin, and you can't decode backwards. You can do it speculatively, but ambiguities are still possible.
Given that the full technique apparently works well for userbinator, we can assume that this isn't a very significant issue. The chances of a phantom call instruction will be pretty low, and there's a fairly low maximum instruction length.
Yes, but checking for valid CALL instructions is probably sufficiently accurate.
Is this just an artifact of old return addresses on the stack not being overwritten?
Probably most valid code addresses on the stack are from older calls that are no longer part of the current call chain.
There's a chance of that, but in practice it's easy to filter them out, since only one of the chains will be complete from the very top of the stack to the current stack pointer. The others will almost certainly be partially overwritten by other stack activity and incomplete, or not match the current stack pointer.