Is this the result of 'open use of pointers' that inevitably causes problems here and there?
Or is this problem due to the fact is just impossible to use even hard idioms to avoid such problems? (Or hard to enforce them?)
On some small-to-midsized projects I've seen narrow and rigorously enforced rules around pointers that seemed to work well. So what's up?
"In theory, yes - in practice, no."
You can use static analysis, vendor-specific annotations, runtime tools like clang's various sanitizers or valgrind, thorough code audits, use smart pointers instead of raw, have clear ownership semantics, follow MIRSA C guidelines, NASA guidelines, etc etc etc... but eventually your project will grow to the point where you'll botch an edge case, and the number of botched edge cases all your coworkers manage to add up will turn into a statistic.
Even if me and all my coworkers were all godlings incapable of making mistakes, we'd still end up debugging plenty of memory safety issues - in second/third party code, sometimes without source code, and writing bug reports / workarounds as a result.
Some of these idioms are not zero-cost though. My understanding is that to prevent use-after-free you basically can't use bare pointers or references, at all, ever. You need shared ownership everywhere (std::shared_ptr, not 0 cost), a garbage collector (In c++ that's probably another smart pointer type, not 0 cost), or additional metadata like the lifetimes fed into Rust's borrow checker.
Based on my reading here on HN, I think Chrome has a reputation of using modern C++ features extensively to try to improve its memory safety, but it's really hard to do in C++.
Edit: I realized this answer sounds like "yes idiom will fix it". To directly answer the question of "can you use idioms to fix this", no you can't. Evidence shows we screw it up. It's broken and you can't bolt on fixes to make it work.
"Today we were unlucky, but remember we only have to be lucky once. You will have to be lucky always."
Many people underestimate what is possible using C++ these days. Complex for sure, but also very powerful.
And that's the problem. There are very few true C++ experts, and most people will run into these issues as a matter of course.
This just feels like a reduction to the usual: if you are perfect, you will write perfect code, and never have memory-safety bugs. Thanks, but I'm not perfect, and I'd rather write in a language with a compiler that rejects programs with memory-safety bugs.
And yes, that does mean sometimes it'll reject some programs that are perfectly ok, but the Rust borrow checker is getting better all the time, and sometimes you just have to accept being hamstrung a little for the greater good.
However, while lifetime management with smart pointers is certainly important, it doesn't help with non-owning pointers between objects.
Of course, one could make all the back references weak, but that turns out to also significantly hinder understanding of the code. It ends up being really easy to lose all assertions about when any given object that's weakly reference is actually supposed to be live, and you end up with code that's littered with checks everywhere "just in case".