The sad truth is that I have seen, several times, people justifying real code doing questionable things with pointers in C(-like languages) by saying "well but pointers are just integers".

I am not sure what the best way to teach C is. Maybe it is a good idea to start with a "lie" and explain pointers as integers. But most C books/tutorials never go to the part where they explain that this is a lie, and that is a problem. The problem will only get worse as compilers get smarter and hence better at exploiting the UB that lurks in so many programs due to a naive treatment of pointers.