I don't see how understanding pointers and memory layout help at all with borrowing concepts.
You can get very far writing software without even being aware that your language is even doing something called garbage collection for you, or without even having a basic concept of memory being tied to values, sizes, etc. If you lack this context a borrow checker just seems like a pointless nuisance.