If you code does intresive computation with structures containing a lot of pointers, you’ll get much more cache miss when using 64 but pointers.

The performance hit can be quite heavy!

Cache is a limited resource, and you want as many pointers as possible to fit there.