It shouldn't be a matter of preference. Use a pointer if you need it to be relocatable or if you need it to be optional (it may be nullptr). Otherwise use a reference.
There are also a number of cases where passing a pointer into a template function can be more concise. E.g. since pointers are iterators, if you have a generic function that iterates over things and you just want it to look at one thing (or zero) for testing, you can usually do this with pointer arithmetic.
Hm slightly tangential but could you install some kind of page-fault handler on a reference such that the control flow jumps on write back to the last place it was accessed? Not that you’d want to do that in the real world, but it would be an interesting “pub-sub” type control flow..