SQL databases _could_ conceivably optimize foreign key constraints behind the scenes by replacing/augmenting the referencing column with the physical location of the foreign row, but what do you do when the target row moves around in the physical store? You'd have to hunt down all the references and replace them. If anything can point to anything, that could have a cascading effect.
You could have conceivably have an indirection here: Instead of pointing to the physical location of another row, you pointed to the physical location of a special intermediate file that itself could contain the physical location. I don't know of any database that uses any tricks like these.
You can, it just requires one further level of pointer indirection. Rather than pointing to the physical location of the data, you store a pointer to an offset in an array of pointers.
Alternatively you can point directly if you're prepared to update those pointers when you move things around. Luckily a graph database is ideally suited for this kind of operation because it makes discovering those pointers exceptionally simple and efficient.