Try remembering and telling someone your home's coordinates. Then try telling them the coordinates of the restaurant up the street.
Decoupling from that seems like it would be set to fail for human memory, since there would probably be nothing to reinforce the memory over time for your restaurant example.