C-x C-SPC (pop-global-mark) will work across buffers.