This is not my area of expertise, but I’ve seen concerns of implementing the unified memory architecture off chip. But maybe it’s just that it hasn’t been done yet.
This has also not been a problem for any other unified memory system (which typically also do not include memory on-package), and I can't see any reason why it would matter.