Some tools (e.g. IDA) will just completely corrupt their database trying to relocate to a base different from the one in the image. It also makes for random looking crash reports.

Not as much of a problem if you have debug data.