The goal is to produce real valued noise - you won't get that if the spectrum is not mirrored.