Edited to add a link to a simple Matlab solution of this problem: https://www.mathworks.com/matlabcentral/fileexchange/42919-p... (click on the Functions tab to see the source code)
Of course, if you start with a real valued signal, then you already lost that before even getting to the transform part. Or in other words you have to transform a signal twice as long as it needs to be, because half of it is mirrored and then discarded (I assume).
You can think of it in terms of degrees of freedom. A real-valued length-N signal has N degrees of freedom. A length-N spectrum is complex-valued, meaning 2 degrees of freedom per frequency bin. When you constrain it to be conjugate-symmetric you bring the degrees of freedom back to N, which matches the real-valued signal.
The FFT segment starts at 4:30 (linked), the specific part about phase and magnitude is at 5:45, and the part about negative vs. positive frequencies starts at 6:35.
Here's an earlier version of the synthesizer (licensed under AGPL3). The MIDI CCs for controlling different parameters are listed in the source code. You'd want to clone the repo, run through the installation instructions in the mb-sound repo, do a bundle update mb-sound, then run bundle exec bin/complex_synth.rb. https://github.com/mike-bourgeous/mb-surround/blob/3823de44a...
Here's the core sound repo (licensed under BSD) with some examples for getting started: https://github.com/mike-bourgeous/mb-sound
I don't plan on making the visualizations available, in part because the system is too convoluted and they probably only work in my specific environment.