I do think it could change the sound. Consider a simple octave interval, with zero phase or 90 degrees out:
https://www.wolframalpha.com/input?i2d=true&i=Chart+Sin%5C%2...
https://www.wolframalpha.com/input?i2d=true&i=Chart+Sin%5C%2...
The first looks a bit like a sawtooth wave, the second is more like a chain of pulses. Wouldn't that sound different?
The Fourier series of Sawtooth wave, Square wave, etc. contain phase info as well as frequency.