I wouldn't go that far. If you have two signals emitting from physically separate location and they are out of phase, you just get comb filtering. Is it audible? It can be, but its not the phase you're hearing, it's the drastic frequency notching in amplitude. More importantly, you're going to get comb filtering no matter what you do with a multi-speaker setup. Just move your head a couple inches out of the ideal sweet spot equidistance from each speaker and you'll have created an effective phase shift and get the same type of comb filtering. In other words even with a theoretically perfect pair of time aligned speakers with perfect cross-overs, you'll still get comb filtering if you take various measurements around the listening area. Just moving the mic 4 inches can have drastic effects on the measured response. Incidentally if you've ever seen someone taking a measurement with a sound meter and rhythmically moving the mic around in a strange fashion, they are doing that to try to even the effects of comb filtering.
> This is a critical aspect of crossovers- since they contain the same signal, any phase differences between the intersecting components are easy to pick up by the ear and very difficult to measure since it won't show up as a frequency or amplitude differential.
If you're taking a measurement in the crossover region and there is a phase shift between the drivers you most certainly will see an effect in amplitude as you'll have at least partial wave cancellation. You can certainly take measurements in locations what won't show this, but that is always true. You can take measurements of a perfectly time aligned speaker that make it look like it has phase issues too, if you put the mic in the right spot.
Likely the largest improvement you get from time alignment of the drivers in a multi-way speaker is the ability to control vertical lobe tilting, but you can 'fix' that issue with MTM layouts without time aligning the drivers as well.