I think you have two incorrect assumptions here. You're assuming A) that parties and political ideologies exist in 1 dimension, and that B) the second party is closer to the third party than the first, such that its votes will go to the nearest neighbor, which is then the third party.

Even granting A, B is definitely not true. There are many similarities between the two major parties, and removing one in favor of a third party would likely split voters between the third party, and the other major party.

Also, A is not true either, so the distance between parties becomes even more unclear.

I for sure believe voters should be able to reject the current political choice on offer! I support IRV over our current system for sure. But, my main point is that IRV is actually not what many people believe it to be, and is far worse than other, simpler systems, such as approval voting.