Except that you've got the point completely upside down! At first people didn't believe in heliocentrism because it used circular orbits, and instead used to believe that the universe is better described from the geocentric point of view which was one ginormous mess from the mathematical point of view. Then people realized that the beautiful addition of conical curves solves the problem completely. I say 'beautiful' because clearly you don't have a sense of mathematical beauty. Mathematical beauty is not only the simplicity of the solutions, but also it's robustness and fluid way in which it fits into the rest of our knowledge. The real simplification that came from it is far deeper than any 'circular orbits' picture you imagine. Naturalness and search for beauty can never be proved wrong. Those principles can perhaps only be applied prematurely, before enough data is collected.

Not recently.

Dirac did believe his equation had to be right as it was so elegant, however it turns out we now interpret it differently now