You are correct, group theory did start with the permutations of roots of polynomials. In the other hand, the modern perspective is to consider these groups as coming from geometry (the etale fundamental group point of view) and is really a very successful theory.

I think it is generally useful to try and give a geometric interpretation to whatever we are interested in and groups certainly benefit from such an approach.

Moreover, the groups relevant to physics are very geometric things (they are literally spaces in the sense that the real number line is a space ) and the geometry plays a large role in these groups.

In fact I am struggling to think of an application of groups to mathematics where geometry does not come into play.

Even the abstract classification of finite simple groups uses ideas from geometry on a very crucial way (representation theory).