That just means you've never worked on a big project, where the component you are responsible for is used by >10 different other components. One very quickly reaches a limit where manual refactoring is an option.
Its just that I don't believe in succumbing to the crutch of tools. If the IDE is getting in the way, get the IDE out of the way - too many projects are incapable of doing this.
Those two are orthogonal concepts. There is e.g. car or medical embedded software that's tiny in scope compared to a big enterprise project.
No, you really don't need the IDE to make big software.
Programming, as most technologies, is not about what you basically need. We are not cavemen.
It's about what enhances what you can do.
The entire code base running a modern car goes well within millions of code lines. Our IDE changed in time from Blue River X32 v5 to internally modified Eclipse plugin. The only major use for us are X32 type contractions. [1]