Plus, in the real world, many of us use what we call "third party code" which could be as simple as something as a database driver or some kind of utility class. If you think you're going to remember perfectly everything all the time, then you are dreaming.
And on refactoring: if you don't think it's essential, please do yourself a favor and find someone who can give you a tutorial. Refactoring is the key feature of all of these IDEs.
Say you have 100K lines of code in a project. You want to move a class to a new package. In your world, you spend a week doing it and probably typing javac over and over on the command line. In my world, I can do it in less than 5 seconds and know the IDE is 100% correct. When you have power like that in an easily accessible frontend, you tend to use it. That benefits your code by making the big ugly maintenance tasks into afterthoughts.
Perhaps you should learn to use the command line if you think refactoring without an IDE takes a week. :-/
Anyway, the automated tools sort of fake this workflow, but sometimes the real thing is nicer.
The automated tools don't fake your workflow. A refactoring tool performs code modifications that it can prove make no change to the semantics of the code. You can sort of fake this, in a painful way using perl, but as you've just stated, your system is so unreliable that you create a branch to do something I may do a dozen times in an hour when I have some time to go back and clean up.
Yes, judge thousands of people by one person. That has served society very well over the years.
Anyway, in Eclipse, every non-trivial rename "requires preivew" and often requires cleanup. If you just want to change the name of a class, the GUI is fine. If you want to do something more involved, you end up doing it manually anyway.
For trivial renames in Emacs, I have a function called "rename-this" which is mode-sensitive and Does The Right Thing for languages I actually use (Perl, Haskell, and Lisp). (Most things are non-trivial, though, as dependencies on a class name are not always in the form of the class name in the source code. "${prefix}::Foo" is hard to detect, but occasionally appears. Refactoring works well in Java because there is no possibility for syntactic abstraction, and it must be a function of your text editor. That approach is why I avoid Java.)
Edit: BTW, if you get to make a snide comment, so do I:
"Your post has finally convinced me that the only people who cling to IDEs for large-scale development are true believers who are more comfortable clicking buttons than writing software." Tool building is an important skill that IDEs actively discourage.