Plus, in the real world, many of us use what we call "third party code" which could be as simple as something as a database driver or some kind of utility class. If you think you're going to remember perfectly everything all the time, then you are dreaming.
And on refactoring: if you don't think it's essential, please do yourself a favor and find someone who can give you a tutorial. Refactoring is the key feature of all of these IDEs.
Say you have 100K lines of code in a project. You want to move a class to a new package. In your world, you spend a week doing it and probably typing javac over and over on the command line. In my world, I can do it in less than 5 seconds and know the IDE is 100% correct. When you have power like that in an easily accessible frontend, you tend to use it. That benefits your code by making the big ugly maintenance tasks into afterthoughts.