Are you genuinely attempting to argue that thinking ahead and fully understanding the problem isn't preferable to tweaking one's way to a solution?
Sometimes, exploring the problem space can give you a fuller understanding of a problem faster by forcing you to confront pitfalls that may not be obvious until you try a solution. We use all kinds of wonderful terms for this - "Agile", "Prototyping", etc.
Both extremes - fetishizing planning and up front design, or fetishizing short term iteration and poking things without deeper thought - have their problems, and occur too often. Neither tool is a panacea, but both have their place.
It's good that you mentioned Java, because it is a language which I find extremely IDE-centric, and I suspect that's also what causes easy premature generalisation --- creating new classes with tons of boilerplate automatically generated by the IDE is so easy that it encourages programmers to. That doesn't help one bit with the details of the algorithm, unfortunately; it often gets "OOP-ified" into a dozen classes and much-too-short methods created as a result of the "fiddle with it until it works" mentality.
I'm not saying that there are no unjustifiable over engineered java libraries, but the current hype cycle of web frameworks seem to indicate the burden of proof of good design should lie with current technologies as well as previous.
"Everyone uses it" or "it's producing value" doesn't mean it's not an overengineered mess that everyone recognizes as such - it just means that imperfect code still trumps no code. Switching languages usually means tossing out your old codebase, leaving you at "no code".
I have worked on such messes, created such messes (oops!), and cleaned up such messes.
That said, I'm sure there is a Java project out there which actually benefits from stereotypical levels of Java abstraction and patterns - and I'm sure there's a few codebases out there where "my" and "others" opinions differ exactly as you say.
> I'm not saying that there are no unjustifiable over engineered java libraries, but the current hype cycle of web frameworks seem to indicate the burden of proof of good design should lie with current technologies as well as previous.
100% agreed - not that I'm qualified enough at web dev to have much of an opinion on this. If anything, the churn of web frameworks smacks of being both overengineered (do you really need a whole framework for that?) and underengineered (wait why are we replacing things yet again?) simultaneously.
But it's the exception rather than the rule. Once you have something like Spring in your codebase, to take care of modularity and reuse, everything else should be coded with as little "abstraction and patterns" as possible.
One could just as easily hypothesize that these tools let you avoid thinking in the small, and help you form a big picture overview that would otherwise be difficult to understand.
I went quite a while with no tools other than a hex editor to type in op codes. I don't think it did anything except hurt productivity.
Maybe you learn or work better that way. I don't. And I don't see how you justify assuming all new programmers would.
Would I recommend an IDE for a low level language like C? Probably not, because it forces a kind of laziness on the programmer.
Maybe an IDE isn't the solution, but a starting point to build upon. Something that's an interactive environment like LightTable has, where you can quickly eval blocks of code and see the end result without having re-compile your entire program. Certain languages are better suited to this, and certain paradigms (reactive programming comes to mind).
Of course it's possible to push code around on the page until it seems to work, just as it's possible to push symbols around the page until it seems to work when answering a mathematical question on paper. (Unfortunately some languages/compilers will run code that doesn't make any sense, but that's more true on paper, not less)