OOP languages basically back you into this corner once a problem becomes complex enough. As an example, DI is supposed to decrease the coupling so that tests can actually test units and not wind up performing integration testing. Low coupling results in a more agile codebase, but does not intrinsically lead to more understandable code.
What goes wrong is people think that you need to "DI all the things." Like any tool it can be used incorrectly and, from what I have seen, violating the single responsibility principle [when using DI] is what most often leads to unwieldy code. "Got a model class/object? That needs an interface too."
Abstraction is not to blame, it is the incorrect usage of abstraction that is to blame; which is sadly what you are taught in school and college. Spending even one day learning a functional language is enough to drastically improve your understanding of how abstraction is supposed to work in OOP languages.
> I mean like even if you had the source code it's so convoluted you could never make any use of it.
Some IDEs can help with this (VS2015 just added "go to implementation" for C# interfaces), as does the most reliable way to understand code: stepping through it line-by-line (or at least method-by-method) in a debugger - once you have a run-time itable the callee is no longer ambiguous.
TLDR; it's a necessary evil for OOP languages but is often taken too far due to bad theory that is taught to us all.
I think the main source of overabstraction is the idea that abstraction is a virtue in itself rather than only a tool to accomplish a task. This often takes the guise of different artificial value systems like "testability", "decoupling", and "reusability". These can be good things, but only as a means to an end. People lose track of that sometimes.
Overabstraction is often simply a matter of poor problem understanding. Sometimes you don't really know how to solve a problem right, so you bite off parts of it, work your way through many layers until it is solved. The abstractions are not ideal, because not much planning was involved, and you didn't actually have the experience to generalize appropriately but...you needed some way to decompose the problem.
Abstraction is a tool. It has a cost, but sometimes you need to pay it and sometimes you are better off not. We really need better ways of abstracting and, when necessary, unabstracting.
> identify the nouns in the problem
This is what I was getting at by mentioning education as a possible culprit. You get out of university with this bad habit right off the bat, instead of the correct one:
Identify the responsibilities in the solution.
Identifying all the nouns in the problem sets you up for failure because you are implementing a solution before you know what the solution actually is. Said another way: you are in a very literal way implementing the problem and not the solution.
I keep coming back to RPG's "worse is better" essay. The problem is that while the artifacts that we create might be horribly messy, they are better than the artifacts that are never created at all. Evaluating at a program purely by its end resulting state does not capture the entire, or even most, essence of what it means to write a program.
s/really/actually/
And yes, it is hard, but it actually works.
1) Professional programmers are constantly hammered with the idea that abstraction and decoupling are always desirable - you've probably heard about TDD. Somehow OOD has been taken over by this school of thought and there are very few good resources on what good OOD is and looks like.
2). Dynamic OO languages force one to have high coverage and many tests to make up for the lack of static verification.
And I don't agree with the TLDR, over-abstraction is not a disease that affects only OO languages.
Agreed, it's present in other languages and has been with us a long time. It does seem to have reached a fever pitch in mainstream OO culture. OO seems to encourage taking even a small piece and giving it the full-blown treatment of layers, design patterns, DI, and the rest. Because each piece is a whole, in a way, so it deserves it, right? Give each part of a program that treatment, and the whole becomes unwieldy to a degree that was seldom seen before OO.
It's a form of cargo-cult thinking combined with the fact that a lot of beginning programmers are exposed to the "abstraction is good" dogma without ever realising that it can turn into too much of a good thing. Vague statements like "single responsibility principle" (what is a "single responsibility"?) can encourage a ridiculous amount of over-refactoring, driven by the idea that shorter functions are better --- which is true up to a point, but I've seen it taken to extremes where more lines in the file are function declarations and their opening/closing braces than actual, purposeful, code. They almost always argue that a shorter function is better ("after all, isn't it easier to read 1 small line than 10?") but neglect to realise that they've actually increased complexity of the whole by forcing readers to go through deeply nested function call stacks. Meanwhile, names tend to get longer due to the specificity of these tiny pieces, which doesn't help much with readability. In some very extreme cases I've seen, the name ends up being longer than the code it describes.[1]
Instead, I think we should be teaching abstraction as a tool like any other --- use it when it helps reduce complexity by reducing code duplication, and warn against its overuse. Under-abstracted code is tedious and repetitive, but reasonably straightforward to understand, since most people tend to find reading things linearly to be quite easy. Over-abstracted code is highly nonlinear and takes you through an elaborate maze-like path.
[1] http://git.eclipse.org/c/aspectj/org.aspectj.git/tree/org.as...
So now we jump trough hoops to isolate things that are extremely resistant to isolation.
Incidentally, besides Smalltalk, there is another language, which implements this original idea of OO: it's Erlang (just view processes as objects and it all fits).
The pattern you describe is caused by OOP and inevitable in many of today's OO systems because OO does not offer sufficiently powerful abstraction capabilities [1]. Imperative often suffers from "We won't attempt to abstract this at all" (which is sometimes better [2]) and functional sometimes suffers from "powerful abstraction capability which requires a high degree of knowledge to wield". Pick where you want to be in the spectrum.
[1] See AbstractSingletonProxyFactoryBean in Spring. Spring is an expert team with as well reasoned architecture as is possible in Java, anti-abstractions like this get written because they are the least-bad solution when a shitty problem comes up and the language has you cornered. http://docs.spring.io/spring/docs/2.5.x/api/org/springframew...
[2] "C vs C++ linus torvalds" https://www.google.com/search?q=c%20vs%20c%2B%2B%20linus%20t...
Linus is a great programmer and in the kernel space I would listen very closely to what he has to say - minus the profanity and insults. Outside of that space he is as subject to cognitive bias as any other well respected authority. There are many respected programmers that will take the opposite view in the C vs C++ debate.