Overcoming Intuition in Programming
amasad.me
amasad.me
It's a form of cargo-cult thinking combined with the fact that a lot of beginning programmers are exposed to the "abstraction is good" dogma without ever realising that it can turn into too much of a good thing. Vague statements like "single responsibility principle" (what is a "single responsibility"?) can encourage a ridiculous amount of over-refactoring, driven by the idea that shorter functions are better --- which is true up to a point, but I've seen it taken to extremes where more lines in the file are function declarations and their opening/closing braces than actual, purposeful, code. They almost always argue that a shorter function is better ("after all, isn't it easier to read 1 small line than 10?") but neglect to realise that they've actually increased complexity of the whole by forcing readers to go through deeply nested function call stacks. Meanwhile, names tend to get longer due to the specificity of these tiny pieces, which doesn't help much with readability. In some very extreme cases I've seen, the name ends up being longer than the code it describes.[1]
Instead, I think we should be teaching abstraction as a tool like any other --- use it when it helps reduce complexity by reducing code duplication, and warn against its overuse. Under-abstracted code is tedious and repetitive, but reasonably straightforward to understand, since most people tend to find reading things linearly to be quite easy. Over-abstracted code is highly nonlinear and takes you through an elaborate maze-like path.
[1] http://git.eclipse.org/c/aspectj/org.aspectj.git/tree/org.as...
So now we jump trough hoops to isolate things that are extremely resistant to isolation.
Incidentally, besides Smalltalk, there is another language, which implements this original idea of OO: it's Erlang (just view processes as objects and it all fits).
OOP languages basically back you into this corner once a problem becomes complex enough. As an example, DI is supposed to decrease the coupling so that tests can actually test units and not wind up performing integration testing. Low coupling results in a more agile codebase, but does not intrinsically lead to more understandable code.
What goes wrong is people think that you need to "DI all the things." Like any tool it can be used incorrectly and, from what I have seen, violating the single responsibility principle [when using DI] is what most often leads to unwieldy code. "Got a model class/object? That needs an interface too."
Abstraction is not to blame, it is the incorrect usage of abstraction that is to blame; which is sadly what you are taught in school and college. Spending even one day learning a functional language is enough to drastically improve your understanding of how abstraction is supposed to work in OOP languages.
> I mean like even if you had the source code it's so convoluted you could never make any use of it.
Some IDEs can help with this (VS2015 just added "go to implementation" for C# interfaces), as does the most reliable way to understand code: stepping through it line-by-line (or at least method-by-method) in a debugger - once you have a run-time itable the callee is no longer ambiguous.
TLDR; it's a necessary evil for OOP languages but is often taken too far due to bad theory that is taught to us all.
I think the main source of overabstraction is the idea that abstraction is a virtue in itself rather than only a tool to accomplish a task. This often takes the guise of different artificial value systems like "testability", "decoupling", and "reusability". These can be good things, but only as a means to an end. People lose track of that sometimes.
Overabstraction is often simply a matter of poor problem understanding. Sometimes you don't really know how to solve a problem right, so you bite off parts of it, work your way through many layers until it is solved. The abstractions are not ideal, because not much planning was involved, and you didn't actually have the experience to generalize appropriately but...you needed some way to decompose the problem.
Abstraction is a tool. It has a cost, but sometimes you need to pay it and sometimes you are better off not. We really need better ways of abstracting and, when necessary, unabstracting.
> identify the nouns in the problem
This is what I was getting at by mentioning education as a possible culprit. You get out of university with this bad habit right off the bat, instead of the correct one:
Identify the responsibilities in the solution.
Identifying all the nouns in the problem sets you up for failure because you are implementing a solution before you know what the solution actually is. Said another way: you are in a very literal way implementing the problem and not the solution.
I keep coming back to RPG's "worse is better" essay. The problem is that while the artifacts that we create might be horribly messy, they are better than the artifacts that are never created at all. Evaluating at a program purely by its end resulting state does not capture the entire, or even most, essence of what it means to write a program.
s/really/actually/
And yes, it is hard, but it actually works.
1) Professional programmers are constantly hammered with the idea that abstraction and decoupling are always desirable - you've probably heard about TDD. Somehow OOD has been taken over by this school of thought and there are very few good resources on what good OOD is and looks like.
2). Dynamic OO languages force one to have high coverage and many tests to make up for the lack of static verification.
And I don't agree with the TLDR, over-abstraction is not a disease that affects only OO languages.
Agreed, it's present in other languages and has been with us a long time. It does seem to have reached a fever pitch in mainstream OO culture. OO seems to encourage taking even a small piece and giving it the full-blown treatment of layers, design patterns, DI, and the rest. Because each piece is a whole, in a way, so it deserves it, right? Give each part of a program that treatment, and the whole becomes unwieldy to a degree that was seldom seen before OO.
The pattern you describe is caused by OOP and inevitable in many of today's OO systems because OO does not offer sufficiently powerful abstraction capabilities [1]. Imperative often suffers from "We won't attempt to abstract this at all" (which is sometimes better [2]) and functional sometimes suffers from "powerful abstraction capability which requires a high degree of knowledge to wield". Pick where you want to be in the spectrum.
[1] See AbstractSingletonProxyFactoryBean in Spring. Spring is an expert team with as well reasoned architecture as is possible in Java, anti-abstractions like this get written because they are the least-bad solution when a shitty problem comes up and the language has you cornered. http://docs.spring.io/spring/docs/2.5.x/api/org/springframew...
[2] "C vs C++ linus torvalds" https://www.google.com/search?q=c%20vs%20c%2B%2B%20linus%20t...
Linus is a great programmer and in the kernel space I would listen very closely to what he has to say - minus the profanity and insults. Outside of that space he is as subject to cognitive bias as any other well respected authority. There are many respected programmers that will take the opposite view in the C vs C++ debate.
we should be careful to not enable the belief that programming should be as easy as gluing things together
Absolutely agree, although a lot of it is.
I find the whole "do it yourself" mentality to be a form of NIH syndrome.
One case I learned the hard way (once!) is parsing/writing CSV files. It seems so simple. But there are 3-5 special cases that are hard and rare enough that you always should use the standard library written for your language.
Perhaps admitting your original decision didn't work out and and starting over is the real important skill...
https://tools.ietf.org/html/rfc4180
But in my experience, a lot of generated CSV is non-standard, and in that case finding a library to do it doesn't help --- you need to write your own parser so you can adapt it to the quirks of the variant(s) you're consuming.
Maybe coding based on researching solutions is a good way to do things - we just need to make it more efficient.
Looking for an existing implementation, when a "10m implementation will do" - How do you know it will do if you've not researched more? When do you stop?
I think risk-assessment should be more of a thing in software development. We can't really know where the bugs might be, but we can know where we'd like them the least?
This is why perl and CPAN is my goto language.
Edit: It always feels as if Dijkstra had been a true 50/50 growth/scarcity mindset kind of guy, in any case it's a very worthwhile read I think so I've re-shared your article over here:
Dijkstra was once the head of the "programming is hard and programmers should suffer" school of thought. Few people read his books today.
He wanted to bring programming back to mathematical fundamentals, so it is simple to reason about. It is my personal opinion that this ethos is the drive behind the renaissance of functional programming.
Programming is hard. I've been programming for 25 years now and if anything, it never gets easier. To find the right abstraction, so only the correct things become simple, is one of the hardest tasks.
Whether or not it's natural mapping functions to and from concepts people might already operate with isn't the same problem. I'd argue that one can be offset with practice, though it's certainly not something that is easy for everyone. In that sense, FP might be a bit more immune to bad intuition, if only that you have to give up on mapping structure to insufficient analogs. The late object oriented point of view makes this step trivial for our minds and is thus a significantly complicated trade-off to quantify.
Anyway, FP people are nailing composability and reusability to never seen level just in front of your eyes. You just have to keep them open to see. OOP did it at its time too, it just hit a ceiling; but there's one reason every imperative language is OOP nowadays.
The unfounded anti-OOP statements on this site are so ridiculous. That is entirely untrue.
Unfortunately, a lot of real-world programming doesn't have mathematical fundamentals and is instead what I call "information bureaucracy". Some data items go in one side of the system; we sort, collate, bend, fold, spindle, mutilate them and they come out the other. Also called "business logic", although it is often infuriatingly illogical and crammed full of exceptions to obtuse rules.
Typically business rules systems are an exercise in DSL design since you need to construct interesting combinations and programs within a very novel domain. Doing this is rather mathematical!
This is fine for academic CS, but can become a unicorn hunt - of the worst kind - for real world applications.
>To find the right abstraction, so only the correct things become simple, is one of the hardest tasks.
This is very true, and underappreciated in CS. The problem is that every language has its own set of ready-made abstractions, libraries offer further abstractions, and philosophies - OOP, design patterns, functional programming - offer a further set.
For any given problem, all of these may be non-optimal.
CS could maybe spend the next fifty years working on a useful theory of abstraction design, instead of trying to bake assumptions about how abstractions are supposed to work into the tools we use.
I'd love to see a meta-language which could be used to reason about impedance matching between abstractions and problem sets. I'm not sure such a thing is possible - Haskell is some way along, kind of - but it would be an improvement on languages that make it hard to design new abstractions because they come with rigid assumptions about the kinds of operations that are possible with code and data.
You could argue no restrictions exist in Turing complete languages. But for many problems the limit isn't Turing computability, it's how easy it is to find/invent expressive abstractions that help make hard problems more tractable.
That said, while I'd like to use as my sole development tool, I'm not sure I'd be able to. The (proprietary) framework I'm forced to use is basically impossible to navigate without some kind of IDE support. Yeah, it's not ideal, but it's the state of the world I have to deal with.
This is what people often talk about as code smells. When you've trained yourself in how good code looks, then bad code stands out to you. Once intuition has led you towards where attention is needed, then analysis can begin to devise the best solution. I find this to be a critical technique and tell people all the time to train their intuition. It's a tool like any other, to be honed and applied with skill.
Everything works great -- until it doesn't. And then you're spending your valuable time and brainpower learning about some framework instead of about how to do the job you're trying to do. That's almost always a fail. As the author mentions, it just gets worse: many times you're suckered into thinking the answer for your problem is yet another plug-in or framework. Now you have even more stuff to keep track of.
OOP forces you to answer the question: what does this system represent? Once you do that, then you're left with several metric tons of how to represent it well, which can lead to a _lot_ of stuff. FP, on the other hand, forces you to answer the question: what, exactly, do you want the system to do?
As it turns out, unless you're strictly happy-path, the shortest way to an answer is just to do the work. (Of course, nobody is saying to throw out core libraries and start writing in machine code. But if you're bleeding to death from paper cuts on WhizBang 4.0 that you just downloaded last week, you screwed up.)
We're also trained as programmers not to reinvent wheels poorly. Or not install a library to do something that is already supported by an existing library you're already using. When you need to do something like, say, open a new browser window, it's not that surprising to find a lot of posts about how to do it with jquery.
I don't find it always bad. As long as you understand that you are using a library, it's ok to go searching for its edges.
> Don’t code blindfolded. Attempting to build an application you don’t fully understand, or to use a technology you aren’t familiar with, is an invitation to be misled by coincidences.
Did I miss something, other than the attempt to coin the phrases "framework negative space" and "framework intuitive space"?
The state of most commercial codebases and a lot of colleagues I've worked with, and my own code as well show that it is indeed that hard, I'd like to argue. It's easy to state, difficult to do well.
While I understand what he and others have said, which is what I had believed in the past, I think that the emphasis to think like a machine is heavily misguided as we as users need to steer programming languages toward their natural end (pun intended) which is where AI is headed.
But we're miles away from that so the argument he puts forward is temporarily valid.
I think the biggest hurdle for programming languages currently is flow control. Specifically, programming languages are mostly still written top-down, leaving the implicit assumption that code is executed in the order you read it in. This is obviously a false impression for pretty much any program written nowadays. Inventing a new way to present code so that program flow is easier to visualize, is in my opinion the "next big thing" in programming language UI. This is something tools and concepts like UML have tried and failed to revolutionize, so I'm not sure what the way forward really is.
I think I can relate to this frustration/confusion because I went through it when I was obtaining my B.Sc. degree but I cannot exactly remember nor reproduce how I overcame this phase, or made sense of "it" all (for lack of a better term).
Can you folks help me with this? How would you guide or help a beginner like this person in programming?
I often think about how to answer this question to beginners as well. I think he is right, the answer is probably "it doesn't matter".
Tooling matters a lot if you want to be doing this for a living.
If they wanted to know, they'd have to ask me. Very few times, they did.
Resume Driven Development is where it's at. Docker? Our "modular monolith" works just fine, but with Microservices we can now rewrite everything in Rust and Elixir. Lock ourselves with yet another build / automation tool just to remove trailing whitespace. Add RethinkDB and Redis and that new graph database now that we're at it. Let's pray for updated NixOS binaries!
For toy projects I'll stick to OpenBSD, Perl and ancient tools like make, awk or even rc.
Treehouse's learning tracks (https://teamtreehouse.com/tracks) are useful for a beginner to look through to get a general idea of what that would involve.