The Complexity Trap: Think Before You Leap
danielwestheide.com
danielwestheide.com
Scala is a wonderful language and well-designed, especially in contrast to Java. The learning curve requires a good deal of effort, but with experience, it becomes incredibly fun to write Scala code. Once I got past the learning curve, Scala became my go-to language.
That said, there’s an underlying arrogance of Scala developers who, in my experience, tend to often abuse the language to show off their cleverness at the expense of users and maintainers. I don’t know why this is. I suspect it’s because Odersky set the bar so high with his obvious brilliance that some feel insecure.
One take away quote for me:
“Unfortunately, we don’t like boring. We like to take the language and the compiler to their limits. We also like to adopt the latest fad. But the software systems we work on in our jobs should not be a playground or a test bed for the latest fad.”
Scala developers aren’t doing anything new here. People have abused fads in programming for decades. “Design Patterns” and “Test Driven Development” and “RESTful” come to mind. However, it’s unfortunate that Scala developers in particular do this. Because, Scala seems to be a language specifically designed to avoid past mistakes.
Solving things with list comprehensions is very idiomatic python, so the thought process goes "the more of these I can stuff inside each other, the more idiomatic my code is."
Also, not nesting things is very idiomatic Python ("Flat is better than nested" is part of the Zen).
So, people generally don't think of nested list comprehensions as idiomatic.
How many levels of indentation are there on each line?
https://i.imgur.com/s1WBHxU.png
Spoiler: Line 3 has no indentation while Line 6 has one level. Line 4 has a mix of tabs and spaces that would confuse Python.
The fact of the matter is that indentation is white space, and white space is implicit by definition. It only exists as the gap between other things.
You will get a "TabError: inconsistent use of tabs and spaces in indentation" exception if you use Python 3.
I worked for years in Scala, and the biggest problem I had was OOP fanatics who clung to stale design patterns and out of date knowledge of general programming principles despite FP being both the simpler, cleaner, more legible, and more maintainable approach. Part of this is the fault of the individual not asking more of themselves, and part of this is the language, encouraging people to feel good about their FP+OOP hybridity, which is entirely incompatible as a concept. I've never had a new grad who could not make the switch to an FP mindset. I have, however, had older or more established devs who don't want to learn and thought they could get away with shit work (which is the main cause of scalability issues, not new concepts).
For the record, I never used more complicated concepts than Monads and MTL in my day to day usage. If you can deal with Monads, you can deal with MTL. If you can't deal with Monads, then you didn't put in the 1 whole day of work it takes to become comfortable with the concept. If you can learn what an AbstractFactoryBean is, you can damn sure learn that a Monad consists of 4 functions across 3 interfaces and that Functor:fmap => Applicative:(pure + ap + fmap) => Monad(>>= + pure + ap + fmap).
Not a part of that group, but I think this is a false narrative. 20% might be accurate, or it might be a bit high even. But it has nothing to do with `top` or `bottom` performers. It's purely to do with exposure to functional styles.
Learning Haskell first doesn't make you a top 20% developer, much as I'm sure HN in particular will want to argue with me on that point.
All I'm trying to say is: there's not an intelligence spectrum of `imperative` to `functional` with `object oriented` in the middle. They're different, and many who learn one first struggle with the others, but that doesn't mean they're lower performers in what they do know than the ones who can talk about currying vs partial application.
Everyone learns imperative first. "Wake up!" or "Eat your vegetables!" are imperative. No-one says to their kids, "map(vegetables, eat)"
If you want to use that example it'd be more like this snippet that no one on earth would tell their children:
> Starting from n = 0, take the nth vegetable from your plate, insert your fork into that vegetable, raise your fork to your mouth, bite that vegetable, remove your fork from your mouth, masticate, add one to n, and repeat.
1) Aim towards your functions depending only on the function inputs, to calculate a return value, with no side effects
2) Aim towards your functions having a return type that exactly describes the possible outputs of your inputs.
A lot of the magic after that is clever composition that, while having the benefit of making your code shorter and smaller and more "expressive", is also obfuscating to people that are less experienced.
The purpose of the advancement of technology is to make things easier. The practical use of these languages is to create something that makes money, with as little overhead as possible. A language should be selected to make things easier, with little overhead. Not being able to easily scale to 100 people is a very bad business decision, especially when the same functionality could be achieved in languages/methods that require less education (cheaper) and have a larger talent pool (cheaper). I'm talking about the use of tools that are meant for productivity, in general, not Scala.
> methods that require less education (cheaper)
This is not a good thing. Trying to devalue a skill is just asking for trouble, especially in engineering.
Also, it is not seldom that you hear them say "It took me n years to fully grok the _whatever FP concept_". Or Martin Odersky doesn't get the _whatever FP concept_.
So how do they expect the majority of sw engineers will use/adopt those principles if the guy who invented Scala "does not get it" and even themselves took so much time to "get it"...???
They tend to use it as something to show off their ego. It's a bit too academic and it often fails to understand the user's need.
It's often ignored about how the tool/library integrates with their user's code. For example, Http4s wants you to form your application around it, rather than just being a single component of your application.
--Brian Kernighan
How many times have I bitten myself this way? Dear past me: stop being so smart.
Personally, I want a mathematical structure of "problems" such that we can track when we're actually making things more complex or less complex. In the article, removing boilerplate was a problem because internal domain models were being exposed publicly. But what I really want to know is "what are we actually gaining by removing the boiler plate and what are we losing by exposing domain models". In like a general mathematical sense such that if everyone about programming changes (maybe suddenly we switch to APL or something), the lessons we learned are still valid.
So far the best I've managed is [1]. I've done a bunch of refinement in the last year and a half, but the general structure is about the same. I'm not suggesting that what I have is complete or even necessarily the right direction to go for this sort of thing. But what I am saying is that in order to make definite statements about how the problem and/or code you're dealing with is complex (that don't rely on appeals to authority) then you probably need to have done at least as much "work" as I have (if that makes any sense).
[1] - https://www.sep.com/sep-blog/2017/04/25/objective-code-quali...
My favorite part is the knitted castle.
This is a valuable insight. Especially if your main programming language supports both functional and procedural and/or OOP approaches; This statement offers a good guideline as to where the boundary should be.
I find that functional programming is great for business logic but not as great for more concrete logic where performance is important. For example, when dealing with low level concrete objects like sockets.
It's not hard to see that modern software engineering is cherishing accidental complexity, and as it comes from the "tools we use", the fair share of problem is in the languages, frameworks and tools we use. Unfortunately, there are very little of them that really take a tough stance on fighting complexity, instead of adding more features. Actually I can think only of one such language – Go, and, unsurprisingly, it's critizised heavily in many circles exactly for the lack of accidental complexity.
Tony Hoare wrote this in 1974: “But the pursuit of simplicity is one of the most difficult and challenging activities of the human mind. Progress is likely to be extremely slow, where each complexity eliminated must be hailed as a breakthrough. We need not only brilliance of intellect but breadth of experience, nicety of judgement, excellence of taste, and even more than our fair share of good luck.“
I was quite impressed after reading this essay (it's from "The software design: a parable" [1]) – written almost half a century ago, and there seem to be a little progress, if not negative. So books and talks are unlikely to change anything – we, people will still use the path of least resistance and always will be subjected to congitive biases – so probably only tooling can make a change. Go, for example, simply doesn't allow you to do many overly complex things (on purpose) and it results in extremely readable codebase – chances that you open random piece of Go code on internet and will not be able to understand it are almost zero.
I find it a terrific achievement, and wish other (future) languages could embrace the importance of it.
This is a little-acknowledged industry-wide problem that extends beyond hot-shot overengineering to reems of banal code too, so I would rather it be written from a less expected perspective.
Good functional programmers are happy not to introduce an extra layer of indirection yet, because they know that their code is easy to refactor. The article itself seems to come to that point later, yet it gets it completely backwards here.
Generate JSON formats the easy, zero-boilerplate way for now. As and when you have a requirement for a JSON representation that looks different from your domain representation - perhaps because you want to evolve your internal domain model while keeping the JSON representation the same - you can introduce manual JSON codecs (or DTOs, or whatever you prefer) at that time.
> Object-oriented programmers call this technique stubbing, and they get mocked for doing it, mostly by people like us.
They get mocked for doing it because it breaks all the rules of the language. It usually relies on reflection, bytecode manipulation, or other such horrors. It's often thread-unsafe. Having tests that you can't refactor the way you normally refactor code is a huge, mostly-hidden cost.
> But you have no clue if your production interpreter, which is talking to an external system, is working correctly at all. So abstracting over the effect monad and using the Id monad in tests doesn’t even help you to write meaningful tests.
Tests need to be decomposed like any other code. There's a huge value in testing your business logic separately from testing your interactions with external systems. Functional core, imperative shell achieves that, but at a higher cost: you can't interleave your interactions with external systems and your logic in the natural way in code, you have to explicitly, manually push all the interactions to the edge instead.
> What if we did server-side rendering of HTML instead? There wouldn’t be any need to design an evolvable JSON API, which, unlike simply exposing your domain model, is not trivial, and there would definitely be no need for defining any JSON codecs. And that’s just what you get on the server side. On the frontend side, you would have one piece of software less to maintain, you’d get faster load times and you could get rid of duplicated business logic.
You'd get slower load times as you had to do page transitions, and more duplicate logic as things like form validation had to be done on the frontend and backend with separate code.
> At some point, we realize that we have an architectural problem at the macro level. Maybe, if we continue to ask why long enough, we will come to the conclusion that all these services we need to communicate with exist in exactly this way because of how the company is organized.
And which is easier to fix? At some point you decide that the entirety of western civilization is fundamentally flawed, and you can either start ranting about how we need to rebuild society from the ground up, or you can put a bit of complexity in your software and make something actually useful.
> I was recently talking to someone about Rust, and that person said, “A language without higher-kinded types is completely useless!”. Yes, certain abstractions are not possible without them, but I have never seen a project fail due to a lack of higher-kinded types.
That may have been me, and I absolutely have seen projects fail due to lack of higher-kinded types. Not in an obvious way; rather it felt like a general sludginess to development, a need to repeat the same tasks over again, until the project eventually dies from a thousand cuts. (Hell, for all I know the company I left several years ago may still be trying to make that project work).
> Projects fail because the problem domain is not well understood, or because the architecture is not a good fit for the requirements. They also fail because of how the organisation is structured and because of politics, be it intra- or inter-organizational. Projects fail because of people, and because of people not talking to each other.
This is facile. Projects fail because of people talking to each other too much. Projects fail because people tried to fix their organisational politics instead of implementing technical workarounds. Projects fail because the technology choices were simply bad.
Reading this article was weird, because I found myself nodding along with most of the woolly principles but then massively disagreeing with every single example. It makes me think the article is arguing at the wrong level of abstraction somehow: no-one thinks you should use a complex solution when a simple one will do. Very few people will claim that the problem with today's software is that programmers think things through too much and should rush to write code instead (though for the record I think that's actually the case). But those consensus principles don't actually help us at all when it comes to discuss specific decisions.
People coming from easy to refactor languages have widely different rule books from the ones coming from boileplate-filled ones. It expected that the same person thinks people should rush to write code earlier and gets to understand how a project can fail because of lack of higher-kinded types. The hard part is explaining to people how those weird languages make one more productive, because their thought follows completely opposite rules.
What I was expecting: I was expecting the fad pushing an article about how too many people push components together and then they're trying to sell some kind of microservices nonsense.