Do We Worship Complexity?
innoq.com
innoq.com
What we ended up with was O(n) _classes_ (where n = the number of computation operations).
I remained in Java, but took a functional and dynamic approach yielding a three-line implementation of the engine where each command was simply a function. All the tests passed.
I was kind of holding my breath at this point. What will he say? We just went from this huge implementation to three lines. I thought he was going to love it. The crazy simplicity of it all.
But to my draw-dropping surprise he muttered something about adding new types and how he liked that the commands were coupled to the engine.
Is this complexity worship? Or what is this? It's almost as if his brain was refusing to see the solution. Like it couldn't possibly believe that the solution could be that simple. So it just ejected it, put a big black censorship block over it.
I'm currently reading a book on brain hemispheres. Apparently experiments have shown that the left hemisphere will completely and blatantly reject obvious evidence even if the right hemisphere 'has a clear counter proof'. Sometimes I think our industry suffers from an abundance of left hemispheric dominance. Or something like this...
Speaking of complexity worship...
Big O notation seems totally irrelevant here, unless the question involved scaling to arbitrary number of types of operations. Why not just say "we ended up with a class for every operation?"
(It's also faster to write, or even say out loud, "O(n)", than it is to write/say "proportional to number of").
Or, to put that another way: there is a constant cost in human parsing complexity when writing "O(n)", but the number of parse-nodes in the text when using "O(n)" only increases as O(n), whereas without "O(n)", it increases with O(n^1.3). ;)
A system that has O(n) code artifacts for n "business requirements" (e.g.) is far more costly than one that has O(1), say.
I understand what you are saying, I'm just objecting to how you are saying it. Unless you are interested in the limiting behavior as the number of "business requirements" goes to infinity, big O is the wrong tool. An O(1) approach that requires a foundation of 1M LOC is probably worse than an O(n) approach that requires 100 LOC per business requirement.
Sorry to obsess over the details here - I just think big O is overused in general and saw an opportunity to make a bad pun on complexity worship.
I.e. it's coder slang now.
> Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent. - orwell
RE that Orwell quote, I have mixed feelings. I agree with it in so far as it means "pick easiest words for your audience at the precision level you need". But in general, words are not equivalents, even if they're listed as synonyms - each word has its own specific connotation. Say, "car" and "automobile". Technically, they refer to the same thing, but they feel different. That subtle emotional difference may not be important in formal setting, but it's an extra dimension of communication in informal cases (like e.g. this comment thread).
Like the word "orthogonal" - always know I'm talking to a techie when that one rears its head.
I guess there just comes a point where you've solved so many optimization problems that it's hard to not think of a bunch of solutions with different attributes as being embedded in an N-dimensional space?
Versus
That is an entirely different concern.
The second is more approachable but requires an adverb to have the same meaning, and adverbs are also discouraged. I think orthogonal is fine.
So I guess we could nitpick over "entirely" and "basically" just as easily. Maybe he should have just singled out the engineers and other pedants.
I tend to use "mutually independent" instead of "orthogonal" when talking with non-tech people.
It feels analogous to the widespread use of "exponentially" to mean "a lot" or "quickly" which is a really bad, silly thing. The difference is that few physicists and mathematicians misuse "exponentially" in casual conversation, whereas you are claiming that software people deliberately misuse "Big O". I'm not sure I believe you but either way this seems regrettable.
Take the context of a high-profile art magazine, Frieze. (I googled "frieze increased exponentially"). This is shooting fish in a barrel—but the most egregious example in the first page of hits is this one:
"Seppie in nero’ – squid in its own ink – is my favourite Venetian delicacy. Although customarily served with polenta, I prefer it on thick spaghetti since pasta exponentially increases the naturally squirmy quality of the creatures’ tentacles, creating a Medusa-like mound of inchoate, salty matter."
So you've got an art critic writing slightly pretentiously about food, and he throws in an "exponentially" which has nothing to do with a rate.
This and similar usages of "exponentially" are extremely widespread now. People talk about exponential increases without any mental model of the rate of growth as a function of time at all—just the woolly idea that something is growing fast.
"The term "exponentially" is often used to convey that a value has taken a big jump in short period of time, but to say that a value has changed exponentially does not necessarily mean that it has grown very much in that particular moment, but rather that the rate at which it grows is described by an exponential function."
https://books.google.ie/books?id=aVovDwAAQBAJ&pg=PA36&lpg=PA...
The battle is lost on this word, as it is with "literally". To the man on the street "exponentially" really just means "a huge amount" now.
The Orwell quote is valid at the heuristic level, but when
a) there's an installed base of people who know the jargon, and
b) the everyday English equivalent takes a lot more words to say the same thing, and
c) something coherent is meant by the jargon that could be so translated if necessary,
then that's exactly when you should use the jargon.
Give me "a^2 + b^2 = c^2" over "the sum of the squares of the lengths of a and b is equal to the square of the length of c".
And in fact approaches to system design that try to make the code size independent of number of bussiness requirements and their possible changes in future lead to exactly the kind of "complexity worship" discussed in TFA (various ad-hoc turing complete VMs that interpret code represented as rows in bunch of relational tables and what not).
FWIW, Big-O itself, even in technical contexts, gets imprecise with e.g. calling hashtables O(1), which is not possible, even under the idealized computer model (instant memory access, etc).
Is there a shorter way of saying "scales proportionally with n" that you would suggest the tech community prefer because of its greater precision?
> What we ended up with was O(n) _classes_ (where n = the number of computation operations).
vs.
> what we ended up with was several classes for each operation.
The second is shorter, and says no more than what is relevant.
It would be wrong in both cases because the context can make clear what a variable or pronoun refers to. If the problem context makes clear what the binding constraint is and you just need to talk about scaling behavior, then it is indeed shorter to say "O(1) rather than O(n)" vs "doesn't depend on the operations rather than being directly proportional".
>>what we ended up with was several classes for each operation.
>The second is shorter, and says no more than what is relevant.
It says less: the O notation is used to indicate that as you add more operations, you will need to add more classes, rather than only needing to add classes when there is logic the operations don't yet implement.
Speaking of brain hemispheres, the left hemisphere doesn't get jokes like the one you made. The right hemisphere handles jokes, metaphors, etc.
Looks like I'm a left-hemisphere-dominant pot calling the kettle black...
My view is that, even in algorithm analysis, Big-O has been overused. Too many people will point out crap like "That is 4 items per entity!" When I point out we only have 50 entities and that 200 is an easily handled number, I just get evil glares.
Wikipedia describes why people use the Strassen algorithm in real-world implementations, despite its inferior asymptotic time complexity [0] :
> unlike the Strassen algorithm, it is not used in practice because it only provides an advantage for matrices so large that they cannot be processed by modern hardware.
[0] https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_a...
Basically, I was hoping some of the optimized methods were competitive nowdays. Spoiler, still not practical. :(
okay, so what do I do when I need to dynamically load an additional command by name from an external .jar ? that's like, the most basic thing Java was meant for.
Also, the answer should not be "large dependency injection framework".
Yes you can load Java classes into the JVM. But why not have a serialization format that is independent of the vagaries and specifities of the JVM?
Still, even if you lean on the JVM, that changes nothing about this particular problem. You don't need command pattern. You don't need a type hierarchy.
Asking first to understand the needs and intention of the application should be a first step anyhow. Diving into the deep-end face first and coding is rarely the correct approach (except maybe a startup racing for market-share).
Reminds me of a blog post I read,
https://www.sebastiansylvan.com/post/the-perils-of-future-co...
At what point does it become over-engineering?
I can't remember a situation where undergeneralized code came back to bite me. I have however seen (and admittedly written some) prematurely generalized code that became messy legacy code.
It's quite possible he was trying to test something else and the answer you gave didn't give him a meaningful answer to that question.
So while your solution may have been fine, it's not something you're going to run into in production code. But that hierarchy+command pattern is probably used in a more appropriate situation. And he wanted to see if you could deal with the pattern itself.
Of course, I could be wrong. He could just be enamored with the overly complex solution because it's more "clever".
Interviews are weird.
I've noticed in myself a tendency towards programming "aesthetics" that just "feel" right. Sometimes that intuition seems to overlap with faculties that help avoid complexity and find elegant solutions... but there's also plenty of times when it's either habit based in a collection of odd assumptions or even apparently arbitrary, and so it's something I've been working to interrogate.
OO approaches specifically have something like three decades of pop-tech discussion as being professional and sophisticated. That's a kind of conditioning that's hard to overcome.
The right hemisphere is visuospatial. Also, the right hemisphere 'sees' time/process.
The left hemisphere puts together contextless symbolic model. So I see where your question comes from: the left hemisphere can do the contextless proofs. In this case (and in many cases) the more complex solution "works" and the left brain can "prove" it works.
But a 'simplicity' proof seems better suited to the right hemisphere (I am no expert here, mind you) .... So in my anecdote the "simpler" solution is the one that I would guess the right brain "sees" as simpler: i.e., it is visually much smaller, also much easier to manipulate (over time), etc.
The book is "The Master and His Emissary".
The point of a 'command pattern' is in effect it's generality, i.e. it implies less coupling in the scenario. The most classic example would be the 'action' that might be passed to a UI button when it's clicked.
So, if the situation calls for command pattern, use it, if not, don't. It's not a matter of 'in your face complexity'.
The notion of doing '3 things with 3 functions' is pedantic: it's obvious. It wouldn't make the basis of a 'question' so it'd be rather pointless to do it.
Maybe there was some confusion as to the point of the question ...
You don't need that at all.
Here is a "proof" of sorts: http://mishadoff.com/blog/clojure-design-patterns/#episode-1...
But either it makes sense to use Command Pattern or not in any given situation and it has nothing to do with complexity really.
I don't see how someone could possibly use a command pattern when simple function would do - that would be beyond pointless. Which makes me believe that there must have been something to the nature of the problem question ...
(Though Lambda's wipe out so many of the simpler use cases of command pattern ...)
What happened was that you showed off how you're better than him at finding a better solution. He's probably a top dog looking for underlings. That's the reality of this world.
99% of developers at Google are doing commodity work. It is are practically IBM or HP now.
It's obvious that you should use a library function if there is a library function.
Sure I can teach constant vs. linear time. But what incentive or reason do I have to spend time teaching these fundamental concepts when I can just hire an engineer who demonstrates the understanding of basic CS at the interview time itself?
Given two candidates with all things equal except that one demonstrates the understanding of CS fundamentals and other does not, why would I want to hire the second person and spend our time teaching him those concepts?
Someone commented on here recently that they would take motivated candidates over knowledgeable candidates. That's one reason. I can easily think of at least two others.
There are balance points here that vary according to all sorts of things.
I mean if there are two candidates who are equally motivated and are more or less equal in all things except that one is strong at CS fundamentals and another isn't, is there a good reason to reject the candidate who is good at CS fundamentals.
In many hiring decisions, I am faced with a similar choice, and I go for the person who is good at CS fundamentals. If two candidates are good in all other ways, then the understanding of CS fundamentals becomes a tie breaker. I see no rationale for selecting the guy who does not demonstrate his strength in CS fundamentals.
A real life task might be implementing a much more complex library over several days or weeks. Asking the person to implement a simple function like shuffle is the closest you can get in the span of an hour.
And very likely that's what really happened. I have been in such interviews and it is sometimes hard to guess whether the interviewer wants the practical answer (calling a library function) from engineering standpoint or the conceptual answer (demonstrating that I can implement the inner workings of the function) from CS standpoint. Often I would just ask a follow up question to clarify exactly at which level of abstraction does the interviewer want my solution to be in?
In situations where I offer a solution that does not match the interviewer's expectation, they clarify the expectations.
Of course, the follow-up question is going to be: Randomize a list that doesn't fit into memory.
She thought of a different method and wrote it out, and they dismissed it again. After that, she said that she didn't know what they wanted, and they condescendingly told her that she should just use the equivalent of a string.reverse() function (I don't remember which language it was).
And on the other side, if the interviewers are looking to identify whether a candidate knows about String.reverse() and the candidate starts writing out some 20 line algorithm on the whiteboard, why would you just sit there and watch instead of being like "actually, we're looking for something else, let me ask it a different way; do you know how to reverse a list using the standard library? We're looking for a one-liner here, not an full algorithm implemented on the board."
If your story is true, then apparently there are at least some interviewers willing to just sit there while a candidate writes out 2 completely different algorithms on a whiteboard without interrupting and communicating what they actually want. Such an interviewer is either socially incompetent or trying to make a fool out of the candidate. Either way it doesn't reflect well on the company.
I think CS interview questions can be hard because sometimes you don't know what sort of "model of computation" are you operating on. Am I on a totally abstract setting where all I have is an abstract machine. Or do I literally have an Intel CPU running Linux? Or am I even higher level than that and can think in terms of the abstraction of python. I think sometimes this is not clarified.
One other time in a different interview, I was asked "how does OS free memory in constant time". Having implemented malloc etc a few times I thought this was a stupid question because it depends on the C library implementation of malloc/free as one could also implement free in O(logn) using tree-like structures. Anyway, said something like "it just clears the pointer in the linked list in sbrk()'d space so that node is inaccessible" which apparently was the "correct" answer.
Which may be the point, they want you to ask "what am I optimising for", to be aware that there's no simple best answer without needing prompting. Also that the optimisation might be at the business level, like time critical implementation, or use of excess resources gleaned from some other part of the corporation.
So instead of telling you they want a one-liner, you saying "what are our constraints; how long have I got to implement it, ...".
I don't get the fuss about interviews. They seem to largely consist of basic programming exercises.
You don't need to know the solution before hand and it is easily intuited on-the-fly. I would never hold it against someone to miss some corner-cases or maybe go for a naive implementation first.
I remember a few years ago, someone was complaining that they had been rejected for not being able to reverse a binary tree even though they had a copious amount of OSS.
The thought process is simple and it's an exercise that students do within the first few weeks of their freshman year.
(struct node (value left right))
(define (reverse-tree root)
(cond
[(equal? root 'EMPTY) root]
[else
(node (node-value root)
(reverse-tree (node-right root))
(reverse-tree (node-left root)))]))
There is one base case: when we reach a leaf.
We want to reverse the left and right subtree at every stage of recursion.
Combine all of this and it's done. I would even be content with a pseudo-code implementation.
Then I come across candidate B who shosw up on time, with a good attitude, takes heat and shares credit, builds people up, isn't afraid to speak their mind and offers a genuine and insightful answer to a problem designed to evaluate whether the candidate understands the underlying concepts.
I am definitely going to hire candidate B.
I've asked similar questions and if I forget all the constraints just a simple, hah that's clever + reframe the problem is a reasonable approach and doesn't come off as being a jerk(keep in mind the candidate is interviewing you/company as well).
I would want the candidate to ask me whether they can use their language's standard library. I would say no/yes depending on how the question is framed.
The absolute worst is when question is framed like 1) - the candidate just makes an unwanted assumption, and then acts outraged or smug when I tell them I expect them to demonstrate that they aren't oblivious to the underlying implementation.
Then we venture off whether they can make improvements/trade-offs if the list becomes 10K, 1MM, 1B, etc. records of N bytes etc.
The point of a library function is that I don't have to worry about how it works, so I can focus on writing things that are not already in libraries!
The point of a library function is so it doesn't have to be re-implemented everywhere, not so you can be oblivious to how it works. If your oblivious to what is going on behind every function call you will write terrible code.
It's always better to demonstrate in an interview that you have a lot of knowledge about whatever that is being asked. It's not just being able to code, but also the ability to explain why and how it works.
It all comes down to the question asked, if they were literally asked to randomize a list then OP gave the correct answer, if they were asked to implement a list randomizer they were wrong. If the interviewee didn't make this explicit they were wrong.
Granted, there should be 1-3 people around who can be tapped for algo knowledge when bottlenecks need to be addressed. However the rest of the dev team just needs to worry about productionizeable code...
That's just proving the point of the person that you replied to, though. If you could give a crap then you obviously care about it.
I do note that that is an atypical experience to extending programs in general, and I don't have the alternate, functional approach to compare it to (the emulator was written in a very old version of Java + Swing).
This case sounds simple, but I still expect the functional approach to save you from a lot of line noise coming from Java class bureaucracy. But in more complex cases, it may be a difference between a couple dozen lines and a couple dozen files.
(although it might sometimes serve well as a configuration language alongside high-performance C++ code like in gamedev...)
The C++/gamedev "data-oriented programming" is about structuring your in a way that's friendly to CPU cache. It turns the programmer-friendly abstractions inside-out in order to make execution as efficient as possible.
The Clojure/Lisp/functional "data-oriented programming" is about building your (programmer-level) abstractions around basic data structures like lists and maps. It's meant to simplify your code and make it more robust.
The two "data-oriented programming" paradigms are pretty much opposites in every way except the name. They both focus on data, but the resulting data structures are vastly different.
Wrote something small like that for a random data generator for an in-house properties tester in Java a short while back. It makes it easier, in my opinion, when you have all the code available in a single screen than if it’s spread out.
O(n) is short hand for here is "what happens when n gets large, but it I'm telling you nothing about when n is small". For example some sorts are O(n log(n)) and some are O(n^2). Naturally for big sorts O(n log(n)) is preferable - but when n is 5 say the O(n^2) will often be faster. People use O(n) precisely because it carries all this unsaid nuance.
In your example you probably have upwards of 30 instructions but possibly hundreds, which sits comfortably in the big side of things. If there was a way to reduce that to an O(1) solution (which implies that if you can add many new instructions without adding lines of code) then you've coded it badly - but we all know that's not possible for a micro controller emulator.
In his example he used O(n), but implied a O(1) solution is possible. The same logic applies. His use of Big-O means for large n, it's possible to add many new cases without adding code. If that is true, his solution is far better than the one wanted by the interviewer. However, it's likely n was small because this code had to be written for a job interview. It's entirely possible he managed to find a enough commonality in those small number of cases to reduce it to being 6 lines, while in the real world problem is indeed O(n). In that case the correct solution depends on the instructions he was given - whether he was asked to solve this particular problem in an optimal fashion, or demonstrate how he would solve the problem in general using those n cases as an illustration.
He doesn't say, so we don't know. His use of O(n) does hint - but the grey beards among us would like to see proof the problem really can be solved with O(1) and the interviewer was demanding an O(n) solution despite that. It does seem like a really weird thing to demand.
On a related note, this is also why I don't like at all the current C#/Java "modern" style of dozens of files with perhaps one or two actual useful lines of code (the bulk of those files being boilerplate fluff and stupid comments like "// @return the return value".) Debugging is especially irritating since stepping through the code becomes substantially nonlinear with deep call stacks.
(A CPU must almost by definition must be simple to implement.)
There are conversational styles you can apply to code that make it simple to test (or in his case, make coherent assertions about) that still keep the methods small but cleave it in a way that you can still track in your working memory.
It’s a style I aspire to, and put a great deal of time and energy into. But reading my old code I can say it’s harder than it sounds and it doesn't sound that easy to begin with.
I don't see why. For example, imagine a graphical editor. GUI events will trigger some actions and those actions will trigger some commands being execute (and placed in an undo buffer etc.). I don't see where that central place should be and why you would need it at all.
And now imagine a large project with different ways to execute commands ("execute", "executeWithUndo", "executeInTransaction",...), people composing new commands from existing ones, etc. Soon grep becomes your best friend. Or just press Ctrl-H or whatever in your favorite Java IDE to see the class hierarchy.
Because in the functional model, you're not abusing the filesystem or inheritance hierarchy to do your bookkeeping.
Also, I said functions, not necessarily anonymous ones. There are ways to solve this use case. You can stash your actions as named public static functions in a "Commands" class. Or you can create one dummy class with @Functional on its only method, that essentially forwards the call. So in your UI handlers, you do button.setOnAction(new Command(someFunctionSomewhere));, instead of having a separate class for every possible action. You can now find all actions by looking up calls to Command c-tor(s), and you can build subtypes of Command as you need some special functionality. Note that this way, you don't commit yourself early to a huge type hierarchy.
In my experience, even in large projects, Command objects tend to sit in the filesystem, wasting space and people's time. But if you're absolutely sure you'll need this pattern, then go ahead and use it. It just shouldn't be the default, go-to way of solving this problem.
It wasn't clear if that was a shift in thinking or just improving ones skills at a particular way of thinking. In other words, no evidence of any loss just a shift in score on a right/left test. The other notable thing he said is that the right brained people who complete an engineering degree were rather likely to spend some time working in the field and then just change to something completely different.
I remember this well because I tested as a dominant right-brain thinker and have always had this odd feeling that I might want to just stop and do something completely different. Fortunately I've been able to find ways to use creativity in engineering :-)
1: https://www.health.harvard.edu/blog/right-brainleft-brain-ri...
Yes, there really is, but the way it's described in American media is a load of crap. I don't think anybody published actual real science about this in English.
The logical/creative dichotomy is bullshit based on fake science.
In reality the left brain (dominant for most people) prefers sequential, step-by-step thinking, while the right brain prefers simultaneous and spatial thinking.
Math professionals are actually biased towards right brain people. (It's hard to get anywhere in real math using step-by-step algorithms.)
But can it really be simultaneous (at the same time)? I mean there's still a "micro sequence" involved. it seems that the real skill here is context switching, pattern recognition and applying patterns to different contexts.
When I have to think about something hard I try to do it without words in my mind. I believe this makes it easier for the brain to subconsciously work on those, without having to "translate" it first.
However I still use "pictures" in my mind. I wonder if it's possible to actively think about something without using things linked to our physical senses (hearing, seeing, touch,..).
Sounds like intuition based on experience, where a new situation is compared against your experience, with the brain filling in the gaps.
>When I have to think about something hard I try to do it without words in my mind.
Hard in the sense of difficult to solve and unfamiliar, or in the sense of the level of focus or concentration?
For example, I know a heavily right-brained person, and when he reads a book, instead of the intuitive sequential pattern (even page-odd page-even-odd-even-odd) he may read it in a zig-zag (even-even-odd-odd-even-even).
The analogy is synchronous processing (left brain) vs multithreaded (right brain).
On the other hand, right-brained people seem to have problems doing 'obvious' things like finishing chores in order, telling left from right or positioning something on a blank page.
>> In reality the left brain (dominant for most people) prefers sequential, step-by-step thinking, while the right brain prefers simultaneous and spatial thinking.
I've never seen anyone make such a direct contradiction so clearly ;-) You define left-brain thinking the same way the "bullshit" does, and then you define the right brain in a way that reflects creativity - at least to me.
Analogously, code is cost, infra is cost, only our sold product is revenue.
A thing I did not understand for a long time is the effect this has on value. Something that produces a low value over a long time may well have been value-negative over the period because of the hidden carrying cost.
Also there may be a balance between "some hierarchy in the code so it can evolve" and adhoc code that just works for the current problem. Only having to write a new class for a new feature because everything else is handed smoothly is a bliss.
For example I could write a massive list comprehension in Python in one line that could solve my problem, but it would be much more difficult to read and would be a pain to modify. I'd much rather work with the 10 lines of code it would take to actually spell it out.
There are some other aspects to this idea of code economy: correctness and reusability. This definitely depends on the programming language.
In Java(or oop for that matter) you can get correctness but to get reusability you need a whole hierarchy of classes, lots of added code/cost.
In Python you get more reusability out of the box but to get correctness you need to add swaths of unit tests, lots of cost.
In contrast with a functional language such as Haskell you get both with less effort, fp plus a rich type system reduce code enourmosly.
I wish Rust would have gone with an ML syntax though, in particular the Hindley-Milner type signatures allow me to reason better about code reusability, the "fn" and braces just adds noise to me, too much Haskell I guess.
He told me he took a ukulele to the interview. Each time when interviewer asked him a question, he would play ukulele untill he found the answer.
He got the job even when his interviews went wrong.
His job offer arrived months after an interview because he was the only guy who the interviewer could remember.
Often you hire for a spot in a company, after sometime that person leaves and management asks you to name any other person and when you try to recall last interviewee batch, the ukulele guy is only one the interviewer is able to remember.
I didn't have enough money to buy new clothes at the time, so instead I've picked the shoes with the largest amount of holes and would never do any grooming before hands. I would come stoned to the interview. So the gist: instead of trying to appear on the lower mid end, I went all the way through to the lowest low end.
Thinking retrospectively I think that's pretty much the reason why I barely received any negative responses unless I had miserably failed at the technical side of things (e.g. I didn't know how to use generators in Python at the time and the whole list of questions would be about them and their syntax).
I have to admit that in this scenario I was quite good at the technical side of things. But the general philosophy was that if I don't try to appear too good - people will assume that I was better. Put humility and definite sprinkle of character on top of that.
I have two monitors on my desk. If I am stuck on anything I drag my emacs from one side to the other and look at the same code there. It works pretty well!
If someone has invested a lot of time trying to understand OO and design patterns like this, and had the feeling of accomplishment when it all comes together, then the are going to want to apply them where they can.
I think all that stuff is invaluable knowledge but people shouldn’t fall into the trap of doing something one way just because they have invested the most time in learning how to do it that way.
Also, many software professionals probably find simplicity boring. Simplicity should be taken more seriously as a goal in software development. Sigh.
It's like they are trying to prove that they are really, really smart without anyone asking them to prove it. But here is my challenge: anyone can overcomplicate anything, but if one is really smart, make a complex thing simple.
Complexity creates opportunities for fiefs, little areas only one person understands.
I think we don't recognize what drives a good portion of people, in the tech industry. People pretend to, and are often expected to understand more than they do. Complexity helps us when reporting or manage (up and down). It's harder to grill you, challenge you or commit you to things when everything is complex. Complex requires more people, which is the main scorecard for corporate success.
It's also just unintuitive that simple is harder and better than complicated. Even if you get it, someone else won't.
I've come to the same one. I think it explains a great deal of what we see. Even in the hiring process and such. It's actually very primitive at its core. Who's part of who's tribe and who is jockeying for status in that tribe.
Coming in, I was told that the specs were difficult, the tech had to be nailed, and they had already tried and failed.
I took a week getting used to the customer, stack, and team. Digging through what they had? They needed two lines of code. It was the same exact situation.
That was one of the more difficult consulting assignments I had. It took me quite a while to gently get the lead architect on-board. I had to position it so it looked like his idea. If I would have pushed hard, he would have pushed back...perhaps getting rid of me and hiring more of a team player.
I've seen this a lot, in a lot of teams, and I think it has its roots in OO/Platonism, although it happens all over. I think the key things most of these teams are missing is that all structure is derivative. That is, there's not a line of code that's in your app that isn't in there for a reason. You're either really careful about that reason....or you add stuff because it looks good or you might need it someday, you can "imagine" how it might be helpful, the guy sitting next to you said you had to have it, and so on.
If you do it differently or with a different language, there must be some negativity to latch onto in order to justify the way they did it. The simple ability to say "Oh...cool! Hey Bob, come look at what Jim figured out!" is a rare find.
First to go was the distinction between accessible and non-accessible seating, partially obscured views, basically any extra info about seats except their identity. Okay, great, that simplifies things.
Likewise any idea of redundancy or failover. "Just let the process restart. You should be able to ensure no failures anyway." Okay, that's kind of opposite to how I've thought in the past, but I'll keep my mouth shut and roll with it.
Just to check, I asked if I should support a concept of buying tickets for assigned seats. Nope, no assigned seating. Okay... so are there different price classes for each event? "No, the tickets are all the same!" Okay. Each event has a single price and a single pool of tickets.
But to sell the ticket the user has to be able to buy it, right? Nope. No payment process. Great! I said. So there's no need to put a hold on a ticket while the user has it in his cart. The interviewer looked like he was going to burst a blood vessel.
Then I said, okay, at minimum I think we need to record who we've sold tickets to so we can send them tickets or in some way ensure that the right people can be let into the concert and people who haven't bought tickets can be turned away. Um... right? "No, don't bother. If they say they bought a ticket, they did." Okay. Moving on.
You wouldn't think the requirements could get much simpler than that, but you would be wrong. It turns out that the system didn't need to know when a concert was happening so it could stop selling tickets at some point. Ticket sales for a concert would go on forever. What should I do if the concert sells out? "Don't worry about that. It won't happen." At this point, I wanted to scream, "Your honor, permission to treat the witness as hostile!"
It also turned out there was no need to track different events or venues. There's only one concert, it has an unlimited capacity, and it never happens.
In the end it emerged that he just wanted a simple TCP service that listened on a port and served a random long integer (a "ticket") to anyone who connected without repeating the same value, unless it got rebooted, in which case it was okay to potentially serve some of the same numbers that it served during its last lifetime, as long as it didn't exhibit any patterns that might enable an attacker to predict upcoming values.
What #%$%ing similarity does that have to selling concert tickets? I mean, in the phrase "sell" "concert" "tickets" "online" three out of the four words were completely irrelevant and misleading. I'm guessing it was just what he was working on that week, and in his system the values were called "tickets" so he threw out "let's sell concert tickets" in an interview hoping I would solve his problem for him. We were out of time, so we didn't have time to discuss it. I didn't get an offer and wouldn't have accepted one.
It's really fascinating to study modern CPU design. Modern high-performance CPUs are horrendously complex, with the very notion of superscalar architectures and pipelining resulting in guaranteed complexity explosion. Yet, as far as we know, there is no way around this. You need pipelining and superscalar execution to get adequate instruction-level parallelism in real-world code. Unless you want to use microcontrollers for everything, that complexity must exist.
Compilers are another example. Many people who go to implement compilers read the Dragon Book and think that all the complexity in GCC and LLVM is needless bloat. Then they quickly discover that they can't compete with them in performance.
It is of course desirable to avoid complexity where possible. But all too often the response that we as engineers have to discovering that difficult problems require complex solutions is to become defensive, stick our head in the sand, and stand by our simple "solutions". This is how we ended up with crufty Unix APIs, the security problems of C and C++, the pain of shell scripts, and so forth. We need to learn how to accept when complexity is necessary and focus on managing that complexity.
Obviously that is a generalization and there will be exceptions, but as someone else said in this thread "code is a liability not an asset".
But I can’t agree with this observation. There are a lot of smart people who are bored and solve a problem we never had to keep from going nuts, but the solution is so complicated it drags everybody else down.
But there are also a lot of people out there who think that if you pretend hard enough that our problems are simple, then a simple solution can be used.
If you oversimplify the problem enough, everything looks easy.
I've been working outside of software recently, and noticed to my delight that this Law hasn't applied at all. In one case, I was hired for 3.5 days of work, and we got finished after 2.5 days so we were sent home early -- nobody was dragging their feet to make it last 3.5 days. In other cases (more common), we've temporarily had too many people for the job at hand, so the team lead said "Just wait", and we do nothing until there's more work ready for us.
Why have I never heard of any software team ever saying "There's nothing for you to do right now, so just wait"? My first thought was the endless supply of bugs, but that can't be right, because I've never heard of a team lead saying "We have no work for you today so go fix bugs for a while", either.
It really does seem like every software team manager thinks that the proper amount of complexity in a system is perennially $(current_complexity + 1). The cases where program complexity approaches a constant asymptote (like Redis and perhaps SQLite) are so rare as to be notable. They're also frequently mentioned as being developer favorites.
Maybe the field just needs another 50 years to mature.
This thought is pretty obvious from my humble POV. Do we know of any field this extensive that matured (by any reasonable definition of "mature") in less than 100 years?
Civil engineering took thousands of years to get to a point where it's not taken for granted anymore that the construction of a large building will cost the life of some construction workers.
(will people still use emacs?)
For the most part, I think major new inventions of the 20th century have taken much less than 100 years to mature. We have science and engineering now, so being able to apply them to new fields is generally feasible.
I'd flip it around: what fields still have not matured yet? In what fields, since the dawn of science and modern engineering, is the median project still a failure? I'm hard pressed to think of any outside of software.
Social sciences (the replication crisis).
Also, observing that software engineering is not really doing much worse than social sciences at producing results certainly doesn't make me feel very good about software engineering.
In addition to Parkinson's Law, there's this one: "A poem is never finished; it is only abandoned."
There's never nothing to do, because we can always improve things. And to a business or to a manager, "just wait" costs about as much as "work on something of little importance" but provides less benefit. It might be different if programmers were all on zero-hour contracts.
(OTOH... Firing engineers could work. I wonder why sites that seem "done" don't decrease their payrolls. Pride?)
Every manager wants a bigger team and impactful projects for their resume. This bubbles up to the top, where rarefied execs don't see a reason to go to war with their own org and possibly lose. As long as it's still profitable, everyone's doing fine.
When Tom Wolfe was finishing up a new novel, adding more writers wouldn't improve the story. When a cancer patient is undergoing chemo, adding more physicians this afternoon won't improve the outcome. Adding cooks to the kitchen of my favorite noodle shop isn't going to improve the noodles one bit. Adding more actors to a film's shoot schedule isn't going to make it go any faster.
In all these cases, even if you gave me 100 more skilled people for free for a week, I'd tell you that we don't need them, and it would actually hurt us for them to participate. Changing the plan or going off-plan has a real cost. Mature fields like civil engineering have a great track record because they don't just let extra people make contributions at any point in the process.
BTW, according to Wikiqote, the correct (and unabridged) quote is: "A work is never completed except by some accident such as weariness, satisfaction, the need to deliver, or death: for, in relation to who or what is making it, it can only be one stage in a series of inner transformations."
For programming to be a mature field, "the need to deliver" must be a necessary component, and "satisfaction" should be the goal, not merely an "accident". We can't utilize the process of a 19th century French poet and expect to get results like 21st century civil engineers. This was a guy who (according to his Wikipedia article), "around 1898, he quit writing altogether, publishing not a word for nearly twenty years."
Where I currently work we're using Azure to do a shitload of computations. At the same time, many modules don't even bother to throw away intermediate calculations. They literally have a giant array of them, and save every result, for each step, even if it's not necessary.
But hey, the project is seen as a huge success, because it's so complex.
Even though I do database / web app level programming most of the time, I usually find Linus' quote very relevant:
"Bad programmers worry about the code. Good programmers worry about data structures and their relationships."
I divide my officers into four groups. There are clever, diligent,
stupid, and lazy officers. Usually two characteristics are combined. Some
are clever and diligent – their place is the General Staff. The next lot
are stupid and lazy – they make up 90 percent of every army and are
suited to routine duties. Anyone who is both clever and lazy is qualified
for the highest leadership duties, because he possesses the intellectual
clarity and the composure necessary for difficult decisions. One must
beware of anyone who is stupid and diligent – he must not be entrusted
with any responsibility because he will always cause only mischief.
Stupid and lazy developers can be entrusted to come up with simple solutions to simple problems.
Clever and lazy developers are able to find simple solutions to complex problems, a very desirable trait.
But stupid and diligent developers, given the chance, manage to implement complex solutions to simple problems!
I have been saying for many years that overengineering is the plague of modern software. Almost everything seems far more complex than it needs to be.
Take JSON API, a standardization of HATEOAS / REST principles around JSON and HTTP, for example. If you just naively walk up to it you think to yourself:
> Wait, what? Can't we just return the simple data we know we want? Why complicate this all with relationships, links, and meta data?
But after a while you realize that 90% of what you're doing could be abstracted if only you had a predictable output. So EmberData comes along and you write a Rails backend that has the nice advantage of not needing to worry about HTML (outside of OAuth / emails, anyway) and you harmonize your API. You use Ember Fastboot for slow clients and call it a day.
Someone else looking at what you've done may say:
> Wait, what? Why don't I just create static HTML pages instead of using your over complicated service?
And they're not completely off-base. In fact it's what I do for my own personal site. It's just that the context of their situation has different tradeoffs.
The same thing is true for a lot of software. Excel is "over-engineered" for most people. So is HTML. So is Unix. But in general what happens is that the person with the most complex requirements usually wins because they usually have the fattest wallet and everyone else papers over the complexity with abstractions or uses something less complex that meets their needs.
The Unix Hater’s Handbook [1] is a pretty good read (or skim). It’s healthy to remind oneself that, even though unix-likes are a savior from the deeply unpleasant alternatives, warts are present.
[1]: https://homes.cs.washington.edu/~weise/unix-haters.html
This is what I feel about the current JavaScript ecosystem. I feel like web development became popular so programmers from other disciplines (C, Java, etc.) jumped in and found it to be too simple so they've hijacked the ship and created a new JS ecosystem that is as complex as their old environments.
No, no, it’s that webdevs looked at other areas of programming and thought “we have to be complicated as them in order to be taken seriously”.
Desktop development is blessedly simple, in contrast. The problem is that desktop development APIs and UI toolkits have been neglected for many years in favor of web and mobile, so now desktop development is also a fractured mess because much of it is outdated or hasn't kept up with current graphic design and UI standards. But, at its core, desktop development doesn't struggle with oddball concepts like promises and other bizarre features that were introduced to overcome issues with the overall design of the language/environment.
From my experience, complexity to me means how many combinations of states are possible a program, how many paths one can take at any point, how many side effects are possible... etc. An abstraction is supposed to manage complexity, if it doesn't, that doesn't necessarily mean the abstraction is too complex, but that the abstraction may just not fit the problem or cover all the edge cases. Thus a leaky abstraction.
At least, in my experience.
At large companies the output of architects is generally things like directives, white papers, and other agglomerations of words. They are expected to be smart. Can non-technical executives judge the actual smartness of the work? Not really. So architects often get judged by sounding smart. Confidence. Complexity. Negative judgment. Performing intellectual toughness.
That's the opposite of what I really want in a technical leader, whose job is generally best done with humility and subtlety. The best technical leaders I've worked with are quiet and make a lot of small interventions that add up to big long-term results. But that's rarely what executives are impressed by.
A diligent person may be someone where creating more work isn't a blocker to an acceptable solution, whereas a lazy person has reduction of work as a requirement for theirs.
So a lazy person might work more at their solution, so long as it helps them avoid more work in the future.
Or it could just be that people have a certain mental budget for maximum complexity, and they'll try and make sure they spend it all in the belief that it will cover more uses.
Or it could be that complex systems tend to stick around because they are far more immune to random management changes because everyone goes "oooh, we'd better not touch that". Simple systems might becomes the victim of their own ease and get subsumed by a more complex monster.
Dude. That "big, ball of gas" "just" spontaneously ignited due to gravitational forces and supports life on this planet. That's fucking wild. It's way more cool that it's simple. You try squeezing air hard enough to make it explode.
They are quite not the same since one dimension denotes the complexity of the thing whereas the other denotes the complexity of the act of building, maintaining and evolving the thing in question.
We should always strive for easiness, however complexity shouldn't always be avoided. Quite to the contrary, staying away from complexity locally often leads to that complexity being sprayed on a global level, and that's when it turns into something complicated.
I have too often seen "complex" code, i.e. code that works on "complicated" datastructures such as trees and graphs be discarded in favor of solving the problem "simply" and directly, which means by disseminating the problem's logic accross the codebase and with multiple, gradual bugfixes because the problem being intrasequely complex is underspecified and cannot be tested in isolation of the system.
Of course the same people that are baffled by "complex code" and think simplicity sums up to looking away and not anticipating future needs, are the same who advocate it. Actually, they like good design and typography, focus more on indentation than datastructures and generally have a taste for nitpicking with as many subtle details as they can come up with, not seeing past the filter of their own opinion about what simplicity and complexity entail, and of course that makes the social process of building code a slow nightmare that does not converge.
I sound harsh I know. What I want to point out here is that by denigrating complexity in favor of simplicy, we may get rid of medieval savants, but we open the door to plain idiots disguised as zen masters.
Wanna hear something harsh? Most programmers I ever worked with for 17 years of career do not deserve the right to touch a keyboard; they should work on farms. That would be [somewhat] harsh.
You are on point with everything you said. Ego, strong opinion enabled by a secure job position where no amount of professional failure will ever get you booted (because you are isolated from the business outcomes), echo chambers of fellow bros who think like you, and plain old fear of change is what drives most humans -- and programmers aren't an exception.
IMO it's high time some formal certification and legal liability to be introduced to our profession. Also, pick 5 imperative and 5 functional languages and make them "official". People love their religious language wars but it has to stop at some point because billions of bucks are being wasted on 25-year old egos.
As for a formal certification, here in France "engineer" is a state-certified status. There is no legal liability I can think of (there might be some when you're engineering bridges, but I have not heard of something similar concerning software). To be frank, this status is bullshit. At work I'm using a payment API from a local shop. The CTO comes from a top rated school (the french Calltech), yet their API does not ensure the reception of async notifications like if communication failures wasn't the crucial characteristic of a computer network. In addition to that, they are unable to provide a dedicated infrastructure to their big customers and ended up implementing a rate-limiting system on top of their absolutely business critical payment API (and boasting about it in blog posts). Last time I checked my company experienced a 80% drop rate in payments due to errors or the rate-limit being exceeded.
I say "last time" because I haven't been to work in months, my sick leave being extend month after month by my psychiatrist. I worked a lot these past couple years. In 2017, I think I averaged 70h per week. Actually, I have no idea. I just know I worked a lot of 80/90h long weeks that year. What's certain is that my hourly pay rate dropped under the legal minimum, which isn't surprising since I have the lowest salary in the team (a little less than 40k€/y). Everybody comes to me to fix their shitty problems, and when I push PRs to avoid them code aesthetics and "simplification" takes over in the code review and it takes foreeeever, and I'm never guarded against bug i may have introduced, only indentation and inessential shit like that, so I have to review my code on my own, and these PR never see the light of day and then I'm suddenly considered the master of unfinished work, almost in a self-satisfied tone by people who work almost twice as less as I do and earn almost twice as much. Also I'm not allowed to work remotely (but everybody else in the team is) and I have to be at work before everybody else, and the fact I sometimes come two hours earlier is never taken into account. Oh and my sleep and medication is monitored and I have been subject to very condescending and just borderline illegal remarks about it.
I'm just insanely butthurt to be honest. I could just as well start complaining about being "talked to like a dog" (these are not my words, but what two persons independently told me about the way I was being treated in the team).
And you know what, they are all very concerned about keeping the project's complexity low. Coming from a Clojure background I exactly know what this means and what's wrong with their approach, i.e. they see complexity as opposed to simplicity, but what's pertinent is to actually oppose it to easiness. See Rich Hickey (Clojure's inventor) "Simple made Easy" seminal talk for an in-depth overview of software engineering from this perspective.
To give a concrete example of what this misunderstanding leads to, let me compare the simplicity of markdown, praised by many minimalism hipsters and whose main force is to make us forget its limitations, fascinated as we are by its fixed-width typographical beauty, VS the simplicity of acknowledging that mixing text with control characters (the old-fashioned unix way) is not a sound way to build complex things at a large scale.
Anyway, since then, I've become extremely wary of those who advocate simplicity just like an early or "private" christian would dislike Church (or what it has become).
I'm now getting back to working on extending Clojure's compiler to experiment with the idea of integrating the notion of IDE and editor directly at the language level, but deep down I'm not sure I can continue with this kind of bullshit career, where ability to handle complexity is not just disregarded, but is punished.
Usually I would tell you "leave your job" but me being from Eastern Europe and not in the cozy enabling environments of most of Europe... I realize you might not have the choice.
I don't know your situation so what I can recommend is probably misguided. Still, here goes, in case it can help you (and in case it's not obvious):
1. Leave the job if you can. You already are not working and are apparently still collecting some kind of paycheck. I am not sure any amount of medication of psychiatrists will help you overcome the fear of eventually coming back to that hell you have been working in. How do you feel about that prospect?
2. Take a creative break. It seems you already are doing something along these lines by working on things outside your immediate duties. Do they fulfill you? If not, definitely just stop. I lied on my ass for 2 months before actually going back out there and starting to get stuff done. Sometimes you just need it. Also, programming in your spare time isn't always relaxing.
3. Talk to people about your situation -- not only to people who are paid to listen to you though. That's a very broad advice so apply it as you like. ;)
It's OK to be butthurt/salty in this situation. We aren't angels or saints, these things can and do get to us. And you have been wronged, many times.
I faced the same. I actually come across as quite manly and assertive to a lot of people but that's mostly my appearance and my attitude of gettings things done without tarrying on petty differences. I strive to never argue emotionally or engage in yelling competitions with people and they very often mistake that for me being a pushover. What I usually do is: if somebody is becoming an obstacle, I just go to a higher manager and talk to them about it. If they don't care or have the wrong idea then I just reduce my efforts in the said work significantly and just move from paycheck to paycheck. Done that many times and now I am suffering financially for months -- very severely! -- because I want to choose a workplace where things are not like that. It seems the mythical "cultural fit" is not BS after all...
What I take from your gently shared pain is that you are not a conflicting persona. That's okay. You don't have to be. Do your best to find an environment where you don't have to fight daily with people. They do exist.
There are many engineers though that like to over-complicate and add complexity because it makes them look smart or they think they are expected to create complexity.
Complexity through simple parts is ok, but overall the job of an engineer is to take complexity and break it down to simplicity and simple parts.
In the game dev world for instance, Unreal used to be needlessly complex, still is a bit, then Unity came along and made things simple, so Unreal then looked to make things simple.
Or in the web dev world, a framework might abstract away the underlying standards and add complexity on top to seem simple, but actually make the domain more complex with more to learn and push for developer lock in to the framework over basic standards and simplicity. The first version of .NET with WebForms was an example of this, other web frameworks can be seen as this as well.
Lots of engineering people like to look smart by managing complexity but it doesn't always need to be so complex. Sometimes time pressure can create complexity and thus technical debt due to the complexity. Some simplification is misguided, a one liner that isn't understood 6 months later is not simplifying.
An engineer that takes something simple and makes it more complex for no reason other than job security or to look smart is the worst kind of engineer.
I've been saying for years that this concept, that abstraction has a cost and isn't just a free simplification, is one of the primary misunderstandings that is holding the web ecosystem back. If every JS (and Python, and PHP etc) dev understood this, we'd be in a much better place, building much better software.
I don't think most people set out do do this, most complexity comes from attempting to simplify things. A develop will see two similar bit's of code and try to stuff the commonality into a base class. In their minds it's simplified because there is less code but then when someone else picks up the maintenance it's more complex because now the logic is distributed. Then the requirements of each diverge slowly and over time each simple change is hacked into more code paths of this distributed logic. Eventually most simple changes take time because you have to be sure you're not breaking other potential code paths.
On a more macro scale, complicated architecture is another symptom of this over optimistic pattern matching that humans are susceptible to.
It's a very classic problem and it's something that code reviews and pull requests for anything you do are such a good idea -- provided that your team is not a total echo chamber of course.
This bias isn't necessarily intentional: human nature (competitive evolution) just naturally pushes us to interpret the world in a way that makes ourselves as valuable as possible.
In IT, complex solutions that we create or learn keep out competition. "Only Bob knows how to fix this monstrosity" is a common pattern.
Simplicity should be added to the employee evaluation process. This includes parsimony in both features (YAGNI) and in how the features are coded.
Further, avoid "eye candy" UI gimmicks that add complexity and fragility. End-users often love them, but they are often a longer-term maintenance headache. Beauty ain't free.
For the surgeon example, the simpler explanation is that, given a particular ailment, they just know only the surgical treatment, not the alternative drug-based therapy.
Or if they know, they know it way back in their head and don't think of it unless explicitly prompted, whereas the surgical procedure is probably recent experience.
I could also see this apply to IT. Given some problem, maybe there would be a simpler solution if I implemented this particular problem in, say, Python, but I'll implement it in Go because I work with Go a lot and all its idioms are much more salient in my mind.
What I see is the opposite; companies that would do perfectly well with a server or three, and a simple monolith, trying to instead use Kubernetes and an Amazon service salad because that's what Amazon and Google say to do.
Unshared microservices are a waste of time and code. Make sure your org is actually ready for sharing, because it does add inter-work-group dependencies that the management and team structure must be comfortable with. I've seen "build it first and they will come" fall really flat.
In my experience, complexity in the code often reveals not only technical problems, but frequently also points to product and business issues. Exploding complexity and long iteration cycles are often a consequence of bad business decisions. Looking at the points of exploding complexity in the code can sometimes help identify these issues.
The key is to make the disciplines work together. Business and product decisions should not trickle "down" to developers. Instead, there should be a working feedback loop. For that to be achieved people need to be team players and talk to each other frequently. It also helps to have a 10% or so generalist in each team member's head to guarantee a shared understanding of a high level perspective.
I know, I know — this is all a given in agile methodologies. But in reality, it's unfortunately rarely executed that way.
In some ecosystems it is more consistently complex (Java?) but others just have many approaches with different complexity levels for you to choose from (node?).
I do think JS has a lot more variation in complexity than other ecosystems though.
Too many people these days manage complexity through ever more complex sets of tools instead of simply getting rid of it. They use complexity as an excuse to introduce more complexity and it just keeps growing.
One important thing I discovered for myself is the practice of starting with minimum viable representation of the problem. If I work on a system, I try to write down what it needs to keep track of. If I work on a component, I start with the minimal configuration it requires to instantiate. This helps to avoid anchoring myself with available tools and "common" solutions.
In other words, rather than imagining the complexity wrangler inventing layers of simplifying abstractions, they imagine him/her to just have a radically extended short-term memory or something on those lines.
The article points out the case of architects worshipping complexity in some cases by essentially over-engineering. I think this is a mistaken explanation. I think it's more of a fear-driven thing: the architect is like, "Oh shit, this problem I'm gonna be dealing with is gonna be really complicated; I'd better throw everything I've got at it!"
(Popular books on OOP and code style definitely do not help keep things simple.)
Me: "Wasn't the earlier version 30k SLOC?"
Management: "Yes."
Me: "So what was added?"
Managment: "You can load software via FTP now, and there's an embedded HTTP server!" (what they said)
Management: "270k SLOC!" (what I heard)
Me: "But now we have to manage that, maintain it. What capabilities does this give us?" (answer: none, the old way of loading software was only slightly slower, and because this is embedded you always needed physical access anyways)
Management: "But our programmers can handle it."
Me: "Today, because they wrote it. J over there is leaving to be a project manager on something else. Who will handle this after he's gone? Oh, and you actually started putting the maintenance on a group of contractors who weren't involved in the development."
Management: "We'll rewrite it again!" (what they'd have said if they hadn't walked away)
In my experience everyone does, but the difference between the complexity-worshippers and (for lack of a better term) simplicity-worshippers is that the former like to take a simple problem and blow it up with complexity, while the latter like to take a complex problem and make it simple. In other words, some people like the challenge of making things more complex, while others like the challenge of making things simpler.
To make a concrete example, complexity-worship would be something like using half a dozen different new web frameworks/languages/libraries to set up a personal blog, while simplicity-worship would be more like an H.264 encoder in a single file[1] or a self-compiling C-subset JIT[2].
One thing that seems somewhat obvious about the complexity-simplicity divide, and which could also explain the prevalance of complexity-worship, is that it is relatively speaking much easier to generate complexity than to reduce it, and unfortunately it seems the majority of developers just don't have the skill to reduce complexity. It's easy to glue a bunch of existing code together without really understanding how it all works; it's much harder to take a spec that dozens of experts have worked on for many years, and condense it into a concise implementation.
Maybe we have different definitions of "worship", but this seems like accidental complexity, not reverence. If anything, people tend to worship simplicity.
Efficiency is another common excuse offered for gratuitous complexity.
Simplicity is hard.
There's also 'simpler for me' vs holistically 'simpler'. A great example of this is a junior engineer comes into company, sees existing solutions and doesn't fully grok them, and jumps to 'lets throw this out and implement something better'. This is pretty common and the underlying motivation is indeed to make things simpler! But actually only from that individual's perspective.
Local incentives matter too. In any social system there's an incentive to "flexing your muscles" and showing off our abilities, since that helps to build prestige and respect.
When coding moves away from "impact", and "scaling" and "agility" and back to old fuddy-duddy engineering staples like better up-front design, partitioning problems properly and solving them in toto before shipping (i.e. doing it right the first time), allocating the right problems to the right people--difficult to do in a team with junior members that need space to grow and experts that can dash things out in minutes--then we'll hopefully be sober enough to see through cargo cult practices down to what works.
Social posturing is certainly largely to blame for over-complications, but I think more deeply, a lot of people hate to let go, to be told that the very thing they were responsible for, that they worked hours/days/... on isn't necessary any more, or perhaps never was. And then in the other direction, there are what I refer to as "exponential requirements", for lack of a better term. You buy a large bag of coffee that you could simply use as you go and store in the fridge. Or, you could have a jar with some of the coffee in it, and the rest of the bag still in the fridge. Then a plate to put the jar on. Then a new shelf to put those plates and those jars. By the time you've followed this track long enough, you're looking at buying a new house, even though your essential requirements haven't changed from the beginning.
Worship of the complex, or at least a particular focus on it, may be built in by Chuck Darwin.
You might say that in both cases there is a longing for a dissolution of self. Or death even.
Is the machine-lover a death-lover?
Fast forward to 2018 and stuff like timing chains are buried under a mountain of engineering sins. I have a honda with the oil pan, air conditioner, exhaust, and oil drained and in various states of disassembly and for what? a chain.
As i get older, I start to worship it. its inevitable that engines come with a spinal cord of three or four dozen sensors you need to carefully disconnect before doing anything. But what I cant abide by is the inclusion of demonstrably poor quality parts. Companies hope the engineering complexity will just baffle people into accepting the idea of disposable plastic valve covers and plastic water pumps, but I remember when these things didnt get thrown out.
I saw this part:
https://youtu.be/NZAWeR46z_Q?t=238
of this review the other day and began to wonder: how prevalent is this really?
It is interesting, because I've seen Conway's law used as a weapon against modularizing something. I think the key is that you need to weigh what you are getting out of the modules. Even if it is, in some ways, more complicated, if you can more meaningfully delegate ownership of parts of the problem out because of the the modularization, it is a good chance it is still worth doing.
I don't think I can give a brief example. In my case, we had three teams, rather than trying to put everything on one team, I was trying to ask for what the three major components of the system were. If we couldn't agree on that, it seemed unlikely we were using three teams well.
Here's the related HN discussion link: https://news.ycombinator.com/item?id=18093158
The old fallback of functional decomposition into parts that can be reasoned about and developed independently, is problematic due to the need to constantly rework and extend the hierarchical decomposition as more about the domain under consideration is specified.
How to predictably build a payroll system, or air-traffic control system or a much simpler (at first glance) yet inevitably complex real-world system, remains out of reach.
Surely we don't worship complexity, but outside of computer science, just still don't know how to deal with it?
You can start with a trivial problem and grow it into a system so complex it's hard to even begin grasping what it actually does and how the software actually works.
If you start with a complex problem the only viable path is down to less complexity; often we won't reach simplicity but at least we're somewhat equipped to handle the underlying complications.
https://en.wikipedia.org/wiki/Variety_(cybernetics)#Law_of_R...
Edit: The paper must be this: https://www.tandfonline.com/doi/abs/10.1080/0020772700892022...
But I'm still interested if there's a better introduction.
Edit #2: Looks like Ashby's 1956 book around page 207 has an argument based on information entropy. I haven't read it, just skimmed it. I'll look more closely later. You can download the book here: http://pespmc1.vub.ac.be/ASHBBOOK.html
However the theorems you are talking about have useful, well studied analogs in the field of control theory.
For example the Law of Requisite Variety is very much related to the concept of under-actuation (https://en.wikipedia.org/wiki/Underactuation).
The good regulator theorem becomes the internal model principle. https://en.wikipedia.org/wiki/Internal_model_(motor_control)
I think you'll have much better luck finding good introductions to these topics in control theory than in trying to learn more about cybernetics.
Intuitively, the theorem says that to control a system exactly you have to see any external disturbances coming in advance, predict their effect (which requires knowing the entire state of the system) and generate control inputs to counteract them. But practical controllers usually don't bother with this, and just apply negative feedback based on the observed state.
For instance, to steer a ship exactly you need to know all the waves and wind coming at it in the future. But to steer it adequately, you can just look at the heading and apply a low-pass filter to generate rudder commands. (The low-pass filter might require state consisting of 2-3 numbers).
Or to control data transmission over the internet exactly you need to know the size and timing of everyone else's data packets, but you can do an adequate job with TCP just keeping track of ACKed packets.
"How do you know if you're engineering if you're not overengineering?"
Now I don't direct this at Google specifically. This is more about the mental traps we, as engineers, can easily fall prey to.
Take interviewing and the FizzBuzz thing that was popularized (if it didn't originate with) Joel Spolsky. The beauty of FizzBuzz is that it's a quick and great _negative_ signal. This doesn't mean that if you ace it, you're a superb engineer but it does mean if you fail it, you're almost certainly not. Hiring (from the employer's perspective) is a numbers game. Every candidate costs you time (giving interviews, writing feedback, etc) and is a huge opportunity cost (other work that could be done, other candidates that could've been interviewed) so the goal is the filter out the "no"s as quickly as possible.
So the engineer trap when faced with something like FizzBuzz is to think "oh wait, that's too simple!" and they go on to change the problem to something you'll pass if you've heard of the particular obscure algorithm and will probably fail if you haven't. It's a easy fallacy to fall for. Harder is better. However now you're not optimizing to cull early candidates. Now you've designed a test that optimizes for people who do well coding under pressure (with some degree of luck) on a whiteboard. That is completely different from the original intent and (IMHO) almost entirely useless as a hiring signal.
Another example: another thing I was fond of saying at Google was that there was a hierarchy of engineers:
- At the top level (in their minds) were the C++ engineers. You're not engineering if you're not writing code in C++ (basically)
- Next tier were the Java engineers. They thought you weren't engineering unless you were writing in Java or C++
- Next came Python and Go
- Last came Javascript
Now at Google I met some engineer who were _superb_ C++ engineers. Like truly amazing. I also met others who were like walking Wikiepedias for the C++11/14/17 standards. And no, that's not the same thing.
So one example that springs to mind once was surprised to learn that I didn't know what perfect forwarding was. This is something that probably only needs concern you if you're writing a widely used C++ library using templates (especially template metaprogramming). As a user of such libraries, it's not often something you need to know.
My point here is that with all the knots C++ has twisted itself into over the years as a result of its origin and legacy, the trap some fall into is viewing such complexity as a virtue instead of baggage.
IME these aren't the people you want making design decisions on complex systems. The people you want making design decisions on complex systems are the ones who reverently follow Postel's Law.
Complexity of a system actually creates meaning for people that do understand it and can navigate it.
Complexity also justifies headcount and effort put towards maintaining, enhancing something.
Finally, people associate complexity with being smart. If you like simple things you are either a newb or you are [very] senior/old/experienced.