Woo for Its Own Sake
scottlocklin.wordpress.com
scottlocklin.wordpress.com
One company had several different teams of programmers that were great at building things, but they were all going in different directions with different tech stacks (two in Go, one in Rust, and one in Python) and completely different data models for the company's basic business domain. When a directive came down from above that two systems HAD to be integrated, because a customer paying for one product wanted the other one as well, as sales and product had been hoping and planning since before the products were built, the engineers gave them an estimate of four months to do the integration. The business knew that meant 8+ months. How many programmers did they have to get this disjointed? Less than twenty.
Lots of other companies have similar stories. I disagree that it is about lack of financial compensation. These companies were paying good money, and their engineers were living comfortably and abusing the relationship, whether wittingly or unwittingly.
Anybody who reads HN can whip up an impressive, Netflix-blog-worthy design for any problem their company has. If their company is unlucky, they might even have the energy and raw hacker ability to get it mostly working. Where we're failing as a profession is in educating people that this is poor engineering, that it's bush league bullshit. It isn't impressive. It doesn't show any ability except a certain amount of shallow raw intelligence, which is necessary and valuable, but which is more like raw material to build ability out of than ability itself.
In other forms of engineering, the culture is very clear: simplicity is an achievement. Solving an easy problem with a big, expensive solution is not an achievement. It's unskillful. The best engineers come up with solutions that make you see the problem as an easy one. Solving easy problems with difficult solutions proves you aren't qualified to take on the problems that need those solutions. How do we establish this as a standard belief in software engineering as well?
Sometimes it's also not resume-driven development, at least not intentionally. Sometimes it's because the engineers just want to work on overcomplicated shit. For example I have my personal blog running on half a dozen AWS features - S3, Athena, Cloudfront, Cloudwatch, Lambda, SNS, SES, ...
I don't need any of that for a simple blog, and I write on it like once a year. And the site is still slow as shit (relatively) even though it's static because I haven't optimized the font and script loads to be non-blocking. But it was fun to set up!
Fixing this requires lots of communication and leadership. Each project may need to select sub-optimal components (like language) for itself, but in doing so it can remove overall complexity.
The difference is that nobody paid you for doing all that unnecessary stuff. Doing unnecessary things for learning is good, but not what a company is paying you for, usually.
I'll grant that collectively an engineering team might decide to overengineer a simple, noncritical service to get production experience with a technology or two that they are targeting for future adoption. Nothing wrong with that, and there's nothing wrong with making room in the schedule for engineering experiments to test out new technologies.
Managers and the rest of the business, for their part, need to understand the need for experimentation so it can be done honestly and not under the guise of other work.
I think engineers try to do the best they can. It's just old guys might do it the mainframe way, slightly younger unix, then windows, then linux.
Or folks who've been with the culture do it that way, and new folks from outside the company bring new college grad or last-company-i-worked-for culture along with them.
And folks who try something new suffer from the 'fog of war' and your "the docker solution" that got everything started so well makes you a real expert in nested docker configurations and firewall issues. :)
If a mechanical engineer designs something that needs a ton of fancy moving parts, those moving parts significantly and directly impact their budget (I assume), because they're going to have to be purchased or made for each widget that gets manufactured. They also unavoidably impact physical properties of the object, like weight and dimensions.
Whereas a programmer can spin out as much code as they want, basically without material consequence. There is no constant gravitational force keeping them in check, requiring them to justify each piece of complexity as they go. It's ethereal, weightless, and its disadvantages usually only surface years down the line.
Code is a liability, in other words. It may not be tangible, but you can think of it as weight that the team has to carry everywhere it goes.
I know having completely independent teams is the in thing but there needs to be some high level design and coordination. I'm not even against different teams using different tools but the integration story has to be there from day zero if it's a business requirement.
Ultimately there needs to be an engineer who has both the power and knowledge to coordinate design decisions.
EDIT: Going further, I think the people trying to hire me as an engineer to fix their problems should have replaced their front-line management with managers who understood code, so they had some clue what their engineers were doing wrong and how to fix it. The idea that engineers should be able self-manage is a rare best case scenario, not a best practice that can be replicated across the industry.
My experience with things like foreign automotive components suggsts this is not exactly true in general... but it's true that monsterously complex solutions seem to be far more common in the software world.
A cynic would say the complexity does not even have to bring extra capability; frequent and costly maintenance is an important part of the status signaling.
This is diametrically opposed to e.g. Toyota's simple, traditional, robust, and scalable transportation appliances.
- Lower spec versions are available and common; the brands themselves are not inherently special or upmarket
- This means mechanics who work on the brands do not automatically command a premium
- Parts are not subject to import duties and supply chain cannot get away with as much markup (after Brexit who knows?)
- Culture of better compliance with scheduled maintenance
- Depending on location, better maintained roads and milder weather
Also they have invested a lot in reliability in the 2010s and some of the reputation is a holdover from previous generation models. Problems will still be outrageously expensive to fix compared to Japanese cars but are not so disproportionately likely to occur.
My dad briefly owned a BMW as his midlife crisis car. The bi-xenon headlights looked better than any I had ever seen. But when one burned out it was $1500 between the part and the labor to remove practically the whole front end. This is considered a characteristic, “classic BMW” kind of problem.
Civil engineering is sort of the oldest and ultimate example, where a bad design puts people's lives at risk. In deference to said lives, senior engineers in that field review every drawing or calculation that goes out the door, and they also mentor the junior engineers, who have to be graduates of accredited schools, and also can't get certified until they've spent a certain amount of time being mentored and passed the exams, etc.
The mindset you're talking about is basically all about craftsmanship or elegance of a solution. Formal mentorship or apprenticeship programs do a great job of maintaining and transmitting that. School isn't really enough, and certain fields have recognized that fact.
In software however, and I'm not sure whether this is the problem or the symptom, the number of developers apparently (I was told recently) doubles approximately every 5 years. Which means the field is always overwhelmed with new people, full of energy but not experience, who want to do something cool and exciting, but haven't been, for example, burned by their own past decisions to try to do something cool and exciting. They see only upside. Which is valuable in itself, but they need more direction than they're getting, from the older engineers who have the war stories and who aren't especially concerned with being cool (which c'mon let's face it, is essentially a mating display) and who are boring.
Can you imagine a bureaucratic oversight committee rejecting a CQRS design with streaming everything and telling an engineer, "Seriously, WTF. This is a config console. It will have less than 300 users two years from now even if you live up to that insane hockey stick graph they showed the investors. Just build a basic CRUD app backed by RDS. Use the same front and back end frameworks you use for everything else. The engineering standard says you can't afford to waste time on this because there's a lot of other work you're going to need to do this year, and some of it might turn out to be hard, even if you don't know what it is yet." Can you imagine that?
Math and physics also have a strong cultural orientation towards simplicity, which seems to be universally embraced. If you solve a homework problem in half a page, and the professor calls on somebody else to put their proof on the board, and they walk up with two pages covered front and back, you don't think, "Wow, they're smart. Their solution is so much more complicated; it must be better." Conversely, if you and all your friends took three pages to solve a problem, and somebody else presents a solution that fits on one chalkboard, you're in awe. I really don't know what the difference is that makes the culture of software different.
I think if it's being enforced by standards or licensure, it's too late and too external. It needs to be internal and cultural, at a gut level, to aspire to do more with less. In software, the primary measure of cost, the definition of "less" in "do more with less," is the intellectual load on the software development team. (We have that in common with mathematics.) Do more with dumber. How do we learn to value that as a profession?
The worst bridge that can stand up to the load is good, cost efficient engineering. The worst code is not. One, because bad code hides bugs that you probably don’t have rigorous enough testing to find. Two, because there will be new demands on that code in the future, and bad code is harder and riskier to change. You might look like a responsible adult when you ship it today, but three years later the new maintainers will know you were a child making a mess. And probably end up doing a rewrite, or at least needing to.
All this “woo” with languages and frameworks and services and whatever else, is the search for higher quality code.
There is a certain myopia about new things vs. getting better at design with old things. There is a certain blindness to iterative improvement, e.g. Rust is much sexier than modern C++ as a solution to the problems of C++98. But rather than condescending to this you can channel the energy into more responsible directions.
I work in a team where many people have zero aesthetic sensibility about code. Not like “I know this is ugly but we have to get it done today” or “I know this is ugly but it’s the best we could come up with,” just zero sense that it’s ugly. Grass is not greener.
> I’m imagining the archetypical unicycle-juggler buying a shop full of solid printers and weird CNC machines and forgetting to buy cutters, hacksaws, files and machinist squares. As if files and machinist squares are beneath them in current year.
Of all the things I could have commented on, I have to call out this last line simply because I only just saw a YouTuber last week 3D-print a part for a project he was working on: the part was the exact size and shape of a 4-inch long piece of standard PVC pipe you could pick up from Home Despot for a few bucks and saw the length you wanted in a few seconds.
More generally, I’ve also printed stuff you can buy OTS because it can be less labor than going to the store, and occasionally cheaper. Also, sometimes it’s just more fun.
Back in the day, in perl I'd bless regexes and use AUTOLOAD to provide the class implementations at runtime, as any good unicycle rider would. Changing the Rube Goldberg machine is hard. grep won't really help. I'd resort to even dirtier tricks to change behavior.
Later I worked in java. grep would work, but even better, I could make a class private, create an interface and just chase compiler errors. So much nicer than hunting down the right line of code to hijack in perl. It's pretty good, and I think go provides a similar level of support, but there's still a bunch of room for Class.forName(foo); //haha! grep will never find me! and it's cool, cause you'll only find out about it at runtime.
Anyway Haskell is neat because you can throw it all in a do in IO. hack away for days. When it's time to change, just change it. the compiler will help. The compiler will tell you everything you need to know to scope down to the finest level you want. It's super easy to be way too fancy up front, but any mistakes can be fixed.
Anyway, Haskell is probably never the right choice but it's the only thing I know of that lets you easily change the design whenever you feel like it.
It's Saturday, and I've thought about this too much. Time to put on my silver pants and go unicycling.
The bit about us not being smart enough for some of the technologies also rings really true. Thinking of a coworker who was "smart" enough to reach for threads right away, and then inevitably spent the next 2 years of any project dealing with thread issues in production.
1) a problem that requires a very high-assurance system with very low tolerance for bugs, errors, and failures, and
2) a large complex system where refactoring and other maintenance-over-time activities are made materially more manageable by Haskell's strong type system and other tools,
3) a need for large-scale parallelization that a purely functional language like Haskell facilitates, and
4) a world class team of engineers who can become productive in it relatively quickly, support new hires doing so too, and won't foreseeably be outsourced to some low cost overseas sweat shop.
Rust is also worth considering under those criteria, in addition to embedded systems, hard/soft real-time or constant-time systems, etc.
But obviously not all software projects have all those criteria. Other languages like Go, Python, Javascript, Java-ecosystem, .Net, etc may be ideal in other circumstances. You just have to match the tools for the job.
That said, the idea of different data types in C could have been considered a fancy addition by BCPL developers.
The key is to use critical thinking to determine what developments may be useful longer-term, and also to watch out for any (typically accidental) flaws exist in the novelties that are being proposed - sometimes it takes years, but those can come back to bite us.
It may not make business sense to exert effort shifting to new technologies straight away, but encouraging people to learn the strengths, weaknesses and properties of them can be valuable nonetheless.
Point taken, but I see no reason to straw man Haskell and Rust. They have their uses and new successful features they explore get ported back into the boring languages. This is how technological development works. For a non-programming example, anti-lock brakes were used in Concorde jets before they were used in automobiles.
Also, one must consider that Java was a 'shiny' language once.
Honestly, if OP knows of many companies that are picking Haskell or Rust over the likes Java or Golang, please post links. I'll happily apply.
Java was never a shiny language. It was many things. It was, most particularly, a ticket out of the world of the Microsoft sharecropper, needing to learn a whole new Microsoft Framework every second year, and discovering over again each time all the shit that didn't work, that couldn't be made to do what it was for, and cobbled-together ways to get around it instead.
I managed to never need to code on Microsoft until a bit around 2010, so missed the appeal.
I was amazed to discover that, even in 2010, nobody who had coded for MS their whole lives knew any better than I did how to get an MS OS to do any sensible thing. I would ask, and they would start rooting, trying random things until something seemed to work. The idea of a documented, stable, standard library component that actually did what it said without crashing was an absolute revelation, but the opposite of every shiny thing imposed on them before.
I've tended to find that these people don't write particularly good code (not usually bad, just mediocre) and tend to lack ability in software engineering - writing deeply unrobust code that falls far short of their perceived 'rockstar' qualities.
I can't tell you the number of conversations I've had with managers or even colleagues where their untouchable genius is extolled to the heavens.
Combine this with the fact that those who raise issues and foresee problems are often viewed poorly (possibly even MORE so if those problems actually end up happening).
Ultimately you end up with a situation where those who advocate for solid engineering are, while respected, never really raised above the low levels of the career ladder and those who embrace woo rise up.
But I still like the take a lot, particularly because of the perfect truth of the line "[Go] is a boring garbage collected thing that looks like C for grownups, or Java not designed by 90s era object oriented nanotech fearing imbeciles. "
What are you doing? What are the constraints? What is the problem?
Do you need innovation? Do you need plumbing?
Can you lift your eyes and see the value in a radically new way? Does it look like what you are already doing, or does a beginner's mind really suggest a new approach?
The author sounds like a crabby old guy who doesn't want to change. Probably an artifact of the topic. But it *is* a real problem- people get locked into fancy-chasing. This introduces risk into the project delivery: team based (knowledge) and tech (not adequately debugged).
I've worked at length in "fancy" languages and some of the fancy tech. My conclusion is you have a limited amount of investment you can do. Some of this is team based (language learning, keeping the employees happy), some is in product, some is in tech stack (FlashyFancypants DB and SparklePants.JS). Good tech leadership includes picking out what to take risks on, judging appropriate ROI. Note this holds true for open source, not just business projects!
I work in a Clojure shop, so I interact with the rewards and risks of this strategy right now. My last team/company, we chose Go for the team language. It wasn't as unilaterally good as some might hope. Nor was it as unilaterally as bad! It was just another language.
Less than 10% of it:
https://4e6.github.io/firefox-lang-stats/
Also there were several Rust developers laid off from Mozilla a few months back, including the Servo team:
https://www.reddit.com/r/rust/comments/i7stjy/how_do_mozilla...
In 2020 we call this a "makerspace".
> underpaid compensated in self esteem [..] manager saying “dude yer so fookin smart” [...] brainlet payment system. [..] benefit to boring languages is people concentrate on the problem not interesting complexities of tool
Missing the point raised earlier: "the problem" being solved is that work is boring; Woo isn't added "for its own sake", it's added because being an interchangable cog in a relentless productivity optimization machine isn't what humans exist for, no matter how much employers and the author would like that to be so.
> people who feel like every line of code has to be innovative
Employees don't get paid or promoted for saying "this is good enough, let's take 18 months off", but they do for adding a new widget or showing off to the boss. Broken incentives lead to people following them.
> like bike-shedding, but more perverse
It's pleasure! Spit! Disgusting, sinful and shameful, and the people who enjoy it are inferior. Get back to the grindstone!
> you are presumably solving real world problems which someone pays for
"Impress my manager", "buy some food", "survive this stupid doomed project nobody wants that doesn't have the political backing or funding to get to its impossible goals" are problems people solve for money. Jobs where people solve real world problems seem a small subset of jobs.
> about as necessary and in demand as riding to work in silver pants on a unicycle
The way someone commutes to work has so little to do with anything that it's a big reinforcement this is about image, not engineering.
> developers [..] are often child-like in their enthusiasm for new and shiny things
Everyone knows that companies are at their best when petrified, stagnant, resistant to change and unwilling to entertain new ideas, right?
> Engineers are different: they’re trying to solve a problem
No-true-Scotsman so that "Engineer" now means person who tries to solve a problem (only using existing tools and techniques), and people titled or qualified as "engineer" who aren't trying to solve problems, or anyone who is trying use a new approach to solve a problem or doesn't have the same constraints and goals are "not real engineers".
> Engineers understand it’s OK to solve the problem with ephemeral, trashy, but fast-to-market solutions
Nothing the world loves more than falling bridges, burning skyscrapers. A moment ago the important thing was to build stable, long-term, easily maintained systems with boring technologies (a classic description of "engineering"). Now Engineers are building ephemeral trashy minimal viable products? Is trashy and ephemeral "engineering" now? (No, the point is who to mock and shame, and who is superior).
> if the product manager is going to change it all next week
Could be related to why the idiot in the clown shoes is enjoying building it in Haskell, knowing that it won't need to be long-term maintainable because it doesn't matter anyway. No, it must be because they're inferior to the blog post author, who is all about solving real world problems like making sure software engineers know their status is beneath him.
> Engineers also plan for the future [..] critical infrastructure that lives [..] depend on
Engineers do the right thing in the right circumstances, and you can tell they are engineers post-hoc when it worked? Handwave away any critical infrastructure which failed as "not built by engineers", e.g. Boeing must have put no engineers on the Maxxx, you can tell because it killed people.
> Consider a previous era; motors for small fishing boats. [...] So the motors of the early days were extremely reliable.
Consider survivorship bias, the unknown number of fishing boats with more complex motors that worked just fine, the unknown number with simple engines that were unreliable and unfixable out on the water. Pointing out that some simple reliable engines exist does not show that "motors of the early days were reliable".
> Such engines are useful enough they remain in use to this day
Maybe simpler engines are simply cheaper and the lack of emissions regulations or any performance or luxury equivalent of a Ferrari commercial fishing boat hasn't caused much engine change over time? Complex systems can be useful enough that they remain in use to this day, too.
> this design still sold by the Sabb company in Norway [..] they’re definitely more reliable where it counts
More reliable than what, exactly? Are there competitors making complex and unreliable fishing boat motors that leave people stranded and dying, have been making them for 70 years and nobody notices and they still have customers? No, the alternative is a strawman sneer comparison, a solar powered Sterling engine. Point to a commercial fishing company run by a unicycling strawman using solar powered Sterling engines and then going bankrupt? Anyone?
> in the engine room, you are filled with confidence[..] If you find some jackass on a unicycle, maybe not so much.
And you should judge by how they appear, and not by their track record or by talking to them, since this whole blog post is entirely about who is high status and respectable and who is inferior by image and not really about whether you get home from your fishing trip.
[cont]
Does it? Oracle's JVM - is it a giant simple on-piston semidiesel or untrustworthy? It seems complex and trusted to me. Linux kernel, so simple such that a fisherman can fix it in the dark and cold and wet by burning pile of logbooks? AMD64 instruction set, trivial or untrustworthy? Stuff you can trust doesn't /look/ like anything in particular, it behaves in a trustworthy way long enough to get a repuation.
> Duct tape programmers don’t give a shit what you think about them
It makes more sense to say unicycle riders don't give a shit what you think about them, since they're the ones actually demonstrating non-conformity through their actions despite social pressure to stop doing that.
> the perverseness and destructiveness of people who try to get fancy for no reason, nor do I think JWZ was a “duct tape programmer” -he was an engineer.
Gone from saying duct tape programming is good, to defending someone respectable as "not a duct tape programmer" as if it was bad, then applying the True Scotsgineer label retrospectively - it must have been true engineering(tm) because it didn't not-ship, QED.
> OCaML, APL and Lisp [..] small teams can deliver
Small teams have tried to deliver same in many languages and processes, we see the survivors and say "this tool must be why". That needs a large evolutionary body-count of attempts in other languages which failed, and many attempts in OCAML/APL/LISP which delivered nothing. If nobody had ever used O/A/L because they weirdo unicycling languages, there would be no knowledge of whether they could do this. Rust and Golang came out about a decade ago, author is clear that Go is for engineers and Rust for unicyclers, but both came out of respectable teams that built complex software maintained for years, with good reasons (memory safety, not "add Monads to C++"); at what point in the past ten years did it become clear to the author that Go is engineering and Rust is unicycling? Was it clear that O/A/L were going to shake out well in 60 years, after their first 10 years?
> For mission critical [..] hit by a bus [..] deep sheeyit
Like the boat engines, there are visible examples where "deep shit" happened - e.g. crippling ransomware attacks, systems being offline for days, huge data leaks - and a month later there are no visible consequences; PR puts out an apology statement, bonuses are unaffected, business as usual. If your startup run by a true engineer using Golang gets hit by a bus it could be the end of the company too. If IBM has a team using Haskell and one of them dies, the project is swallowed up and there isn't any deep sheeyit that anyone outside the immediate customer blast radius will notice.
> Using the latest tool is even worse. 99 times out of 100, the latest woo in machine learning is not an actual improvement over existing techniques. 100% of the time it is touted as a great revolution because it beat some other technique … on a carefully curated data set. Such results are trumpeted by the researcher because …. WTF else do you expect them to do? They just spent a year or two developing a new technique; the professor is trying to get tenure or be a big kahuna, and the student is trying to get a job by being expert in the new technique. What are they going to tell you? That their new technique was kind of dumb and worthless?
It's like you understand incentives, but pretend you don't when it gives you an opportunity to sneer at people you want us to think you're superior to. Why don't you see that the same incentives lead to people introducing NewLang because the upside is a potential huge status boost?
> just quietly used by engineering types who find them helpful
probably get used by engineering types who find them unhelpful and wish they could use something else, by unicycle riders who find them helpful, and by unicycle riders who find them unhelpful, too.
> I’m imagining the archetypical unicycle-juggler buying a shop full of solid printers and weird CNC machines and forgetting to buy hacksaws
I'm imagining how easily a CEO and customer would notice the absense of precision lathed widgets, and what kind of world and incentive changes it would take for a CEO and customer to notice that a pile of NewLang wasn't turning out a workable project and take steps to change that. But they're probably too busy making sure the employees are wearing the correct clothes on their commute to work, because that's not straight from American Psycho or anything.
There are vast swathes of counter-examples to the article, to the point where I don't think that, given the business framing of the OP, the article actually holds water outside of "some people glue useless things on and it's a pain to deal with", which is sort of... known already?
The silver pants unicyclists know this too, they just lack impulse control.
But in reality, discussions I have about arguing for simplicity have two opposing sides both basing on tried and true technologies in their opinion.
I find a good solution is to have reviewers and outsiders who can give feedback, and generally trust them when they recommend avoiding something. (But I have seen the cases where this exact approach would have killed a crazy idea that became a huge success story later on)
Almost everyone wants simple. But simplicity is subjective. My simple can be a touch screen and yours can be buttons and we would both be correct.