The Bulk of Software Engineering in 2018 Is Just Plumbing
karllhughes.com
karllhughes.com
There is an art and a science to it. The guy I worked with put a lot of thought into the best way to design these things, often because he'd be the one who would then have to maintain them. He'd also get very upset when we go out to a trailer with a poor plumbing system. The main giveaway would be a mess of pipes going every which way without rhyme or reason. These systems were the ones that broke the most and were also the hardest to fix.
It might just be "plumbing" to someone that doesn't have to do it, but there is a whole lot that goes into it and you don't want a plumber who approaches his/her work with the catch phrase: "Well, it's just plumbing."
A reputed community conference, co-organized by contributors and maintainers of the Linux kernel and related low-level "plumbing" utilities and libraries.
He also said he refused to do work on existing plumbing. He loved to build new systems but hated maintaining old ones. Where have I heard that before...?
I think it's the only time I said "THAT is the kinda setup I want."
Infrastructure, properly done, outlasts its own empires.
Some of the successful ones that own their own businesses are able to make a lot of money (a consistent >500k/year). The downsides are that those businesses are difficult to scale since they are service oriented, and they depend on good physical health. Moreover, consistent quality is mostly the responsibility of the owners, and again, that can't always scale far either. You also get out what you put in, in most cases it seems like the pay is a linear function of productivity. It's definitely not easy or normal to make that much, but it's doable as a tradesman
Before I was a programmer I was an industrial electrician and had worked on sites, after I was a programmer we where down on a site installing some equipment and a plumber walked past and said “your the guys who came 200 miles to turn on a switch right?” And I instantly replied “nah, we just heard there was a scouser doing some work so we had to come look!” the techie I was with looked like he was gonna shit himself but the other guys around fell about laughing.
The thing with work sites is you can’t take any shit but you won’t find funnier people either.
(It was also the most meritocratic job I've ever had- your ability, or lack thereof, was on display for everyone to see. And working on teams where everyone is competent is a joy, virtually regardless of actual task.)
He worked in the wrong field for him exactly, but his skill in his trade is somewhat rare.
The man was not properly educated in many fields, but if it has anything to do with welding, he's a complete wizard. You would never know what he knows about a subject, because he might not understand all of the context or language with which a question is asked with regard to chemistry or physics... yet he does know. He can perform complex calculations in his head, on the spot, and seamlessly weld two or more different types of metals of different thicknesses in the most inhumane physical positions, and that's a Tuesday for him.
I swear he should have been working on rocket engines and not in an oil refinery, but he was never exposed to the possibility.
He expresses sometimes how he wanted each of my siblings and I to have a trade each unto ourselves, yet fails to understand that what I do can factor into that as much as carpentry, welding/pipefitting, plumbing, or otherwise. It's funny... savant-like.
Anyway, in short, I agree. :)
This is a rational reaction, actually.
The problem with he "trades" is that they are often very physically demanding and take a great toll on your body over the years. They also often have a very high potential for debilitating injury.
A 70 year old programmer is no longer rare. A 70 year old tradesman is a unicorn.
Plumbers don't have to craft their own pipes nowadays because the existing piping solutions are very well understood. It's not as important that the plumber knows exactly how the pipe is produced. They need to know how to set up the interface between the pipes.
Similarly in software, we have high quality solutions to many common problems ("the pipes"). While we do have some people focusing on designing the solutions, in 2018 people don't need to know how to make their own frameworks in order to use one. This reflects a maturity of the industry.
The other interpretation is that most software engineering work doesn't require significant theory or novel work. Implementing software solutions to theoretically resolved problems still requires skill. Likewise plumbers very rarely do novel design work, but that shouldn't be a judgement of their skill.
I think the statement, "most software is plumbing" is true insofar as most software developers need design novel solutions. There are existing frameworks, libraries and languages for the vast majority of business needs. That kind of maturity is a good thing.
The Bulk of Software Engineering in 2018 is Just Plumbing
In my opinion it's a bit of a lazy analogy and the author doesn't show any real knowledge of what goes into plumbing or how this analogy provides any value.
Software development is full of analogies that are designed to aid understanding of a large array of abstract concepts. This is good! I just don't see why this "software development is just more and more like plumbing" analogy has become so popular in the dev community. I get what they're trying to say, but it says more about how they perceive/approach their work than it says about either field.
I feel like I could provide just as much value by taking all the points this author made, just replacing the word "plumbing" with "open-heart surgery", but everyone reading my article would be like what the heck? It's almost comical:
The Bulk of Software Engineering in 2018 is Just Open-Heart Surgery
Software is great because you tend to get out of it what you put in. The same is true about plumbing (please don't think about that too much ...). But that's not what these types of articles are getting at.
I don't know if the people making the programing is like plumbing comment realize how highly skilled a plumber is.
I think this is less a denigration of software engineering in 2018 than it is a denigration of software engineer interviewing in 2018.
This gave me the impression he hasn't really thought through what he is saying.
> Just as a good small business owner should hire a humble plumber who knows the standard tools required for small bathrooms, and will pay him market rate, a good engineering manager should hire humble team players who use industry-standard tools to build reliable software, and pay them market rate.
This furthered my sense he hadn't thought through what he was saying.
> Part of the problem is that we over-engineer the software we work on. I am guilty of this as well.
I have no idea what he's getting at here.
> “Oh, so you’re building an app? Does it use AI? Blockchain? ? Take my money and do it!"
Unfortunately investors these days love to hear that. Devs who bring these skills are most definitely adding value, even if the author deems it largely superficial.
> Finally, don’t get comfortable. This isn’t a popular opinion among software engineers, but I believe that many in the field are overpaid relative to the difficulty of the work they do. Sometimes it’s because they are the only people who know some obscure corner of their field, sometimes it’s because their company’s gatekeepers are preventing full access to the talent market, and sometimes it’s because other engineers insisted they overbuild things such that junior devs will take a long time to understand it.
Should plumbers not get comfortable too? At this point I think the author forgot what he was writing about.
I'm not trying to be mean. I was just trying to add value to an analogy I had trouble distilling value from. I am speaking from frustration at seeing comparison here and there lately in developer communities.
To answer your question, I've never been asked in detail about complex algorithms for either plumbing or software work. Maybe I've been lucky or maybe I just exude an unnatural amount of competence (I'm guessing it's the former, I have resting dumb face).
Anyway, I don't think gauging someone's level of understanding is unheard of in the plumbing world. It probably depends if you're going for a junior or senior plumbing position.
Edit: some clarification
I see what you did there
I believe that every programmer is overpaid relative to the difficulty of the work they do -- or, more accurately, that The Market doesn't pay based on difficulty of work. Software pays so well because the product scales so well.
Successful pop musicians don't do work that's 100 times more 'difficult' than software engineering. They do work that scales even better (copying digital music; playing to arenas; branding on merch).
Having worked outside the software world, I absolutely do not believe that software people are any smarter, on average, than anyone else. Ever see a plumbing or electrical or structural system fail spectacularly? Other types of workers absolutely need to understand interactions between multiple complex systems, deal with obsolete and incompatible systems, and deal with changing and conflicting requirements and regulations.
If software is any more complex to deal with than physical systems, it's only because the architects and implementors let it get that way.
and also there are a lot more people around who have the mental capabilities to learn plumbing than there are people around who are capable of consistently apply the abstract thinking patterns that involve software development.
https://www.seattletimes.com/business/real-estate/pricey-sea...
I think it's just confusing different terms. I'd say the bulk of programming is similar to plumbing - maintaining, resolving issues, etc. But that software engineering is more similar to engineering in general.
A mechanic works on cars, a mechanical engineer would design the components. Much like IT consultants/administrators would maintain systems and do minor coding, but engineers would deal with architecture and new designs and more advanced solutions to common problems faced by IT.
A random person from the street would not be able to fix a trivial problem/bug in some software - let's say a missing runtime package on linux.
I think you wildly overestimate the cleverness of the average Joe. I watch users struggle with basic GUI concepts. Add in learning a specialized language and support libraries? Joe can figure out a wrench by looking at it. He's not going to get very far with a shell prompt.
At least on a command line you can get somewhere with trial and error:
$ ?
?: command not found
$ what do I do?
No command 'what' found, did you mean:
(...)
$ help me
bash: help: no help topics match 'me'. Try 'help help' or 'man -k me' or 'info me'.
$ help help
(...)
To put it another way, I've found that the concepts of "hierarchical filesystems" and "drain waste vent stacks" are equally foreign and equally accessible to average laypeople.
Also, people new to programming and command lines are seemingly incapable of reading error messages.
Aren't all people seemingly incapable of reading error messages? (ignoring the rouding-error-amount of people who do)
I have no background in plumbing - in fact, as a new homeowner, I just learned what a compression fitting is ($#@%!). I read https://en.wikipedia.org/wiki/Drain-waste-vent_system and... it seems pretty straightforward. I could build this, if perhaps with a few more trips to the hardware store than a professional might make.
On the other hand, I work with databases every day. I wrote ROLAP engines in the 90s. And yet this is probably the first time I've ever used the term hierarchical database in a sentence. If you said "here's this hierarchical database, generate a report for me", it would take me some hours to figure out what kind of db it is and how to actually use it. If you asked a layman I think you would get a blank stare.
Not to belittle the craft, but there's something intuitive about plumbing. Software is something different.
Initially I wanted to give an example of fixing a missing include/import in the source code followed by building the binaries as a trivial software development task, but I think we can agree that regular random Joe has about 0% chance of figuring out that, so I toned it down to just a missing runtime package.
Poverty is a social tragedy. Inequality is an inevitable consequence of choice.
I do find it darkly funny that so many folks seem to throw up their hands - it is obvious that inequality will always exist, so just live with however much we currently happen to have and shut up about it. (Inflation, however - we can totally micromanage that!) Humans have an innate sense of fairness that becomes offended at some point, and they start punishing even when doing so harms themselves. The peasants get restless at some point, the tumbrels come out, and there's your natural law in action.
So you can't easily choose to have less inequality, unless you also choose to eliminate almost all (other?) choice.
Still, couldn’t you say the same for a feudal structure? That nature has power imbalances, and the role of king only conveys that inequality?
Instead, we’ve intentionally built a more egalitarian system (and we’ve all prospered because of it), seemingly against nature.
Maybe there's a distinction to be drawn between "inherent part of the human condition" and the "default state", as you have described it. Attempting to change something "inherent" seems worthless, but default is merely waiting to be overwritten with something better.
For ex: death is inherent in the human condition, we cannot avoid it and must deal with it. Nakedness is only the default state, and is pretty easily solved.
Back to economics: maybe scarcity is inherent, but "social tragedy" may be merely a very solvable default.
But keeping appropriate timescales in mind, I agree with the conclusion - for given constraints, there are things that are inherent and not worth attacking, and then defaults that are merely attractors for the system state.
Should? You're assuming that mortality is a problem. Extreme longevity or immortality would certainly change the human condition, but it's purely speculative to think it would be better in some objective, widely-accepted sense. Not every "default" is something waiting to be fixed.
There's a good argument to be made that the assumption of mortality is baked into so many things in our economy that it would break down if people suddenly started to live much longer on average. But that, again, is a default - some state attractor. Something we can try to work around.
As a speculative anecdote, imagine spending ten thousand years missing a child who died in an irreversible accident. (Would childbirth even be permitted in such a future? How would overpopulation be addressed?) Or the dampening economic and sociological effects caused by a millegenerian population's entrenched resistance to change. Or the possibility that longevity is only available to the wealthy, or the drudgery of being stuck in a boring job for millenia...
Longevity doesn't necessarily solve sadness, suffering, and social ills -- it just shifts the goalposts.
More egalitarian in the sense of opportunities, yes. But that doesn't mean it will be more egalitarian in the sense of outcomes. The disparity in wealth between the poverty line and Bill Gates, for example, is probably larger than the disparity in wealth between a serf and a king in feudal times.
I'm not sure that's true. While, as I said, the disparity in wealth between Bill Gates and the poverty line is much greater than that between a king and a serf in feudal times, the power disparity between Bill Gates and a person at the poverty line is a lot less. A feudal king could command all sorts of things of serfs and exert all kinds of power over their lives that Bill Gates can't command or exert over anyone. So Bill Gates' money does not represent power the way a feudal king had power.
No, markets pay for things that are worth more than it costs to make them. Only a very small fraction of those things are useless toys that delight rich people.
That being said, profit is a notoriously crude way of measuring utility, given that a lot of tragedy-of-the-commons type problems don't have "profitable" solutions without some type of government intervention. (See: pollution, public education and health, childcare, working week limits, etc.)
There are no non-crude ways of measuring utility among different people with different, often incompatible, preferences.
> a lot of tragedy-of-the-commons type problems don't have "profitable" solutions without some type of government intervention.
They also don't have good solutions with government intervention, because the downsides of government intervention (regulatory capture, increasing power of the state, micromanagement of all aspects of people's lives, political power being used to benefit some instead of all) more than outweigh any increased benefit in a particular area.
Yes, but everyone's desires and needs count in a market. Only the politically powerful's desires and needs count when the government intervenes.
Markets pay for things that people, period, will pay for more than it takes to create them. There's no restriction to "people of means". But yes, the word "worth" can be misleading if it's not understood to imply having something to trade (almost always money in today's economy) that the seller will accept in exchange.
What markets provide is an information-carrying mechanism to determine how you should allocate your efforts to get some of that wealth. If you're poor and want to be rich in a market economy, make a useless toy that delights a rich person. If you're poor and want to be rich in a communist economy, suck up to the party bosses. If you're poor and want to be rich in a feudal economy, you're fucked, because that useless toy that delights a rich person is his by divine right.
While all of these systems still have rich people and poor people, I would argue that the former leads to much more socially desirable outcomes than either of the latter, because at least the rules of the game for how to become rich are stable and well-known and the side-effects of the wealth-acquisition process are beneficial to humanity (or at least to that one rich person, but that's better than them being outright negative-sum, like when the way to get rich was by conquest).
I would argue that the former is self-evidently equal across systems. By definition, if you're speaking in relative terms, for you to rise up the social strata someone else must fall.
I would not argue that this is true in absolute terms - measured in your country's currency, it seems empirically true that certain systems (in particular free-market corporate capitalism) lead to much wider disparities in wealth than others (say, communism). However, I'd argue that if you actually care about absolute wealth rather than relative wealth, this shouldn't matter to you, because the median person in corporate capitalism does far better than even the wealthy in communist countries (we discovered this in the early 90s).
There may be hybridized systems that can still generate adequate absolute wealth while capping relative disparities. I'm skeptical of many of the examples that are often cited, though. IMHO unionized capitalism actively failed during the 70s; people like to blame Reagan or Wall Street for the dismantling of unions & pension plans, but the competitive reality is that the companies that made up this system all went bankrupt because of foreign competition, not because of any particular policies. That's a failure to generate adequate absolute wealth; you can't ignore the existence of outsiders who follow a different system when judging the sustainability of your own system. European-style socialism is intriguing to me, but may suffer from a similar problem: it's unclear to me whether the EU can a.) hold together and b.) avoid getting conquered militarily by outside powers in the absence of the NATO security umbrella.
Today, Some actors(not necessarily the good ones), get lucky and become filthy rich.
So is their luck an inherent part of the human condition ? or is it the result of technology, TV/Cinema ?
And do we have a right, as a society, to choose how technology will shape us, or not ?
My partner is a nurse, she gets payed peanuts while daily ensuring that people don't die. Literally.
What judgment exactly is exercised to have life saving jobs be payed order of magnitude less than "plumbing" jobs? The argument of "higher positive impact on their well-being" is laughably naive.
Consider this - it takes years of formal training to produce a qualified medical professional and pretty much nothing to produce a software developer.
If the software engineer is able to provide more utility, this roughly translates to more demand which means higher salary which means higher supply
It resulted in an explosion of theoretical and empirical literature in microeconomics, and in many ways creates the field we know today.
The average CS major has an IQ of around 125[1], meaning the average software engineer is somewhere around the top 5% of the population in terms of intelligence. The average IQ at the top tech companies is probably a standard deviation higher than that. Despite this, the consensus is that the majority of software developers are pretty terrible at what they do.
If you've ever tried to help people learn to code you would realize a good chunk of the population doesn't have what it takes to do anything beyond basic programming. If it was easy more people would do it and salaries would drop.
1. http://www.randalolson.com/2014/06/25/average-iq-of-students...
...which is another way of saying that writing software is a good use of relatively smart people's time, which is why the market rewards it.
What?? :D
No and no. People get paid for working on things that take an effort. Nobody would pay you if there was no reason for it. Even the "window watchers" (or how they're called) are payed because they didn't change jobs for 40 years, were loyal to their bosses, colleagues and company.
And yes of course, some jobs take a higher amount of "problem crunching" (whatever that means) than others. In fact sometimes this is the reason certain jobs can only be performed by certain programmers. Most programmer jobs cannot be done by none-programmers.
> If software is any more complex to deal with than physical systems, it's only because the architects and implementors let it get that way.
There are not many fields that are purely abstract, namely that's mathematics, logic and theoretical computer science. Those fields are notoriously difficult, every school kid knows this for mathematics and eventually gets that logic is also a hard nut. This completely misses the point that reasoning about completely artificial entities is an effort by itself. It not only gets harder with complexity but also depending how abstract it gets.
In fact their are even weird limitations that you do not experience in the physical world like paradoxes.
If that is the case then why not hire cheaper individuals to do the same job?
There are many difficult things that are worthless in value.
You could be the greatest deep sea underwater basket weaver, that doesn't make your goods valuable. Nobody wants your damp baskets.
He's not saying that programming is easy: he's saying it has high value to difficulty, higher than other more difficult careers.
Having worked many jobs I also disagree that it's easy, relatively. It's probably not due to IQ, it's because it requires constant thinking. Every other job I've worked is a light mental load compared to software and electrical engineering.
Most jobs you don't think about anything half the time. In hospitality you largely sit around waiting for people. Same with finance and sales in many cases. In HR most of your time is spent sending email, entering data on websites, and attending meetings. Or, your job is to talk to people most of the day, something many find more appealing.
Engineering is 6-8 hours of intense focus and fixing difficult to find bugs. As palatable as doing math homework for most people.
I disagree about intellectual difficulty as well. In school I was a teaching assistant and many, many students simply couldn't put working programs together. Even if you know all the constructs, software engineering is basically building something in your head then tanslating it. I've seen enough people try and fail to build any working programs for 6 months to believe that some people just aren't capable
By that logic, the cleaning staff in your local adult movie theater would be millionaires.
Companies should feel able to innovate when appropriate by having hired engineers capable of doing so.
The glamour (and risk to life and limb) of doing so is that of plumbing.
Perhaps a more apt analogy is that we are more like home renovation specialists. If things go well, we can update your plumbing to get away from lead pipes, but if we have to route the new plumbing through a joist, we are going to have to educate you on why we should do something a certain way. If we find a water leak and rot that will put your home (business) as risk, count on us to tell you about the risk and let you decide if you want to address it.
I’d argue that we are simply doing plumbing when things are going really well, perhaps even when the house is small and was well built to start with. If plumbing was all we did, I’d argue that our field would be way more saturated than it is.
If it's so basic, just hire them and take a day to teach them those things.
WTF is with this trend of waiting 6 months for a hire vs. spending a day to fix the holes in someone's knowledge.
An expert is someone experienced and well trained, a rockstar is some inherent quality that's selected.
Our industry needs more of the former.
I’d wager actively recruiting for 6 months takes a lot more effort than training someone for one day, for that matter.
As a manager, I find it more interesting to develop someone’s skills than hire a rockstar (and deal with ego issues). It’s more work, and you can’t always afford to do it, but at the end of the day you’ve contributed to fixing the long-term problem by adding one more resource to the talent pool. And it’s way more rewarding, and arguably one of the best uses of your time as a manager.
I'd say it's also one of the best uses of the time of the more senior engineers who will act as (additional) teachers/mentors.
Having been in the position of needing to explain a concept or even just "show my work" has helped me not just better understand what I'm doing, but sometimes even catch fundamental mistakes in my thinking.
It's also why I like commenting on technical matters on HN. Sometimes forcing myself to articulate my thought to someone else will reveal the absurdity of a long-held assumption, or something similar.
If the examples have to be so big, they might as well be real systems. Then you're not being taught so much as learning on the job as you go.
It's accurate to call them basic concepts but you only learn them through extensive experience.
That's not to say that tech screens are perfect, or even testing the right thing. But they do play an important role: they're an attempt by the company to not give too much responsibility to someone who will use it irresponsibly: whether by accruing technical debt, or dropping tables willy nilly, or by taking down the website.
99% of engineering isn't plumbing. It's pushing back on all the feature requests that can't be done responsibly in the amount of time allotted. It's using your intimacy with the code base and inner working of the company to anticipate problems. It's taking advantage of your prestige as "some smart wizard that makes crazy things happen" to slow down and fix problems before they occur.
Others in the company can't tell whether you're acting like a spoiled brat or doing your job. To them, it's literally indistinguishable. And that's why it's very important to not vest anyone with this amount of authority unless you're very sure that they're going to use it well.
I agree with the rest of your comment, but I don't think you're giving non-engineers enough credit here. Sure, in lots of companies, the managers/executives are non-technical, and might as well be managing a packaged goods company. They think software development is wizardry and have no idea what you do. In most tech companies, non-engineer roles are somewhere on a spectrum of technical knowledge that might surprise you. Maybe they've been through dozens more software release cycles than you. Maybe they are currently practicing software engineers trying out a new role. Maybe they co-wrote the standard that you're now implementing. I've seen all of the above in actual companies. Careful about jumping to incorrect conclusions based on someone's title or the actions their role demands.
Rough amount: number_of_domains * max(ceil(1% * workforce), bus_factor_min))
At small companies, that may be most engineers. At larger companies, it may (or may not?) level out to a more reasonable ratio.
Recently I had an issue at work which took a day for my colleague to research and I solved it in several minutes because of some low level TCP knowledge. So the broader/higher skills in your tream, the lower the risks. May be those people (who’s looking for a person for 6 months) are trying to lower the risks.
all edits are spelling, I’m terrible at it.
Yes, you’ll need some people who know how to make siphons and valves, but that some is only a few, and most of them won’t even be plumbers, but work on building machinery that builds siphons and valves.
I think that in most jobs in the modern world, there’s plenty of low-level details that one doesn’t need to know about to do the job.
But no one is paid relative to the difficulty of their job, dummy!
It's always a combination of what salary you can command (function of your soft skills) and how much money you can make the person paying you (function of market dynamics).
Plumbing has standard ways of doing things, and standard parts that rarely change. Web services need to settle down like that. Really, few sites are doing anything very interesting.
Even a small shop needs to live up to pretty much a global standard, not least in terms of security. In the EU you must engineer with a security perspective in mind. The field is constantly moving. People come and go faster than most other industries.
I think there's an analogy to getting a better bike: it doesn't get easier, you just go faster.
Backends are getting more and more homogenized. There's still a fair amount of work that goes into accurately implementing business rules and a much bigger focus on quality and maintainability than ever before. But there's not a ton of serious thinking involved. I can pretty much follow a script for implementing microservices that will scale up to millions of users. I somehow ended up with a high-paying enterprise architect job and when they ask how we can build a system that will have 1000 users, the answer is that you can do pretty much whatever the hell you want so long as the framework is less than 5 years old and it will be fine. Then I go get a coffee.
Can you get a backend going for a thousand users fairly quickly and easily? Sure. But that's the WordPress equivalent of describing back end development. If you have any kind of process based workflow with varying parameters, integration with hardware, detailed reporting / analytics, hell, if you even deal with money or dates/ times / timezones, which is any non-trivial application, backend development gets real complex real fast.
It may be trivial to spin up a WordPress site, but it isn't too put together a solid, reliable React / Redux PWA with isomorphic rendering and offline support. It may be trivial to serve a few API endpoints straight out of the database, but it isn't when you have workflows and processes and states and have to deal with shit like GDPR.
> But there's not a ton of serious thinking involved.
Give a fucking break. The only way this could ever be true is if you only have experience working on extremely trivial applications with no complex functionality.
Either your definition of algorithm is absurdly narrow, or you are working on some very simplistic problems.
The same could be said about you and someone who started their career in the late 60s. You probably didn't have to write your own OS.
If you were building software that businesses expected in the 60s in the 80s, then your job in the 80s would have been a lot simpler than it was in the 60s. And If you are building the software now that businesses expected 20 years ago, then yes your job is much simpler.
That's generally not how it works though. As capabilities increase, expectations increase, and many times expectations increase even faster than capabilities.
And pulling requirements out of business people who don't really know what they want, then turning them into sensible data models that can produce reliable and actionable business intelligence is still where I spend most of my time.
I guess it depends on what type of backend it is, but I wager the backend of Google, Amazon, eTrade and Twilio require some serious thinking. The backend of simple CRUDs? probably not much
The saddest part about this for me is the realization that there is little room left for truly innovated web technologies that have real impact to businesses. Of course there are counter-examples to this (I would count WebRTC, WebSub, and a few others as having potential to impact some usecase specific best practices) but the majority of new software I see being hyped is either a new implementation of an old idea (React and Vue for FRP on the front end) or something touting decentralizing as the next major step which so far hasn't proved itself valuable in a commercial sense.
And most startups lack any data. its all owned by the monopolies of Google, facebook, Amazon and Apple. So the rich will just get richer, and the chances of startups succeeding in any field is as low as ever.
My advice to new software engineers is to get into one of the more "boring" specialties. You might not be in the lottery for the really big bucks, but you probably weren't going to win anyway. There will still be plenty of interesting and fun challenges, with greater stability and (TBH IMO) more mature coworkers.
* Core kernel (schedulers, memory management, virtualization) * Drivers * Filesystems * Networking
Then there are some specialties that cut across more than one of these - high performance computing, financial tech (not even counting blockchain stuff), security engineering, performance engineering. Then there's a whole world of embedded stuff, from teeny-tiny microcontrollers up to high-performance medical imaging or genomics, plus large distributed systems to control things like military equipment or nuclear power plants.
Really, there are so many different kinds of computing that we just never even hear of on a site like this. It's really the web/app/mobile front-end stuff that's a niche technology-wise. (On my less charitable days I'd call it a ghetto.) It's a hugely profitable niche, to be sure, but to me it still seems to get a disproportionate share of attention and funding compared to other stuff that's just as hard, just as fun, and just as important.
The previous job I had (I've only had two programming jobs - used to do engineering) also wasn't CRUD - it wasn't as algorithmic but it did involve lots of low level USB programming which was interesting.
Are non-CRUD jobs really that rare? Surely if you don't want a CRUD job just find a company that wouldn't need to write any...
Leetcode style questions are valuable for testing a recent graduate on his textbook knowledge, not for what you imply.
But I feel part of the problem is that the companies want to rather hire a plumber (I mean, software engineer) than to buy an off-the-shelf solution. It seems to me that off-the-shelf solutions are very costly and many companies think they can do it cheaper if they build it themselves. Which to me sounds like that the market is not functioning properly, for whatever reason.
Consider Rosenberg’s Law:
Software is easy to make, except when you want it to do something new.
The corollary is, The only software that’s worth making is software that does something new.[0]: https://www.amazon.com/Dreaming-Code-Programmers-Transcenden...
Also, to be able to change the solution, you need to find one with just about the required feature set. In other words, you need a solution with minimal complexity that fulfills your needs. Because otherwise changing quickly becomes too painful.
On top of that, trying existing solutions and finding out why they don't work for you is not fun. It's much more fun to explore the solution space on your own.
If you have a bespoke system built from open source components it can be difficult to find people able to support it.
I think there are pros and cons to both approaches. Sometimes it can be frustrating trying to shoehorn the vendor supplied peg into your business's particular hole.
I've heard a lot of people over the years complain to my manager "XYZ is crap I could write a better solution using ABC" the reply is usually along the lines of "Yeah but are you willing to spent the next 10 years supporting it."
So companies do what they can to have the necessary control and also sufficient support: build it themselves.
Smaller companies leveraging existing products makes more sense in terms of budget and team size.
Sometimes it seems this is because someone who is more interested in building it themselves than doing more mundane work tells them so. They have no way to know for sure if it will be, and often it isn't, but the prospect of cheaper sounds good.
1. The amount of man hours necessary to build a reliable, effective system.
2. The expertise and insight needed to make it actually solve the problem.
3. The ongoing costs of maintenance, adding features, and support.
So, in my opinion, it's not that the market isn't functioning it's just that most companies are delusional about what it costs to build a complex software tool.
With the rise of low-code tools/platforms, building such apps is almost trivial. So, the make versus buy decision is not as straightforward as it once might have been, especially for companies who already might have developers on staff.
*Edit: typo and clarity
Because, judging from all the people I've been working with throughout the years, it's amazingly difficult to write a clean, cohesive, maintainable, scalable CRUD app that will actually satisfy your customers' demands.
Making things that work is easy. Making things that work well, that's another story. And it's the reason why there's so many open spots and high salaries for senior web devs who can deliver.
When was it not?
What were these guys thinking?
The vast majority of software is taking real world applications and dealing with the 'kind of a mess' of situations that can arise.
Even login/credentials can be a minor pain.
Doing it clearly and succinctly is the challenge.
Though it's often more 'solving problems' than 'building something' we should maybe think of ourselves more as construction workers than research scientists.
Upon graduation my skills were filled with so many holes I suggest that every CS grad takes a 3-month long 'power course' wherein the get all the basics, best practices and common tooling down.
But I agree that this is good. The job is just changing. For example, maybe you wrote a custom DAL back in 2010. Now what you likely have is a mess that you have to pay several people a senior salary for because they are the only ones who understand how it works. Meanwhile a general purpose plug and play solution like entity framework is ubiquitously understood among .net developers. So now we don't need those "senior" DAL guys simply because they understand the mess they helped create a decade ago (also, about the last time they read any books).
I see it as more a levels of software development thing. low-level > medium-level > high-level > plumbing.
Then other days I'll spend an hour doing nothing but thinking, staring out the window, trying to figure out the best way to do solve some complex multi-threaded issue, before writing any code, and I feel like a real engineer again.
My current company is basically creating Microsoft Office on the web - because for the targeted niche, Microsoft Office (Google sheets, Libre office) wasn't cutting it. There aren't libraries for that.
My previous company was applying machine learning to the twitter firehose, and present the results in real time. The existing libraries weren't up to the task (or the volume).
Before that, we built tooling to allow ten people to administrate 40 plus companies' on-premise databases. There were a few libraries and tools we used, but nothing that we could just "plumb together".
I saw in the Reddit thread that the "plumbing" argument applies mostly to backend developers who just put REST libraries on top of databases. OK, I can kinda see that, but it's not like the complexity of the programs has disappeared, it was just moved to the frontend. And in a way, that's a benefit to the company (and the employees) - the resources previously tied up in re-creating REST libraries before REST libraries existed can now do other things.
The value of programmers has never been in the movement of data from point A to point B, it's been in the application of business logic to that data.
Sure there is—textarea and http. The rest is just plumbing.
I’m only half-joking. I just don’t see a huge difference in difficulty between business logic and plumbing—they’re two sides of laying out the program, and neither one is more difficult than the other. Both lead to complexity and confusion if you lay out poorly.
I think that if you are good enough at googling, the vast majority of problems businesses are solving have already been solved (or the problems are well-mapped) and you just need to adapt the successful patterns to your domain.
Most devs should be focused on delivering business/mission value. This means custom code, unique ip, algos, GUIs, etc.
However, the current state of devops has thrown all sorts of engineering challenges at devs that lose sight of the actual value that should be continuously delivered. It becomes plumbing when we get caught up in redundant tasks such as dependency management, tests, logging, containers, runtimes, integrations, cloud provider specifics for scaling/ha/sre, and connecting them all to create "the platform" that is supposed to enable true continuous delivery of value. We end up more focused on this internal platform development even though the tooling is pretty advanced. We still have to configure/orchestrate/manage that advanced tooling, and this, in turn, creates the plumping effect. We greatly underestimate the work involved with all of these things.
It is not plumbing when we are focused solely on the continuous delivery of value. We remain lean and agile, constantly testing a hypothesis and developing our "uniqueness".
For every rockstar dev who gets paid a lot of money to make unique solutions and use his brain, there’s dozens of unsung heroes who do the grunt work reading from docs and setting up plumbing so the rockstar can deliver his value effectively.
These developers are not working in cool languages or solving new problems, and many of them will never do so in their life. They are just expendable workers swinging hammers and writing JavaScript.
i.e I've worked at a large enterprise and we all (product devs & ops) became so focused on our grand platform (continuous delivery, heavy integration with AWS, container orch, all using OSS) that product development stalled. Though we were doing "rockstar" things with cool tech and our resumes have a bunch of cool words, but it's true that most of that work was plumbing. In the end that tooling became a bch to manage and continually developing upon. The product continues to stall.
Wouldn't it be better for all if we weren't so caught up in the plumbing of these advance/cool tools? -- that's the thought that actually leads into something like building your own platform, vs something provided. Ops then become more like an overseer sre type of role, and most engineering shifts back to direct value.
This is straight up classism, period. Software devs want to pretend that their high (or even just decent) salaries are due to membership in an elite, highly skilled class. Software development isn't some brain dead job like burger flipping or some gross blue collar job like, ugh, plumbing, right? And there you see the classism raise its ugly head. The denigration of entire job fields. Forcing a separation between "true" developers and other workers also requires denigrating those other workers, how else can "true" developers be raised up higher without pushing lesser workers lower?
A lot of engineering is "just" plumbing. So what? A lot of real-engineering is also "just" plumbing (or the equivalent). So what? Does that mean it's valueless? Does that mean people who aren't in the elite should be treated like second class citizens or that they shouldn't make a decent living for their work? The truth is that people who do "just" plumbing work (whether in software development, in engineering, or actual plumbers) still do valuable, high quality work. Still do skilled work (not that that even matters or is a meaningful term). And still deserve to be compensated well for that work.
Let's stop shitting on burger flippers and plumbers, let's stop trying to define hard class boundaries between some hypothetical elite of "true" highly skilled knowledge workers and everyone else. Let's stop calling for people to be paid less for their work, almost everyone (even software devs) are being underpaid today. It's unhelpful, it's divisive, it's prejudicial, and it's just mean.
"Engineers" design/build components that perform with repeatable results.
"Developers" work out the plumbing details between components in a system.
When hiring, don't mislead people into thinking they're interviewing for an Engineering role when you actually need a Developer.
This, of course, is hard if you don't program and it's not really meaningful to evangelise about, so I'll leave off this rant here.
There's nothing diminishing about being a Plumber.
"Software is eating the world" means there is more demand for developing systems than designing them.
However, our lizard brain status system still associates directing and leading things with more status than just requirements implementation and there is no way around that yet. Also, less demand means that this more "abstract thought" type of positions are more prestigious by the virtue of scarcity.
This dynamic isn't speculative. It's _exactly_ what happened to law. It's what happens to any field that goes through a boom and attracts a certain kind of person who's more motivated by immediate financial reward than talent or passion.
Software in 2018 is better built than in 2008 for a lot of sectors. In 2028 I expect our pipes to be more robust. I also expect a single engineer to be able to do the plumbing 5 developers do now. At least with the right technologies.
I wonder if we in the tech sector tend to romanticize the trades. I don't necessarily disagree, but would only offer a caution: Look at their hands. I had a new screen door installed on my house, ordered it through Home Depot, and they contracted it to a father-son business. The father was not much older than me (mid 50s), but was broken and hobbling.
There's also a widespread view that we will always need tradespeople, e.g., to build houses. But the trades are being replaced by automation in subtle ways that could eat away at their livelihoods bit by bit. Examples include plumbing fittings where you just cut a plastic tube and squeeze it onto a fitting, rather than sweating it together with a torch. Another is the ability to design and order a "custom" house that's built in a factory and assembled on site.
I think it's the same with trade and craft knowledge. Any techie with any exposure at all to the more traditionally "manly" trades (I'd include things like military/police work or firefighting here) tends to flaunt it as a way to show that they're more worldly and "authentic" than all those other techies who have never known any other life. I used to do it myself, except that my "trade" was the simple one of actually being poor. Then I realized that it was just another kind of virtue signaling, and it lost its appeal.
It's possible the same mentality carries through when hiring programmers. I'm not sure it's a problem either.
That aside, the truth is, the vast majority of what's asked to be done has been done before. Even the particlars are likely to be wheel reinvention.
Mind you, it hasn't happen yet, but an object driven drag & drop (or VR) driven solution / application builder is only a matter of time.
Splash on some AI and the role of engineers not in the top percentile is going to trend to zero
I see AI as being the collection and resoltion tool. Imaging GitHub with a brain. The current problem is far too much wheel reinvention. Far too much language specific solutions when - ultimately - the language doesn't matter.
But ... with advances in AI / Machine learning, drone tech, and self-driving vehicles, I think there is a growing niche in software development for more serious science / data analysis. Investor expectations are being set more and more by this niche -- and causing regular application developers to have to push the envelope.
Although I don't feel comfortable going as far as dictating the outcome of some version of the build vs buy decision, I agree that, when I happen to be on staff, tremendous value is lost by not at least consulting me as to what "buy" [1] alternatives are available. As an experienced technology (not just software!) concierge, I often know even more than one.
Unfortunately, by the time I'm hired, it's almost always too late. Especially at modern startups, programmers very much run the show, so the focus is very much on writing new software.
That said, I don't think this notion is unique to ops engineers. It can be extended to programmers, as well, in that very senior ones tend to follow similar models, of simpicity, code reuse, libraries, and, ultimately, eschewing actually writing (especially rewriting) new software unless necessary.
[1] quotes because it's often free-gratis, merely not custom-written there
Contrary to a lot of comments, I still think the word "plumbing" fits well with connecting specific columns in the DB with specific holes in the template.
Like in real plumbing, you get shit when you connect the wrong things:-)
Dug up this old HN thread about Data First if anyone else is interested: https://news.ycombinator.com/item?id=10291688
here's my issue with the article text: it describes
I came to the Graide Network and immediately started breaking our legacy app into microservices.
here's where some of my experience partially aligns with the article's complaints
Investors are guilty of pushing it on us
here's where the author begins upon the path to enlightenment
we can build more with lessI can't believe this guy said "just plumbing".
And the difficulty involved is that we're not yet smart enough to build efficient, easily-adopted systems that use natural language and don't contain a million dependencies and hacky workarounds.
Anywhere that I've worked this will get your PRs or architectural ideals rejected.
Did you get that "plumbing" was just, like, an analogy? The article wasn't really about plumbing.
Computer science has always had this tension between being an actual "hard" science and an applied engineering discipline. A lot of otherwise intelligent people pretend to understand that distinction when they really don't and it leads to no end of problems.
I actually would go much further: I would say that the tension is between an applied engineering discipline and a vocational degree ala, well, plumbing.
I don't think there's much question that fields like quantum computing belong more in the "applied engineering" category (or even the "hard science" category). There's much more question about whether professional software development has any significant relation to the field of Computer Science other than the bare fundamentals.
My hunch in the future is that we'll see the field bifurcate into two very different professions, ala EE and electricians: "Computer Engineering" (or whatever it's called) will become a much more limited field that is overseen by the NSPE and has a licensure system, and "Software Development" will become a vocational degree (if that, even) and represent the vast majority of software development jobs we see today.
That obviously has problems, but I see it as an inevitable compromise that might eventually help resolve questions like, "this set of customer personal data was leaked because of a known but unpatched vulnerability, who signed off on this software release?" and also relives a lot of the pressure on regular software developers to understand these fundamental CS concepts that don't come up much in practice.
The conclusions he draws are questionable, too. Do you need AI powered search? Well, what if it's an elasticsearch plugin, takes 30 minutes to install, and increases your conversion rate by 5%? Should your engineers be required to know SQL and learn Redis on the job? Well, yes, they should.
If the microcosm you're in is CRUD systems, that argument may be valid, but I think the author is overgeneralizing if he equates CRUD to the sum total of everything there is to Software Engineering.
There is a constant lifecycle in this: A new idea is born or first becomes technologically possible. At that point it's very difficult to put the idea into reality, but the demand is so pressing that the market will pay whatever it takes for individuals to live up to the challenge. Then, gradually, it becomes commoditized and, eventually, everything that's left to do is "plumbing". ...but at that point, a new idea will have been born or will have become technologically possible for the first time ELSEWHERE, and the REAL MEN will have moved on to that (leaving behind only the QUICHE EATERS to do the plumbing work).
So, if you've been in the field for a long time, your career path might have looked like:
Writing CRUD systems on IBM host systems in the 80s, when that was still an intellectually demanding job, because you wouldn't have learned programming at university as programming wasn't yet a thing, but you still managed to learn what bytestreams to send down some line to make an IBM terminal show a number in red or green, and you did it in PL/2, a programming language that hadn't yet had a chance to learn the lessons learned from several decades of software systems gone wrong.
Writing some of the first pieces of internet banking software in the 90s. Your employer, the bank, wanted to make damn sure they were not going to make fools of themselves by using this new and unproven technology in a way that would allow funds to be misappropriated. So they paid you handsomely to learn everything you could about telecommunication technology, security engineering, encryption, etc because you actually had to manually code all that as there were no frameworks or libraries for that just yet.
Doing algorithmic trading software in the 00s. Everyone else was ripping databases out of their systems because XML was the future, and the XML was processed by a java programme to run in a virtual machine to run in a container to run in a virtual machine to run on god knows what. Since you still knew how to do real programming for real machines in a way that didn't involve converting your data to XML and back, your software executed orders to buy and sell equity on the world's major exchanges faster than anyone else's, and you were paid handsomely for it!
Doing data science & data engineering in the 10s. Organizations had been run for several decades by numbers-driven people whose computing competence only reached as far as Excel. The numbers-driven mindset did not go away. But the data was now there in such quantities, and stored in such nonsensical ways, that everyone who was previously an Excel-guy now needed one or two of those data engineers at their side. Also, because it now became possible to use the data to answer some pretty complex questions, those questions were now seriously being asked for the first time, and everyone discovered that they had been asleep at university during the lecture where someone explained what a standard deviation was. So that's why you learned everything you could about statistics, machine-learning etc etc. You applied it, and you got paid very well.
So, what really sets you apart if you're not just a software engineer, but a GOOD and SUCCESSFUL one, is that the stuff you're working on is always the stuff that's NOT YET plumbing, because it's a major force right here and now despite the fact that nobody has seen it coming.
...I don't know what the future holds, but I think I'll be well equipped to deal with it, because THAT's what real software engineers do and there's no framework to do THAT.
On a side-note: Stack Overflow developer survey results routinely indicate that like half of the nation of software developers have been writing code for a ridiculously small number of years (like two? don't remember the number now). If you're one of them: The above argument does not include you. You're not a software engineer yet in the sense of the argument that I'm making, above. But don't feel bad about that. Next year, you'll probably switch to sales or become the pointy-haired boss of some REAL software engineers.
Also what is that opinion based on?
Funny, a CTO thinks programmers are overpaid and our work isn't really that hard. Who would have thought?
If Mark Zuckerberg posts something to the effect of: "guys what we do isn't that hard, we should probably get paid less", I'd call him out on it too and I hope other people would as well. I certainly wouldn't consider that a personal attack.
There's a signup for a newsletter-- also before the content. The whole blog is marketing for his CTO advice company.
If someome sets up a marketing page, sells themself as an expert, and then posts a link to a resume backing up that expertise, it's fair game to click the link to the resume and verify.