The Quiet Crisis Unfolding in Software Development
medium.com
medium.com
This does not match my experience in ~10 years as a web developer. I see most of the technical debt coming from low performers, who take longer to deliver a poorer product. Most high-performers perform well precisely because they have mastered certain development processes, tools and good coding practices, which results in higher quality code. I'm thinking this must vary a lot according to your industry.
The author's claim is almost self-contradictory: if technical debt is bad, then anyone using it to create the illusion of being a high performer will end up hamstrung by his accumulated debt.
To give you an example I have been working for over 10 years at the same company and have to support today infrastructure code I wrote 6-7 years ago, and the previous company I worked at I worked for 5 years in a similar environment.
You can bet anything I write is VERY well unit tested and VERY easily understandable, and am extremely careful about technical debt and will push back quite strongly if asked to incur it "just for this ad-hoc fix/feature/deliverable", but that comes from a career spent with a certain mindset, when I was fresh out of university the words "unit tests" and "technical debt" were not part of my lexicon, I had a lot of algorithmic knowledge, but in a university environment you aren't drilled nearly as much as you should be on good code hygiene and practices.
If it was me in universities I would have a course in your 1st year where you have to develop some sort of application, and in your 5th year a follow-up course where you have to extend your original application and a classmate's in some significant way, that would drive the technical debt and writing code/docs/comment so you/others can understand later points very well.
And I would make it very clear over and over again that it's not the case that the less lines of code you write the faster it's executed, and that often the most compact solution ends up being the slower and less maintainable one.
I was the one who couldn't sleep at night seeing all that debt and every now and then would clean things up. At least he would thank me for it :)
I also suspect that many of these "high performers" are very aware of the technical debt they are adding, and simply prioritize shipping over anything else. To them, the mess is just a fact of software development, to be cleaned up only as can be justified by the needs of the business.
In general I don't actually agree with this view, but I also try not to dismiss it outright. (My counterargument usually involves the fact that technical debt will slow you down in the mid-to-long term, so if there is any expectation of keeping code around, it actually saves the business money to invest up front in code quality.)
Nobody ever wants to do it, but I've used a statement of work approach in the past.
I once used the wrong version of coding standards in a shop, and heard about it in a performance review. I'm in no way convinced this was inadvertent. My piece had negative defects - I found a dozen or so old crufty bugs in my testing. The rest of the team took months getting integration right.
OTOH, I am constantly worried I am too slow (the legend of the 10x developer does not help) and tend to omit test cases or leave code in that works but is not easy to read or understand.
At least I use every time I find to refactor code later, but we've had some bugs in UAT which shouldn't have been there.
He kind of mentions it, but in a very early-stage startup, the ability to deliver MVPs quickly (regardless of code quality) may be seen as a positive trait. Of course, even better is when you can ship quality code just as quickly, but such developers are probably the exception to the rule.
One of them is to value support and firefighting very highly. The best firefighter is often one that is also the most familiar with the code, so it's common to see amazing firefighters that are also your arsonists! But when you don't track where the bugs come from well, and have little code ownership, whoever writes the most code writes the most bugs, does more firefighting later, learns more about the system, and distances themselves from the rest of your developers that actually want the system to work. This also increases the amount of tribal knowledge (as systems get quirky and badly documented), making it harder for anyone else to seem like a high performer, as to make any changes to anything that isn't new, you have to learn how the rest of the snakepit works. And as you learn how it works, it gets bigger, because the "high performers" are creating debt far faster than anyone else can learn it.
If bugs are treated seriously, tests are treated seriously, and how easy people join a team is considered an evaluation criteria for the existing team members, not the new ones, things are different.
I'm experiencing this first-hand now. One colleague is writing lots of code very fast. Problem is, this code is far from perfect, and I spend a lot of time to understand it, and I find many bugs quite often. In the meantime while I'm reviewing, the guy writes even more code...
Piggy-backing off of this astute suggestion, I'd also like to say that it signals to other developers in the org that the org appreciates and rewards people stepping up. This can be hugely important for retaining talent -- it's the difference between whether an employee thinks they have somewhere to go in their current org, or if they have to leave to step up.
I currently work on a project where one of the most senior guys was an incredibly smart person. He wrote extremely complex parts of the code that "just work". The trouble is he's now gone from the team, and no one understands why certain things are written the way they are, and people are afraid to touch them.
Despite being really smart, this person was not very good at communicating. Your typical shy and introverted little genius who does it all on his own. When asked to explain things, he would just say "trust me, it has to be this way", but wouldn't really explain why he made certain decisions or wrote the code in certain ways.
So now we're in a situation where there are parts of the code we can't touch because "he said it had to be this way". Great. I call it personality-oriented engineering. A total failure.
This person got to a higher position because management only looked at how smart he was and how he was able to deliver complex solutions that would impress everyone. They failed to realize how bad he was at leaving a legacy of maintainable code that can be moved forward.
Even following those two rules alone, you can avoid building these complex, black-box solutions that nobody is willing to touch.
Who writes unmaintainable, unreadable code? Not my definition of genius at all. I worked with one genius in my 20 year career. He was a marine biologist and knew more about computers than me. He was scary smart and all the advice I had from other people about working with him was - he thinks in steps 1, 6 and 20. You need to get him to explain how he got from 1 to 6, and then how he got from 6 to 20. Often this would take weeks to get out of him, but the though had just popped into his head (often while working on something else.) Very interesting man. One of his quirks was the he had no idea what time of day it was. We had to tell him to go home or eat his lunch, else he'd just think it was perpetually "about half nine"
If I was his manager, "it just has to be this way" wouldn't work, but obviously different strokes.
You know what? That just means you haven't shrugged his influence off. He's still casting that long shadow.
When our local genius left, I felt lost but eventually stepped up, went through his code line by line documenting it, wrote enough tests so that I could be sure that I wouldn't introduce bugs while rewriting his code to make it readable for me. I learned quite a bit from that.
They should not mystify him, they should have made sure he shares knowledge with others and brings other up to speed. And as he is gone, they should make sure the code is refactored to suit the documentation level and code style the remaining devs are used to.
Is not it a subset of "high performers"? The dirty secret of all such "genius developers" is exactly this - they all simply traded rigour and quality for a speedy spitting out of a "good enough" code.
For the outsider this pace may look like magic, for someone who is left to maintain the mess it looks, well, like a mess.
Why do we expect that someone who is fantastic at twiddling bits, to also be good at documentation? It's like saying to a Program Manager "great, you specified how it has to work, so now just take a few hours and code everything up". Or "how come you weren't coding as you went along?"
The solution is to pair these 'unbalanced' developers with people who are good in other areas. Where was the technical writer who was talking with him on a near daily basis? Or the SDET who worked with him to make sure the important aspects of his design were codified into tests?
You can have 10 reasonable devs spend a year getting something to work and have it be reasonably maintainable/documented/tested/etc. (If you actually even have 10 reasonable devs in the first place). Or you can have 1 'genius' do something fantastic in 6 months with minimal support and hope that your other devs can fill in the gaps.
Remember, you don't go to war with the army you want--you can't say "this person would be great if only they __". You can only work with them or not work with them. If they are genuinely doing what they think is best for the company, then I recommend you figure out how to work with them. Their challenge is to figure out how to be effective in an organization that doesn't understand, and your challenge is to figure out how to make an organization that can understand.
There is such a thing as domain expertise. For example, machine learning, compiler design, graphics programming, etc. If you are say, a web developer these people might seem like geniuses to you, and no, you wont' be able to pick up and understand their code. This isn't because it's not documented well enough, or that it's not written well, it's because you lack the domain knowledge to understand it. For example, I've written code that you absolutely will not understand without weeks or more of study in the domain. The code is documented, and has links to science papers that you need to read to understand it. But know that you may need to spend months to learn enough to really know it.
It's not the responsibility of the domain expert to teach you an entire domain. So yes, they may just tell you to trust them instead of telling you to go study some volume of domain work. It would be better if they did the latter, but I suspect that many here would still complain with that answer.
I still wish more of my teammates (past & present) wrote more comments (as in "statement of purpose", at the top of a chunk, not just "this part is complicated", in the middle here and there) in their code, though. :-)
Anecdata: As somebody who did actual paid work in an interpreted / dynamic language in the late 80s during and just after college, I seem to see people who have only been through the C ... Java regime struggling with more "modern" (? alternate, anyway) dynamic / functional scripting type languages. It's just not something they are familiar with (intuitive = familiar).
Not that CSU did a great job teaching FP in the 80s, but we at least got to play with original flavor Lisp a few times...
This drives me nuts, it manifests itself in some many different ways: * The "immediate" thumbs up on a PR * The PR that sits and languishes forever * The chat room of silence when someone asks an architectural/design question
It can get really frustrating. Still no clue of a good way to fix it since the problem stems from a long built team culture that needs to change.
> The "immediate" thumbs up on a PR
This one bugs me too. Really? On a PR with 400 delta lines, you didn't need clarification on anything? I don't know what action to take here.
> The PR that sits and languishes forever
Get your team to agree to an PR review SLA on the order of a day or two. "Enforce" the SLA -- how you do this depends on your team and the situation, but it can help to just remind people that they collectively agreed to the SLA.
Suggest reviewing PRs with the same cadence as checking email (~1-2x per day).
> The chat room of silence when someone asks an architectural/design question
Identify one or more "natural leaders" as mentioned in the article, and encourage them to check on the chat room periodically (email cadence?) to route questions to the most likely person. They should know who that person is because they're generally well-connected, and mentioning them by name creates social pressure for that person to respond.
I typically ask for the devs to push their code into the repo incrementally as long as the change doesn't break anything. That way you have manageable prs and if there's a need to go back to the previous version you have smaller deltas to deal with as well.
That's fine provided each of those PRs gets reviewed and merged almost immediately but if you don't do that then either the developer has to create a separate branch for each incremental improvement (horrible) or let them mount up into a monster pull request (also horrible).
Some changes also kind of need to be large in order to not break anything (deep architectural changes).
My thinking is that, if I approved a PR you wrote and it had a bug, I was as much to blame for the bug as you. That should help with frivolous approvals, but I've never had much of a problem with them, so take it with a grain of salt.
The best code review flow I've seen is where you put in a review, assign it to a specific person or two, and then merge that code.
What my current company has is the godawful github PR synchronous flow that everyone seems to love. This allows a few individuals to hold your commits hostage unless you bend over and implement their pet style. It pisses me off on a daily basis, and I'll probably quit in another month or two.
Except for the last point:
> but a nice pair of jeans and any decent shirt with a collar should do just fine.
Why should there be such a requirement? Why can't people wear shorts and/or t-shirts if they want? Of course there should be some requirement (like having some kind of clothes on) but otherwise I don't think any code is necessary as long as the team is comfortable.
1) company that creates custom modules for firms coal software; it always involves meeting the customer. 2) Open office layouts where customers are given a tour of the Office.
On the other hand, at one place I worked it would have been illegal for the customer to interact with us in our office space, but we still had to wear formal office clothing.
Consider
Case A: Customer knows you well understands you are competent. If you want to leave or are laid off that customer might eityher hire you or know someone who does. Or at least will give you a solid reference.
Compare with.
Case B: Customers and others only know you through your boss.
Why? With clients you wear more formal clothes for PR purposes. It shouldn't be necessary on most dev teams (in my experience).
I think the same applies in the given example: the in-group knows the guy in the shorts and T-shirt is a nice dude who helps others and is super-professional about quality. The customer, as an outsider, just sees someone who's underdressed.
Not to mention having to hear the same old jokes every time the customers go look at the monkeys.
Unfortunately, the company I currently work for has begun adopting and exhibits many of the behaviours and poor choices this article warns against. And is actually on a path to employ more of them, quelling any objection that might come up.
Sorry, it's just frustrating watching a company make such consistent poor choices.
I once sat next to a software developer who always listened to talk shows/lectures while developing. I then resented having to maintain or add features to code he touched.
When I started working from home, I can just tell myself that he's focused and trying his best (which may have always been the case). It still looks like code developed by someone who is distracted but I no longer know that for a fact.
If anything, I communicate more when I'm at home than when I'm in the office... Excessively verbal people don't seem to be able to handle it well, however, from what I've seen.
> Optimizing for the common scenario means giving developers primary ownership of portions of the code base and not allowing other developers to change that code without approval from the primary owner. This results in higher productivity since any given developer is more likely to be familiar with the code in which they will typically be working.
I have worked with someone overly protective of the code sector he wrote, but it has caused problems since I would like to fix some bugs in the API signature of that code, but the developer has stonewalled me in the past & raised a ruckus when I filed PRs to fix some of it. If it is something that affects how other people interface with a feature, IMO it is not a good thing to necessarily designate a primary owner & mandate all changes go through him/her. Granted, some of the problem is that the other developer probably was in the wrong being obstinate, but you work with the cards you're given.
In some ways, it might then almost be better to not have that person contribute to same codebase: to avoid ego pissing contests and to keep the opposing pressures in balance. The pressures being: delivering features on time on one hand, and keeping things clean and orderly on the other hand. Often, if the architect codes then they can't see their own mistakes, but have 20/20 vision of others' grubby flaws, and hence gain subconscious animosity towards external contributions.
That being said, an architect who does not code is even worse! This is where we get ivory tower Architect-as-title, and even less gets done. I wonder what the optimal middle ground is.
Perhaps they lay out a system design as a suggestion on a wiki and ask their teammates including themselves to collaborate on it until everyone's content with it.
Then perhaps they code a particular layer primarily and are humble and open to criticisms and mistakes they make in their own coding. They understand that they and others are not perfect, nor is the code they or others produce.
That and good relationships and communication with the product, design, management, qa, and support teams are what I feel would define a valuable "Software Architect".
Yep. Perhaps this could be softened a bit to say there are different sorts of devs, some seem to work OK in open plan offices and some need private work space. But personally I'm in the latter group. My company moved to open office space and I've come to not even think of the office as a place where I get real work done. Thankfully they allow me to work from home as well, that's one of the main things I'll be looking for in future employers. I have to admit I can produce a feeling of moderately intense anger by just thinking of the Yahoo executive banning work from home because she believed that everyone's job involved frequent impromptu hallway chats or whatever the fuck.
"For some of them that’s jeans and a basic collared shirt and for others it’s business casual."
Which for the author's arbitrary reasons leaves out jeans and a t-shirt - although this guy makes some good points, this one is a bust.
This is repeated over and over in almost every management book; managers are starting to believe it. In the book "The Phoenix Project", Brett, the most technical guy, was the high performer to fix almost everything. The moral in that story was that Brett was also the Fire Starter. No, management need to feel high performers below them actually belong there. Unfortunately technical skills fade and once good managers become bad and incompetent. Worse, most technical managers were never technical. Just this week I had to put out a fire most certainly not caused by me or my code. I inherited a mess from outsourced workers. Any requests to improve the code base were met with disapproval. Some of our custom code interfaces with an external system that has an API. When we upgraded to the latest version of the external system; it became more strict on what it allowed. To help with the migration, management paid many thousands to have three "experts" on the system come in and help make transition to the new version go smoothly. I was not on the migration project. However, when sh@t hit the fan I was called in. Testing was poor and they went live. The three experts could not find the cause of major problems and certainly no one else on the team was willing to step up. I had to spend the entire week becoming an expert on the external system and quickly fix the problem. Yes, I am a high performer and I did not cause the fire. It is because I roll up my sleeves, get to work and never give up. I'm sorry if that scares some management. That they actually need some people.
I would've added a paragraph on stack ranking and curve ranking in general, which although obviously extremely detrimental to morale and retention, still unfortunately rears its head at times, and one about how to handle layoffs/restructurings as those are not easy to do well.
This happened in medical care (the rise of non-medical medical management), academia (c.f. the rise of administrative positions in university), charter schools and industry, too.
The problems he complains about are consequences of this more general shift - when managers do not understand what it is they manage, they seek out brain dead metrics to measure performance, which, in the absence of understanding the field, severely damages performance.
Programmers have more in common with welders, doctors, nurses, lawyers, teachers, etc. than they seem to think.
"... I’ve noticed a quiet crisis unfolding in software development leading to low quality applications, unhappy employees and unhappy users. Silver bullet solutions keep creeping into our awareness (Scrum, anyone?) and predictably keep letting us down.
This is almost entirely the fault poor management — or perhaps it should be called fad management."Fads will always exist as long as someone benefits from selling the fad. And there is always someone selling shovels in a gold rush.
There is a real lack of leadership in the world.
Everybody is so averse to risk and afraid to follow their own beliefs that they end up doing what everybody else is doing. That is not what leaders do. This is how we end up with scrum. It's also how we end up with partisan politics, a dozen sequels of the same movie, and nickelback.
It seems like the only ways to do this are
1) Create a startup, ship something awesome, have a great exit 2) Work for wall street, in a position where your success is judged by how much money you make, which directly drives your bonus.
I'm not sure I do. Shipping software in a mess of technical debt is much slower than greenfield development or development on a well architected product.
I'd ultimately want to be judged by an architect who reads all pull requests and understands at a deep level how all of the code fits together.
Most often I've observed that the more process and organization surrounds a development effort, the more likely that effort will outright fail (sometimes by never getting completed, more often by never actually fulfilling the function that was designed for, then abandoned. The difference is that the second kind is very much declared complete).
Recently a thorough focus on code hygiene (unit tests taken to a ridiculous levels, for instance) is one more warning sign that it's time to find something else to do. The biggest warning signs are project teams way bigger than they need to be, but ... Code hygiene nazis merely join the documentation madness of a few years back, PMs should control everything because Steve Jobs was a successful asshole, we need 5 design docs before even knowing what the problem is, ...
- Burndown charts: Burn lots of tasks fast, even if you are creating more work in the process.
- IBM once rated employees based on added lines of code. The result? An explosion of code. Even code that wasn't strictly necessary.
- Stack ranking: punish the lowest 20%, reward the top 20%. Sounds reasonable at first, but what happens when employees adapt to it? People stop hiring strong candidates. So over time, when people leave, their replacements get worse over time creating a downwards spiral of worsening talent.
My view is also probably skewed compared to what a typical "software development" company does, but I don't believe that Google or YC startups do this kind of management.
Sounds more like a prison cell. How about daylight or a window?
There's now a social network for developer rants: https://www.devrant.io
However I think these days, at the higher end of the spectrum, the general awareness of good programming practices is better than it was 25 years ago. Back then, unit testing frameworks weren't a thing, I didn't see any CI, often code repositories were zips on a file share..., patterns were barely a thing....