Why Good Developers Write Bad Code: An Observational Case Study [pdf]
upedu.org
upedu.org
- Once a project is completed, the team must ensure that the “What” and “Why” of each software item are properly documented.
- In the cases of parallel development of inter-dependent software modules set up a negotiation table to solve conflict between the development teams.
- Make sure that the development team is aware of the CMMI-ACQ or ISO12207 processes for negotiating with third parties.
- Make sure that testers are involved when negotiating with a third party for a potentially vulnerable software component. 
- Plan organization-wide process reviews to detect isolated processes and to promote information flows between processes.
- Planned special budget items to support long lasting corrections or corrections that are likely to benefit many modules. 
- Projects with strict deadlines are risky, and should be carefully monitored to avoid last minute unplanned activities.
- Team members should maintain a careful balance between the flows of information within formal development processes and informal human interactions.
- Team members should make sure that knowledge is appropriately distributed amongst them. For example, pair programming is a practice which can promote knowledge sharing.
- Any intrusion into the team dynamics by outsiders should be done very carefully.
Scratch that, on MOST projects that's the case.
There was one project I saw that had a two line commit message, and there were 29,000 changed lines across 44 files. Who does that?
Of course, we can debate over and other whether mass-correcting existing files is actually helpful (one unofficial rule we have here is "only use the auto-formatter on code you've personally worked on or have taken over responsibility for", because it affects the blame history).
Also, I changed my name last year, and when I did that, I ran a mass find-and-replace to correct my credit in every Javadoc I'm credited on (I particularly detest my deadname, and I want it dead and buried), with the commit message being something like "Correct my credit to match my new name".
I'm currently working on the changes that will actually document what is otherwise a walk of code.
The road to hell is paved with good intentions.
However, my example was a project that was around for about 3 years at that point, and it was basically just labelled: "Upgrade to version 2.0".
Face → Palm
Yes, nobody cares about your commit messages until there's a bug or a major refactor. By then, if the commit messages suck, it's too late to do anything about it.
"Programs are meant to be read by humans and only incidentally for computers to execute"
Is never more true than when you're trying to fix bugs or make improvements. A project you can't improve is a dead project.
(edit: He made this change after a lot of discussion with a bunch of people, and he also sent out a mass email describing the changes to the structure, so he wasn't totally being irresponsible there.)
I'd assume the keys are the commit hashes? Now how to search github for them...
Code typically isn't rocket science. It's the human knowledge that goes into it that's irreplacable.
Example 1: OK, you're using a third-party CSV parser instead of the one built into the standard library. Why? If your code is crap but well-documented, I can read what you were thinking: "Using non-standard CSV parser because the standard one chokes on files bigger than 2gb" At that point I can refactor your code, or perhaps see that this issue has been fixed in a newer version of the standard library. Or maybe I realize that you confused gigabits and gigabytes and you made a bad choice in the first place, and I realize I can safely remove this dependency. But if your code is tight but undocumented... I would have no idea why this third-party library is being used unless I do some painful trial-and-error that still might not definitively answer the why.
Example 2: You inherit Mary's code. It calculates commissions for our salespeople. The code is sloppy and convoluted, because the sales guys change the commission formulas every month... and these changes have been happening for over ten years, often on very short notice, often contradicting basic assumptions made when the software was originally architected. But Mary documented every change. Which is good, because the fucking sales guys sure don't. Her code is literally the company's only coherent record in the entire company of the commission process. Remove her comments and commit messages, and none of the code would make sense, even if it was tightened up into a sounder codebase of seven modules with 300 LoC each instead of ten modules with 500 LoC each.
So yeah. Totally fictional choice but I'll take documentation every time. Code is just code, I can fix it.
(Both those examples are fictional, but I've been coding professionally for nearly twenty years and I've seen variations of them countless times...)
99% of the time what you want is to understand the current code--or at least code at a specific past point in time--as opposed to every transition that occurred.
For the CSV parser, I'd rather see a comment ("/* We use this for >2gb support */") or a test case ( testOverTwoGigsParseable() ) would be a lot more useful than any level of discipline over commit-messages.
For Mary's commission-calculator, it sounds like nobody has access to good "whys" anyway, because they boil down to "salesguy X insisted on it". Instead, the commits are functioning as an auditing/blame tool.
I got a new computer and when I was getting rid of my old one, it didn't occur to me to push all the commits or save the git repo. So, my next commit consisted of all the changes for that entire month.
I was able to retrieve the full final content from the place where it was being uploaded to, but that didn't have any git-related files.
If you make a commit, it would copy over with that, because it's written in that folder.
I'd wager that you simply copied the main files over and tried to re-commit.
"Issue #27" is terrible. Something like (in C#/VS) "Execute CodeMaid against solution" would be perfectly fine.
I miss having a system like that...
[1] https://help.github.com/articles/closing-issues-via-commit-m...
[2] https://confluence.atlassian.com/display/BITBUCKET/Resolve+i...
Some other examples: "Fixes to make work", "Left Over", and their most common message " ".
They're nice guys to work with, but their VC habits are awful.
No excuse, beyond it usually being a minor change well documented in the code ( I always write comments ) and me being utterly buried under work... You know, start the day with 20 things to do, crack off 4 of them and have 23 things in queue at the end of your day.
The "." was sort of a placeholder for "Fuck This, I'm ready to quit." ( and I did eventually )
But it can make for long commit messages. Why also includes what alternatives you considered, and why you didn't use those.
I also like to have a digital "worksheet" for each change I do, where all my thoughts and research goes. So if all else fails, I can reference that. But no-one else can, so I like to transfer as much knowledge as possible to the commit message. At the same time, some of these go on for 5 or more pages. They also tend to be very messy. I'm not sure if it's all appropriate for a commit message.
The whole point of a version control system is that it contains everything related to the code. On lots of web dev and game projects you also commit the finalized assets (the generated javascript from coffeescript, the compressed 3d textures, etc. etc.)
(Of course, the commit message refers to issue ID, and vice versa.)
There are man times when I have seen a crappy piece of code that I wrote a while ago, and decide to "tidy it up". The I test it, then remember there is a strange edge case that required me to write it the "crappy" way rather than the clean way. It always gets commented the second time.
Meh. All the Agile Manifesto really says about process is:
Individuals and interactions over processes and tools and Responding to change over following a plan
Nothing says you can't (or shouldn't) use a pre-defined process, or have a plan. If anything, the core of what "Agile" is, is about being flexible and responsive to change. As long as your process allows for that, it can be implemented as an Agile process even if it was created by a committee.
That's not to say that most firms using things like CMMI aren't doing it in a way that is far removed from Agile principles, but I blame that on the implementors more than the process. YMMV.
I'm very interested in the processes of developing good software so I hope there is some good discussion around this paper.
- In the cases of parallel development of inter-dependent software modules set up a negotiation table to solve conflict between the development teams. Role of the scrum master to remove impediments.
- Make sure that the development team is aware of the CMMI-ACQ or ISO12207 processes for negotiating with third parties. Previous bullet point.
- Make sure that testers are involved when negotiating with a third party for a potentially vulnerable software component.  Testers are stakeholders in this, they should be there.
- Plan organization-wide process reviews to detect isolated processes and to promote information flows between processes. Removal of impediments.
- Planned special budget items to support long lasting corrections or corrections that are likely to benefit many modules.  Got me on this one.
- Projects with strict deadlines are risky, and should be carefully monitored to avoid last minute unplanned activities. - Team members should maintain a careful balance between the flows of information within formal development processes and informal human interactions. - Team members should make sure that knowledge is appropriately distributed amongst them. For example, pair programming is a practice which can promote knowledge sharing. - Any intrusion into the team dynamics by outsiders should be done very carefully
These last four are key elements of Scrum - managing the burn down, be flexible, and ensuring chickens can not interrupt the pigs.
Agile does not mean there is no process or formal rules. It's not a free for all, agile is about the ability to quickly respond to change.
By nature, we programmers often judge our code on its technical merits. But a stinky piece of crap that I might write that shows me clearly what I need to build, or demonstrates the techniques that are required, or shows that something isn't going to work early, etc can have immense value.
Writing stinky code is a tool that every programmer can use. The trick is to know when you should use it and when you should not. If done properly, you can solve some intractable problems dealing with requirements gathering, design, or even political problems. When used improperly it can have exactly the opposite effect -- it pushes requirements gathering back, or defers important design decisions until it is too late, or creates conflict within the team.
I know some people in the industry who are pretty good at this, but I know of nobody who is a master at it. Not only do you need to be good at it personally, you need to be able to influence the team to coordinate their work in a way that is not destructive. This requires you to have impeccable taste, amazing technical ability and sublime communication skills. It is a skill that I wish more programmers would value and work towards.
I love the sheer insanity of this.
1. Write up a plan.
2. Execute the plan.
3. Total project failure. Abandon it.
4. Decide to try again.
5. "It will be easy this time! We've got an existing plan we can use!"
It's like using a treasure map, not finding any treasure, and now thinking it will be even easier to find the treasure now since you already have a map!
For example, they cite both schedule/budget pressure, and insufficient docs. The incomplete docs were "thousands" of pages. Does anyone really believe even half the team would read "complete" documentation, that are thousands upon thousands of pages? Would complete docs speed up development time? Would they speed up development time even taking into account the cost of producing and consuming complete docs? Is the size of module truly essential complexity, or is part of the problem that they're building on legacy code?
The author mentions interop with other teams and third party software as a large source of friction. Why are other modules so large and complex that they're maintained by separate teams? Is that essential complexity, or was it caused by previous attempts to patch their way to release? Would the modules be more manageable, and hence, require smaller teams, if they used other practices/languages/tools?
Access to test hardware, and managing the test personnel was another source of friction. What is the cost of buying more test hardware? In previous companies where I've worked, with manual QA depts and under-funded test hardware budgets, doubling the hardware budget would allow halving the QA salary budget (primarily because you don't need to hot-swap equipment several times a day), and shorten test cycles. Is that the case here?
Further, how much manual test is truly necessary, and how much is caused by the developers not writing automated tests?
- fear or lack of understanding of a legacy codebase
- fear or lack of understanding of a 3rd party component
- time pressure
- too many edge cases to handle
- intra-team communication issues or stylistic differences
- lack of domain experience to understand those edge cases
- lack of time to build the correct design, or rebuild a design if it won't fit
- project management pressure against big-design-up-front and to "start now" when making something more organized is required
- unclear requirements
- not interested in work or company anymore
- lack of use cases representing all possible usage scenarios
- it's actually not their code that was bad, but they are coupled to bad systems that are themselves bad/leaky/error-prone1. Lack of risk management 2. Poor project management 3. Allowing other to set my urgency (like deadlines but more psychological) 4. Not enforcing professional practises I know I should follow
3 and 4 are to all intents and purposes symptoms of 1 and 2
Which is more or less the same thing, except that IMHO it does not suggest that every bad code shall be "good" for someone. There's such thing as code that is universally considered as awful (but for their creators, maybe).
As an example: sales promises new customer feature X in one month and we will lose money unless it's implemented.
My choice is to either: 1) Complete it the right way in double the time or 2) use hacks and cut corners to get it done on time. Since non-technical people just see the output and not what's going on underneath, they often times don't see the difference and when they explain to the boss that they might lose money, it's almost always option #2.
I hated being forced to do this so many times in my career, I started my own company.
It depends a lot on what your company is doing, but for the most part, you'll probably discover that cutting corners to save your short-term time is often the right thing to do.
It's right to the biz side of the house, because they have no idea what the costs of those cut corners are. They just know that they asked for shiny feature X, and then that shiny feature X was delivered, albeit with a complaint that the codebase was now Y harder to work on.
Rinse and repeat, and their interface is still "Oh, well, if I ask, the engineers will make it happen".
As this goes on, though, the engineers incur costs and suffering in order to keep delivering. And for any developer worth their salt, there is a very real cost--I almost want to say psychic and emotional trauma--associated with working on bad code.
And these engineers? Odds are, they have no big stake in whether or not the feature works. They're paid (badly) the same way either way, and don't get any benefit whether or not the new feature works.
So, they'll just leave when the continual cost of dealing with a shitty codebase outweighs the benefits of staying at that company. And they'll probably be rewarded with more money for switching!
The incentives are all wrong, and then one day your sales folks realize that any request suddenly takes waaay longer than is acceptable.
All too often the short-sighted "quickest" way forward is through copy-paste some code that does something remotely similar and massage it into giving the desired output. Sometimes this is necessary, but a lot of times it is not. And every time it leads to technical debt. I've worked on a lot of code bases like this: "Copy file x, rename copy, add copy to project file, rename global names, massage into y, rinse repeat". And then you find a bug in x.
Developers are typically too far removed from customers to ever know, but often the customer is just using time as a bargaining chip. Everybody wants all their needs filled right this instance but most people are reasonable enough to know that the universe doesn't work that way. You just need to gently remind them of it and they will respect you all the more for it, at least if you then deliver quality on time.
Of course this is harder with customers who believe they know a thing or two about development. In their mind it's just change the output of x, but what they don't know is that because you cut corners in the past, you'll need to change the output in y, z, x2, x3, y2, z3 and zz as well...
I also don't over promise features in an un-realistic amount of time.
To anyone except an MBA running a software project for the first or second time, that's pretty obviously not true. (Some people, when they realize this, do "additional process will be mandated until quality improves".)
This is usually one of the fastest ways in an interview to tell how experienced someone is. Younger, less experienced programmers are dogmatic, but their view is extremely limited - they believe defiantly in the current coding practices, but haven't yet learned that their dogma is just the latest, and will be replaced in short order.
Hey, I was once one of those dogmatic kids, too.
You're right though. And it's a balance between technical debt incurred having not enough time (hacky shortcuts) vs technical debt incurred from having full freedom (overengineering of unnecessary layered abstractions and indirection).
It's a tightrope and we're always going to stumble somehow. In retrospect, we tend to focus on the stumbles (wouldn't do that now!) rather than the fact that we actually got across the canyon and delivered business value. A natural bias - we discount the fact that we're making new mistakes even as we look back and criticize our past mistakes.