Why don’t software development methodologies work?

Ross: The most deadly thing in software is the concept, which almost universally seems to be followed, that you are going to specify what you are going to do, and then do it. And that is where most of our troubles come from.

Fraser: One of the problems that is central to the software production process is to identify the nature of progress and to find some way of measuring it. Only one thing seems to be clear just now. It is that program construction is not always a simple progression in which each act of assembly represents a distinct forward step and that the final product can be described simply as the sum of many sub-assemblies.

(Full transcript of the 1968 conference here. http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1968.PD... It's a really fun read!)

Here's my methodology.

* Figure out what the software does, what the business cares about, and what the users care about.

* Figure out what it kinda sorta needs to be under the covers

* Write prototypes of the technically gnarly bits.

* Flesh out the prototype and start writing integration/functional tests for the system.

* Ensure CI server is live, running integration/functional/unit tests and build documents on push.

* Primary work phase: code review, write tests, write code as you go along. Figure out if features are needed or not. Don't overdesign or underdesign.

* Document subsystems as they stabilize using comments and in-repo text files.

* End of project: everything is tested, documented, and working.

Rinse, repeat.

In contrast, I've been on SCRUMed and Agiled projects where the resulting overhead on those who can already get work done without SCRUM Master Jar Jar's constant interruptions reduces them to people who quit within a few months.

But hey, acqui-hires rock and I can't imagine a better way to drive the talent out of a stable Fortune 500 company in order to take a chance at a startup than by driving them $%!@ing crazy and reducing their productivity to jack diddly. Ah the cycle of life...

That said, I see the use of methodologies where the talent is both junior and mediocre. But the former is better addressed with good mentoring (ha ha just joking - mentoring is for wimps, am I right? am I right?) and the latter by not hiring the mediocre in the first place (which means HR needs to be disrupted stat and since they hold all the cards they won't be - cycle of life again).

Sing it, brother! Can I get an "Amen"?

I know of one project with a technical lead who, to my knowledge, has never done anything other than support the vendor program the project is replacing; the team has developed a process built primarily around avoiding Subversion merges (which is not necessarily a bad thing given that they seem to actively resist learning anything about how to use Subversion) and secondarily around adopting anything anyone has ever described as a "best practice", including inventing a few new ones. So far, I do not know of anything their project actually does, although there is a great deal of it.

Oh, and their scrum meetings seem to involve the scrum master reading a fair amount of text from PowerPoint slides (which have far too much text on them).

[Edit] Apparently, I can. http://typicalprogrammer.com/why-dont-software-development-m...

How many people are over weight, who know that all you have to do is eat less, work out and then you will get in shape? It's that simple. Yet because of emotions, eating due to emotion, happy eating, sad eating, or poor self image many people don't get in shape. Likewise many students understand that all you have to do to get great grades is to avoid procrastinating, just start studying on time, study hard, study more, do most of the exercises in your text book, use multiple books and you are most likely to pass with high grades, yet lots of students put it off, poor will power, delayed gratification, it's more fun to goof off.

Same things apply to software, a lot of us know exactly what it takes to make good software, to spec it out, to plan a good architecture, to write good test units, to comment and document the code, to organize the process, to avoid over optimization, to avoid changing and adding lots of new features in the middle.

Yet, a lot of us don't do that, we start writing code before we even code before we spec out, because its exciting! Our emotions in play, we don't practice that delayed gratification of holding off and writing specs. We plan to throw away this prototype, but then somehow, it ends up being what everything is built on. We plan to refactor one day, but feature creep never allows for that. We know we ought to comment, but we understand the code now and don't, then 6 months later we are cussing and kicking the wall. Test units are boring, so we write as little of it as possible.

Until we take into account that programmers are humans, with emotions and different level of discipline, motivations and abilities our software development methodologies will keep to fail us for most projects.

Just my opinion.

I've also been on teams that were not successful no matter what methodology they used. The team just wasn't cohesive, didn't have enough talent, and/or had poor leadership.

The software we create today is tremendously more complex than the software we created back when I started in the 1990s. Part of that is standing on the shoulders of giants (and the libraries of giants), but part of that is also process.

The amount of process required for clear, effective communication increases with the number of parties involved. Creating good interfaces between components and the teams responsible for them is really a very difficult problem. If you're working on a three person team, it's easy. If there are 30, it's much harder. If it's 100, you probably can't even know every single person involved, much less avoid stepping on each other, not without a lot of process.

So what process does is increase the potential scope of a problem above what a small team can do (mostly by breaking it down into several interacting subprojects). That's difficult, important work, even if it's outside the realm of the average HN startup's imaginations and experience.

Solutions to this are tough. My first suggestion would be to use languages which facilitate correctness, although it's usually at the expense of developer availability: the pool of engineers with experience and know-how in true FP is orders of magnitude smaller than more pervasive languages. My second thought is to further embrace math as the building block for non-trivial applications: mathematical proofs have real, quantifiable value in correctness. I find it no surprise that the larger companies have made foundational maths, such as category theory and abstract algebra, the underlying abstraction for their general frameworks. This is even a tougher pitch than the first since most engineers don't recognize what they're doing as math at all - a big part of the problem. So many of us are doing by feel what has already been formally codified in other disciplines.

I'm aware that both require more (not necessarily formal) education than most engineers have pursued and makes it a difficult short-term pitch point for any company, but I think if we're serious about eliminating sources non-determinism from projects, it's important we address them directly.

Restricting a creative process into some pre-baked confines of thinking doesn't make a sense to me. Some rules are vital thought those should not get too much in the way.

I think only organically growing your rules makes sense and you've got to accept there will be situations where you have to break them or they won't allow you to adjust your strategy. Imagine a huge company that standardized on some well-tested processes/methodologies that served well in the past and were adjusted as it was growing. Staying with the same set of methodologies might kill the company's future if the structure of its workforce changes, technology moves forward, competitor has more efficient methodology for achieving returns etc. Often methodologies are used internally for political gains, pursuing promotions and power and "change agents" don't really care that much about improving efficiency, instead riding the wave of a currently cool methodology as recognized by the upper management.

Like almost any other skill, "Social Skills" are developed through practice, and they also degrade when they aren't used. People that spend a majority of their time on a computer may struggle when communicating face-to-face. (I've also seen it happen with stay-at-home parents with young children.)

What good project managers do is much harder than implementing a Scrum method. It's finding good people, making the hard decisions of removing the bad people and get / make a clear understanding of product scope and goals.

The last point in particular should be a well duh, but in reality it's ridiculously common to encounter projects where the PM (and hence even the customer), have a very vague understanding of what's actually being built. Those projects are guaranteed to go massively over time and budget, or being outright cancelled.

Good process is just the icing on the cake. It should be used, process alone does not make a project success.

--Mike Tyson

Furthermore, if you have a very capable team, you may not even need a formal "Methodology".

And that's crazy. Software delivery is at least 50% design/problem solving - and those aer neither completely predictable or repeatable.

To me a software methodology works if it _improves_ delivery. If the result with the methodology is more predictable, and less likely to go horribly off the rails than if you'd used a different methodology (or none at all) then it's a good methodology.

Looking for something that can perfectly transform any human requirement into code, in a predictable manner, is just silly.

I suspect it's the author that has changed, not everyone else. It's probably a combination of a nostalgic bias combined with the author's increasing age making it harder to get on with a typical team of young whippersnappers.

I say this as someone who is over forty and regularly falling into the "things were better in my day" trap (although hopefully self-aware enough to see it).

Problem is that with agile (as I understand and mildly practice it at least) you can't have "identical requirements", because you don't have full, complete and detailed requirements upfront.

Does a good team deliver solid, quality code? What if it's code the user doesn't want? Conversely, what if the team excels at delivering what the user wants, and they love it, but the code is so buggy that it only works 50% of the time? Would that be better than a solid app the user hates? (Probably yes)

Should a team working inside a large company deliver faster than the organization is able to accept change?

Just what is a good project, anyway? One in which we all had a good time and thought we did a great job? Or one in which the person paying us thought we were awesome?

No matter the criteria, everybody seems to agree that having good people is something like 70-80% of the secret. The big debate is what goes into that other 20-30%

ADD: An interesting thought experiment to play here is to posit that the team sucks -- wrong guys, wrong personalities, whatever. In that case, what would you want to happen? The answer to that should be an important part of whatever you want your process to be. Explicitly define your failure mode. (Because failure is still more often the norm than success in software development)

A simple search proves this is false. Paired with author's other unsupported and stupendous claims I have to doubt the validity and worth of anything else author says.

1) You can't get around that things don't work as planned, and too many methodologies assume that things will.

2) It is better to have a PM with A level content and B level process, than the other way around. Professional PMs don't have the intuition to solve the daily project problems, but they're the ones selling the methodologies.

3) Too much focus on tools. Using MS Project to keep the plan doesn't work when 90% of the project doesn't know how to use it.

4) Too much change. Rather than settling on a decent methodology that everyone agrees to, companies keep re-inventing their process every few years.

5) Wrong methodology for the wrong purpose. Waterfall methodologies don't work in research. Agile methodologies aren't the "fits every project" panacea.

I can go on, but you can't enforce excellence with formality. The formality can help outstanding people achieve excellence though.

"It’s common now for me to get involved in a project that seems to have no adult supervision". But my bigger problem is that the youth of today seem to have a much bigger ego than the youth of yesterday.

"Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion". Recently working with an off shore team, they called their tech lead literally "God". And he was pontificating procedures and styles left and right without understanding the core of the software first. He stopped when the product start to have big performances problems. One of the ones that I liked the most was his prohibition to use class indexers in C#, backed up by an example in Java.

I get why process exists, I know management sees it as a way to make software development less lumpy; to bring below average teams up to average productivity, but it isn't a one way lift. Extensive process might raise the below average toward the average, but it can also lower the above average toward the average.

Twice I've seen situations where a team of better than average developers had no really well-defined process (though not surprisingly a sort of home grown process evolved to suit everyone's needs -- agile, as opposed to Agile, if you will) that was highly productive suddenly have process dropped on them from on-high in an undeniably productivity-killing way.

One time it was because higher-ups at the company randomly decided they needed to standardize on the Rational tools (ClearCase, Rose, etc... still have nightmares about that stuff) and out of that insane decision we ended up with some stupidly strictly defined RUP-based process to tie everything together.

The other was when a team was transitioned from one company to another and the other was one of those stupidly-strict "Agile" shops that fully drank the ritual kool-aid and adopted basically every suggested "Agile" strategy they could slap together without giving much thought to the actual original ideals of why "Agile" came about.

I guess the takeaway is be really selective about introducing new process. It might entirely make sense if your project is building some mostly throwaway CRUD app for an internal company department and all of your developers are, well, the kind of developers you can find who will work on such things and the project is off-track. Or even if your developers are all good, but the project is failing for other reasons like lack of cohesive vision. A little bit of process introduced sparingly might help.

But if you have a highly productive team already, don't fool yourself into thinking that because process "improves" software development that adding more of it will make your highly productive team even more productive. Because it very likely will do the opposite.

First of all, creating beautiful furniture takes LOTS of time, as it rests in attention to detail and an uncompromising position towards quality. When you're uncompromising about quality, you require yourself to throw out, to waste, things that are of lesser quality.

Second of all, beginner, poorly trained, and dispassionate employees are never going to produce beautiful furniture. Mostly because of the above: they either lack the ability or the caring to have the attention to detail necessary to do great work.

And the best carpenters aren't going to come work for you unless you're willing to make it worth their time. Coming to work for you means they will have to do a lot more work in a much less comfortable environment than they are used to. Because the best carpenters have their own shop, their own tools, and work on their own time, because they love it. Going anywhere that is not their own setup is automatically worse for them.

It's the same thing in software: great software takes patience, it takes time, and it takes money to convince the best programmers that they should be spending their time on you rather than on themselves.

But at no point does any of that mean that, because Tool X can't magically turn your mediocre working into a stellar one, that means that Tool X is not worth studying. That is completely, 100% backwards. The master practitioner studies all tools, even the ones he or she doesn't like, at the very least to understand what is wrong with them and why they don't produce the results they want. A master carpenter might prefer a hand saw over an electric one because the electric saw chips the corners of the board too easily, or something. But he doesn't know that unless he's studied the electric saw.

You can't have "an absence of methodology" any more than a wood shop can have "an absence of tools". It has to be there. But they all have their pros and cons, and non of them has a pro that includes "makes men out of mice."

How do you know something is developed in a timely fashion, for example? All we really have is a gut feeling.

We could implement the same project multiple times and use a different methodology each time. Then compare. But you'd want to use people who understand the methodology and care about winning the competition.

Even then, I bet if you ran that experiment with 5 different projects, there'd probably be a different outcome every time.

My goal is to provide environment where clients would feel their input is steering project and they are getting results they want, also that developers and others working on a project would like clear direction and ability to show their best skills. Sounds a lot, I know.

Ultimately, it's more about the people involved. Bad teams are going to find a way to screw themselves over no matter what process they use. Good teams don't need a rigid process since they'll just find a way to get stuff done no matter what.

http://www.joelonsoftware.com/articles/fog0000000069.html

I've been a web developer for 13 years. Following a specific methodology doesn't matter, as much as focusing on the important aspects of the project, which are the customers, and what difficult problem the product solves.

The funny thing is that even if a team does this, it is not guarantee of success. If the customer base is too small, or there are lots of competitors, even the best software will fail. Conversely, if the customer base is massive, and there are few competitors, even poor quality software can succeed. Luckily for the software industry, and not for consumers, most teams do not follow these guidelines with the net effect being a TON of poor quality software.

Obviously here, Stackoverflow, and various other programmer focused sites, it's common to focus only on the coding side of the equation. The reality is that the best software comes from the collaboration between customers, designers, programmers, testers, usability experts, and sales. When each group brings their strengths to the table and focus on a common goal and solution, everyone wins. However it's very rare, since often the programmers are seen as the builders, the designers as the painters, testers as a nice to have, usability as a fad, and sales as being helpless.

To clarify a bit: design != usability, functional != usable. A beautiful design does not mean usable, look at the very confusing swipe action based calculators on iPhone. Functional is an auditorium with 20+ projects, usable is having the specific video cable dongle.

It does not work when it’s used as a magic machine of happy fun time productivity…. Sprints and stories don’t translate into more productivity. Focus and teamwork do.

The advice often comes (including in one of the XP books) that the answer to "bad projects/people/environments that you can't change" is to get a new job; this may work at the individual level (sometimes), but it does nothing to actually fix the broken projects, environments, or people themselves.

There is indeed no silver bullet. All these decades later, and many people still refuse to accept that.

As with any process of production the move to change it in to a documented repeatable process complete with middle management has taken place.

Large corporations like to get everything down to a standard process (typically). A software development methodology lets them do exactly that; it promises a consistent process and (hopefully) quality.

In a way this works well - small firms innovate and come up with new methodologies which may at some point get to a tipping point where everybody wants a piece of the latest craze. Not everyone is in the business of questioning what works well or why - "if the competition is going Agile or setting up a DevOps team, then it must be good for us as well"

2. Do you have enough experience and do they work well together, if not your mostly screwed. Right hand doesn't know what the left is doing. Takes too long to train new people usually.

3. Are estimates being treated as estimates or is date,team and scope being dictated from above? project triangle anyone. Stretch goals for managers mean long nights and weekends for the team and bonuses for other companies recruiters.

4. Has management been over sold on buzz words and marketing hype?

5. Is there some kind of incentive to come in under budget. So, much for quality.

All of the above can screw up a project quick.

What is interesting is why this relates to software more than, say, ditch digging or gardening.

I would say that is because software involves dealing with the most complex situation imaginable as effectively as possible. In other situations, a person only has to mobilize some of faculties, in software, a person has to mobilize all of their faculties - or at least much more of a certain type. But this shows how different people's abilities are at the limit. Not just different in extent but different in kind.

1. Software development was tightly controlled in the 70's through a hierarchy of management (project managers, business analysts, senior programmers).

2. Today the hierarchy is gone, reduced to “team leads” and “scrum masters” with no real authority or control.

3. Today programmers left to themselves often adopt or create methodologies stricter and more filled with ritual than software development in the 70s.

4. Today he often gets involved in 1 or 2 people projects that have so much process and “best practice” that nothing of real value gets produced.

5. What makes software development work is “conceptual integrity” or common vision or that feeling working on a team where everyone clicks and things just get done.

6. He doesn't understand why he had that feeling more in the 70’s than now.

Well if you follow the logic, what is missing? The hierarchy of management. Which he seemed to blame for the original reason why software development methodology doesn't work.

I hate to admit it, but maybe the hierarchy of management added more value than we gave them credit for?

He sort of suggested finding that “feeling” in a project/team was unpredictable, random. But perhaps the hierarchy of managers actually had valuable skills that led to higher success rates of projects achieving that “feeling”? Perhaps they had a knack for picking the right people and for managing people in a way so they worked together more efficiently? Of course they weren't perfect and there were some bad apples, just like any profession.

I wonder what the demise of the hierarchy of managers was caused by? Did the hierarchy of managers lose these skills overtime and become useless? Did programmers just not recognize their value added and throw them under the bus for 30 years until they didn't have a leg to stand on? Was it pure budgetary, cut costs and get rid of the resources with the least perceived value?

DISCLAIMER: I am not a manager. I use to be very hard on managers, then I became one for a while, realized how difficult their jobs can be, and now I am one of the easiest people to manage.

For example original waterfall was kind of similar to agile methodology, there were re-estimations, milestones, prototyping... But I never heard anyone recently saying that our presumptions were wrong and we need to re-design and re-estimate.

Agile (scrum to be specific) in many companies is practically form of micromanagement. Number of companies are using 'agile' without automated tests and other necessary tools.

I find more problems when there's less focus placed on understanding the data, manually, first, before placing it into a process.

Methodologies don't make it any less important to find, connect and understand the data first, but it seems to happen way too much.

A methodology I see missing is when we developers obsessively focus on optimizing tooling and code, instead of obsessively finding and understanding the data first.

3rd or possibly lower is the methodology used. The methodology's main purpose in my opinion is to align everyone's working style to milestones, it doesnt reveal anything about whether the project will succeed, only how we will approach it.

My hypothesis is that when you get engineering and management teams to try and manage out the creative aspect of most projects, things just go poorly.

One thing I found that helps a lot is to work on a product that includes EEs and MEs, e.g. not pure software. It really drove home two points:

- process can work if people actually follow it

- the craft part of engineering takes a long time to learn but software is such a young field that practitioners are distracted by shiny objects instead of focusing on learning their craft

-1 for being too short of an article; I wanted to hear better specific examples, and was sad when I saw the comment section starting :-)

In defense applications, software development methodologies work excellently. For example, MIL-STD-498, which I've used extensively in the previous decade, has worked wonders.

Been around long enough to observe 100% correlation, extensively.

And it's a small minority who are competent.

Most management and process exists to turn 1.0x developers into 1.2x (in theory) but turns the 10x into 3x or 2x or sometimes -2x. When you start enforcing process, closing up definitions of work, and take power away from engineers, you lose so much more off the top than you get from bringing up the bottom.

The problem is that very few executives actually want to create an inspiring place to work where people do their best. ("Fuck you, I've got mine.") They want incremental "improvements" (of questionable long-term value) on what already exists. This hand-wringing about process sounds a lot like early communism: it trudges along happily, inventing new structures, in complete ignorance of the human motivations around it.