My Response to “Are we overcomplicating software development?” | Hacker News Reader

My Response to “Are we overcomplicating software development?”

neilonsoftware.com

94 pointsfeyn66 comments

"Process," is the answer to the question, "how do we get a team of talented engineers to design and implement software together?"

The answer to what should that process be? I'd look to the SWEBOK[0] for an introduction to the state of the art.

As for the question at stake here... let me use the words of someone much smarter than myself:

> Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.

> Dijkstra (1984) On the nature of Computing Science (EWD896)

If you don't appreciate the true difficulty of programming you're bound to create problems and spend most of the rest of your time debugging them later on. If your chief motivation is to sell a software product or service powered by software then it behooves you to create complexity. It seems more impressive when you can't explain why some component or feature doesn't work in a single sentence. You just want to control that complexity enough so that you can maintain the illusion of order.

If you want to achieve simplicity you have to work at it. It's hard and doesn't come for free. You really have to think. There's no way around that.

[0] https://www.computer.org/web/swebok/index;jsessionid=306e197...

(I'm Neil, the author)

I can't find a "like" button anywhere, so I'll just say "Like".

There's always the functional style of write something inanely simple and use massive words to describe it.

(in regards to the original post that started this)

I could rant for hours about how so many misuse the term "Agile" and the misunderstanding the idea(s) behind being agile. But I'm at a point where I almost don't care anymore. The use of "Agile" as though it describes a specific, prescriptive methodology is so ubiquitous that it's almost impossible to talk about the subject.

Let me just say this... go read the Agile Manifesto before issuing any criticism of "Agile" and repeat this 10 times - "Agile is NOT a methodology".

Scrum is a methodology. Crystal is a methodology. RUP is a methodology. XP is a methodology. OpenUP is a methodology. TSP is a methodology. One or more of those methodologies may be "agile", but "Agile" itself is NOT a methodology.

(me = author)

My bones ache from seeing Rational Unified Process mentioned, but of course you are correct.

I would say I was "at the point where I almost (didn't) care" about 5 years ago. I then switched jobs and was made lead of a team I cared about, and decided I wasn't done with implementing Agile. There was a time, however, when I was downright apathetic. Companies had beaten me down too much, and I didn't have the energy to fight anymore, so I completely understand.

Interestingly enough, I'm actually a fan of RUP, IF you instantiate it without requiring all the UML artifacts and what-not. One thing I like about RUP is that it emphasizes that you perform work in each area (requirements, design, implementation, etc.) in each iteration. To me, it emphasizes the "iterative" aspect of "iterative development" better than, say, (naive) Scrum.

I also prefer the term "iteration" to "sprint" because the "sprint" analogy breaks down in that you can't actually SPRINT -> SPRINT -> SPRINT -> SPRINT forever and ever in reality. And to me, it just creates the wrong mindset. Admittedly it's a minor quibble though.

I learned RUP before Agile, and didn't hate it. I only learned if you like one thing you necessarily have to hate something else later in life. Aspects of RUP make it into my interpretation of agile. For example, I teach my dev teams UML (sketching as defined in Fowler's book) for whiteboarding and the rare BDUF, which RUP exposed me to. Speaking of BDUF, I've saved many a feature by resorting to BDUF when communication broken down.

I didn't and don't want to start a flame war, but needless to say once in an interview I was asked if I was a "Scrum Master" (after having about 15 years of agile experience). When I stopped laughing, I said, "No, no I am not." I don't think they got the joke, and no - I didn't get the job.

I don't think "iteration" vs "sprint" is a minor quibble. I've seen the "wrong mindset" you speak of, as people take "sprints" far too literally. It's not a sprint, it's a jog. A long, long jog. You'll be stopping for water and stretching along the way. Sprinting is a good way to pull a muscle and have all your devs quit on your one day, leaving you with a big-old case of backlog rhabdomyolysis. I took that analogy way too far.

I learned RUP before Agile, and didn't hate it.

Same.

I only learned if you like one thing you necessarily have to hate something else later in life.

ROFLMFAO.

Aspects of RUP make it into my interpretation of agile.

Same here.

For example, I teach my dev teams UML (sketching as defined in Fowler's book) for whiteboarding and the rare BDUF, which RUP exposed me to.

Yep. My issue is, a lot of people see a dichotomy where you have "big design up front" or "no design at all". I like to talk about "sufficient design up front". I don't know if I coined the term or not, but that's my favored way of expressing that middle-ground, as I see it.

backlog rhabdomyolysis

I'm so going to steal that. :-)

I shall in turn steal Sufficient Design Up Front (SDUF)

I like the old "just barely good enough" as a description of the amount of design required. It means basically the same thing but is a bit clearer about how sufficient things need to be.

I'd never heard that before, but I like it.

Note that Agile values "people over process," so having too heavy a focus on process violates the Agile manifesto by itself.

Exactly. It's almost ironic that people have chosen to conflate "agile" with a highly prescriptive, heavyweight process. :-(

You conclusion is wrong. Agile finds value in process, as well. Read the last line until this sinks in:

    That is, while there is value in the items on

the right, we value the items on the left more.

There is a lot of value in the right process, the process however needs to be a servant to the needs of the people. You can be agile with long complex processes so long as you pay attention to how the processes affect the people and ensure that every time a process is getting in the way of someone it is a good thing.

I think we would all agree that process needs to stop the "cowboy coder" who otherwise would check in code with many syntax errors just before leaving for a long vacation. This example is nearly a straw-man it is so trivial - but if anyone doesn't have personal processes to prevent it the team will need to create a process of some sort. This could be a simple don't do that again talk, a HR process to fire the guy, a automatic backout process, a formal checkout for pushing, or whatever - the choice is up to your team, and depends on team factors like if this is a one time mistake or a repeated offense.

Going to the other extreme, medical software that could kill should have long complex processes. Agile is perfectly fine with this.

Agile is taking issue with some [big companies] that have created such long cumbersome processes in ages past that no longer serve the needs. Sometimes they created a complex process because of a one time mistake that is unlikely to happen again. Or maybe the process made sense in light of the tax laws that were changed in 1986. Some processes actually are required and useful, but because of all the cumbersome ones they are ignored until it is too late.

Well said.

I don't know if you realized this, but all your examples are of how process controls people. Can you even explain what it means to make people more important than processes?

Process is about controlling people, so of course my examples are controlling people. The trick is they are process controlling people in areas where they need to be controlled. Contrast this to processes that control people in ways they don't need to be controlled.

Is someone checking that are login to your computer at exactly 8am every day? That a process that controls people but isn't needed for most programmers. The process makes perfect sense in context of assembly line workers: if even one person is late the entire line cannot start.

One thing that I didn't say but should have is that process should not be the first thing to fall back on. You might have to deal with the guy who pushes broken code all the time, but there are other solutions other than process that you should explore.

Most of your post is still about processes, justifying why they are good. I don't think you get what it means, still: "people over process."

looking back, I accept your criticism. I'm an engineer, ultimately I understand things like processes better than people.

I stand by agile saying people over process, but I have no clue how to actually explain it.

Surely, "people over process" means, when people say process needs to be modified in some instance, you trust them rather than beat them over the head with process.

"It doesn't make sense to test this" - "well do it anyway, that's the standard process".

I think there's an aspect of democracy, too: so rather than using process as a way to control people from above, you are all working together as a team. Process naturally moves into the background because you trust each other to all be working towards the same goal.

I notice you're not having daily standup meetings, are you sure your development is Agile compliant?

| "Agile is NOT a methodology".

in the Ask HN that started this, he did put methodology in literalized fingerquotes, so he didn't actually make the egregious error you decry.

| 5) Agile "methodology" when used as anything but a tool

and his summary point still applies right on target because the Agile "manifesto" does not solve what he identifies as the problem: I think overall we seem to be over-complicating software development. We look to architecture and process for flexibility when in reality its acting as a crutch for lack of communication and proper analysis of how we should be architecting the actual software.

dear Agile Manifesto,

The best architectures, requirements, and designs emerge from self-organizing teams? so therefore self-organizing teams always produce the best architectures... anybody see an error in reasoning?

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Suuure they do, and that's why we see Agile processes adopted by winning NFL teams, or in the ERs at hospitals, and... Look, there is a Zen Buddhist ideal for living your life in harmony, but it's a goal, not an emergent property of self organized teams.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. So, do not look up information in a man page, instead ask me and I will read it to you... that metric-imperial problem that lost that space probe, why another scrum or two would have identified that right away!

I could keep going, but this letter is too long already. Yes, software does offer some advantages over other types of team problem solving (due to the possibility of virtualizing or dummying any of the parts) but because of that flexibility it becomes even more important to nail down as much key design infrastructure as possible, and the Agile Manifesto does not read like a manual for nailing anything down. (And software alone offers the possibility to rip up things that are nailed down, so we don't need to fear nailing down.)

Love,

philosopheer

Sure, I wasn't actually decrying any error in the original post. It was more that the post alluded to something that's a hot-button issue, just enough to trigger a rant from me. I agree with a lot of what the author of the original post had to say.

FWIW, I don't take the manifesto or the principles behind it literally for many of the same reasons you hint at.

My attitude when looking strictly at agile doctrine is a mix of skepticism (especially after having learned and practiced agile for around 15 years now) and thinking, "Hummm...they might be onto something..." I don't often criticize the written word of how people try to describe agile, but instead try to understand "what were they trying to say?"

A point of illumination for me as to which parts of agile I was most interested in developing a deeper practical knowledge of, happened after I watched a few documentaries on Skunkworks, and how that group operated. Robert Kelly's 14 Rules and Practices does (to me) have a lot of conceptual similarities with the manifesto, especially when you see how they were practiced on some of the most difficult technical challenges humans have ever faced:

http://lockheedmartin.com/us/aeronautics/skunkworks/14rules....

I think that if someone were to pin me down and say "Tell me the doctrine you follow!!!!" and I was forced at gunpoint to give a simple single answer I would point to Kelly's 14 Rules over the manifesto.

A deeply unfortunate fact of the agile's history is that the C3 system wasn't really very successful by just about any dimension:

https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...

I've been told by many of Agile Manifesto acolytes that the C3 project's less-than-stellar results don't matter, but I personally think it does. At a minimum, at least you can point to Kelly's Skunkworks team, and see how they shaped modern history with their inventions and ability to deliver improbable solutions on an impossible schedule.

In summary, where Kelly's 14 and the Agile Manifesto & Principles conceptually overlap, I believe there is much worth learning.

C3, I would think, speaks more to the XP methodology than Agile principles.

OTOH, Agile principles are high level enough that they may provide a framework for thinking about methodology, but aren't really actionable in any unambiguous way.

The point about "Agile the philosophy" being better than "Agile the practice" seems like a red herring to me, since so few get "Agile the philosophy" right. It's the same way I feel about things like HATEOAS: maybe it's good as a Platonic ideal, but if no one is reaching that ideal, then how useful is it really? There need to be suggestions that we can actually put into practice.

Exactly my problem with Agile. The philosophy is immense and hard to follow. Everyone sees it differently. The practice of it, thus, is always unique, and never consistent. We need something better.

Agile is never wrong: it is only the humans that have failed it.

More consulting, of course, is required.

It sounds like you're being sarcastic, but in case you're not.

Process is built for people, not people for the process.

That second line is the only thing that distinguishes Agile from communism ;)

> The practice of it, thus, is always unique, and never consistent.

And it should be unique for the team. The software process is not very transferable because people are not transferable (in addition to many other things that are also not transferable). Each team is different and the agile process should reflect that difference.

Assuming there is not a lot of employee turnover, the team should become more consistent over time on a given project.

I suggest then that what you have isn't a process. McDonalds has a process. The process makes it so that no matter which McDonalds you go to, you always get the same food.

If your "process" works differently with different people, what you have is indistinguishable from not having a process.

Your point is difficult to refute.

My thought is as follows: McDonalds is designed around the core idea that people are fungible and that mediocrity is optimal result (by "mediocrity" I mean no better, but no worse than the average hamburger). I do not believe that software development resources are fungible due to the high degree of creativity and independent decision making required to strike the best balance of quality and time-to-market, and often mediocre results can hurt companies during critical stages of growth.

With that in mind, do we then say "If resources are not fungible and results cannot be mediocre then process cannot exist"? My answer is "no." A lack of resource fungibility does demand an individualized approach to training new resources on the process as it exists at a point in time, but provided resources do not change, the process itself should not change much outside of evidence-based optimization, i.e. "this ain't workin' so we better change somethin'"

To me, process is about expectation management. Every other aspect of a process is in support of maintaining or exceeding expectations. I believe that if a process is so catered to individual whims that no expectations can be reliably set, then there isn't a process. Having said that, I do believe you can achieve a relatively consistent level of meeting and/or exceeding expectations with a process customized to the individual needs of the team members.

Still, your point is very valid. Mine were only additional thoughts inspired by your astute observation.

McDonalds is closer to a factory process, and has little to do with software development. For years, factory processes have been applied to software development and for years projects are late, over budget, and never delivered.

I see agile as more a meta process. A process to build a process from. Agile attempts to accept the reality of the world that people are different, teams are different, and projects are different. It even describes the retrospective as a way to change the process so that it works better for a team. Over time different teams should end up with the agile process that works best for them.

To me, the real issue is that the managers got ahold of agile. Most managers just want waterfall, I'm convinced, which I suspect is where so much of the head scratching about Agile comes from.

We developers have done an abysmal job developing, socializing, and formalizing our standards and practices. Needing to keep agile things loose because teams and industries and people and projects differ is one thing. But, we don't even try to establish unambiguous definitions for general topics like test-driven development, big visible charts, or sprints.

Frankly, we need industry leadership and earned certifications for developer craftsmanship. Uncle Bob recently has shared some good food for thought in this area. Not only does he recommend the direction we professional devs need to head, but he also speculates as to what will happen if we don't (loss of freedom.)

https://www.youtube.com/watch?v=ecIWPzGEbFc

Managers don't want waterfall. They want a process they can understand (waterfall is great here), that gives them a specific date they can point to as when everything is done and they can start making money. Waterfall says it gives you that.

Of course we all know that waterfall doesn't actually give you what it claims, but it says when it doesn't you failed to manage one of the early phases correctly which means it tells them how to fix things: work harder there.

Ooo! I wrote another article talking about single date estimations:

https://neilonsoftware.com/2016/07/13/a-far-too-brief-rough-...

Yes, a specific date is absolutely what every manager at every level in the organization wants. As long a payroll is issued bi-weekly, and earnings reported quarterly, the desire for a fixed date will be a thing.

The issue is, fixes dates when it comes to new software development is a fantasy. If you really, really wanted to talk to experts about hitting dates, I'm thinking that a well-funded initiative done in collaboration with 3 of the 4 branches of the U.S. military and the most respected innovator in aviation technology would be where you would go to find them. I would like to offer up the F-35 Joint Strike/Fighter program as exhibit 'A'. Turns out, despite all that funding and discipline (and threats of what will happen if you miss a deadline), they still can't hit their dates. If they can't, what hope do we have? That's not a rhetorical excuse - that's a real question.

Is the answer Agile? Personally, I don't think it is. I think Hollywood has a far better model that anyone else does (pre-production -> production-> post-production). To get new ideas, I study how specific movies are made. One of my favorite case-studies is how they did "Max Max: Fury Road" (lots of storyboards - lots and lots and lots of storyboards). "But hollywood movies are always late/overbudget!!!", yes - which is my point. They know that and have specific adaptions that make this process work decade over decade - and generally a hell of a lot of cash in the process. At a minim, if Agile ain't workin' for everyone, we've got to try to get new ideas from somewhere. My personal choice is Hollywood.

Nah! Have another look at Agile, and not the management focused movements like Scrum. I'm talking _The Art of Agile Development_ type of Agile.

Hollywood is broken. https://m.youtube.com/watch?v=H18RUB1cxfI (NSFW)

(I'm Neil, the author)

I do agree with you. I about learned Agile first from my mentor (his formative years were in the 80s) who I did and do still deeply respect, but over time he and I disagreed over the nature of what Agile is and what it should result it as I gained more real-world experience.

I honestly have no answers as to specific suggestions that work for everyone. I've even asked teams where I implemented Agile practices what they though of my interpretation and the response I get is "Dunno...works fine to me."

The best advice I can offer has nothing to do specifically with Agile, but pertains to my experience customizing agile to an organization. I don't mean to plug my "book" (especially since I don't make money from it), but the best advice I can give anyone I put in here:

https://neilonsoftware.com/books/personality-patterns-of-pro...

Flawed and incomplete I know, but the best I could think to do.

Why does this need its own thread? (Other than ego-stroking the OP) Seems like any response could be in the main thread or linked from there rather than splitting the discussion.

(I'm the author) My bad. No excuses.

I upvoted it. I'm glad you posted it.

Thanks, I appreciate it.

nroach does have a point. I made a judgement call, and can see both sides of if it was a good one.

Are we running out of threads? :-)

Threads are cheap and this came to my attention more strongly this way than if it had just been a comment.

Every piece of prose posted here is, in some way, a response to something.

I'm not sure it's "ego-stroking," strictly speaking. What's wrong with giving the author recognition?

(me = author) FWIW, I can totally understand how it could come across as ego stroking. In this particular case I wasn't trying to, but I have been guilty of it in my life. As for recognition, trust me - give it a week and no one will remember my article ever existed. I honestly just wanted to answer ian0's question thoughtfully, and made the best call I could think of.

Because this isn't just a response, but a piece of long-form content which stands well on its own. We generally use submissions for those.

Comments are a great venue for short-form replies, but not for multi-page analyses of an interesting question.

Response to this: https://news.ycombinator.com/item?id=13426896

Cloud DevOps is complex, but we're just moving on-prem complexities and hardware complexities to the cloud, so it's a fair trade.

Microservices, when done properly, are significantly less expensive than traditional service layers.

Continuous Integration is one of the greatest improvements in software development.

1) Yes, plus security concerns...that were really always there because clearly all those enterprise firewalls were not stopping anyone. Just ask Yahoo.

2) "When done properly" - therein lies the rub.

3) Absolutely agree, but I have seen teams screw it up time and time again. Never really figured out why that is.

I suppose #3 might fail because of the "frog in boiling water".

If you throw a frog in boiling water it will freak out. But if you throw a frog in cold water and slowly bring it to a boil, it will never notice.

I actually like to submit completed modules when they are ready. I find constant pushing makes things a lot more complicated.

edit: fix terrible spelling

Re: #3, perhaps so. I have seen many a good CI process go to hell over time. I could argue no one would have a CI process if in the beginning it wasn't making life easier. It would seem no CI process survives contact with the dev team.

Re: Modules, generally I do as well, but what is a "module"? A function? A class? A library? A plugin? An extension? A listener? A handler? A subscriber? A publisher? A DAO? A DTO? An in-memory service? A networked service? A microservice?

The philosophy that I subscribe to is continuous integration is not the same as "everyone check into the same branch anytime they want as often as they want" but also that it is not "everyone work in their own branch until time has run out and we have to force all of our stuff to work together." That middle ground can be very tough to find, but I find that if the CI system and the architecture are not designed to work together, devs will always thrash around between those two extremes.

I agree with a lot of this, but I think there is a single correct answer for the best way to first learn about the "Agile Philosophy": http://agilemanifesto.org/

(I'm the author) Well...yes...and that's where (I hope) most people will start. What happens next, I've found, is that people's own confirmation bias' kick in, and they take the manifesto and make it describe (in their own mind) what they are already doing and/or want to do.

The manifesto (to me) is how people who understood the nature and goals of Agile summarized their sprawling, complex, interconnected - and valid - thoughts. At the risk of being hyperbolic, I think of it like E = MC^2. Yes, that is a good summary of Einstein's theories, but if all you ever know of them is this summary you'll miss all the implications of what happens when you put it into practice. Probably a terrible example, but the best I could think of.

I upvoted it at "Microservices are a concept, not an implementation."

Thank you! I'm not sure when the industry confused that point, but I guess I can understand how.

I don't really understand what it means.

This is a good place to start:

https://martinfowler.com/articles/microservices.html

The opening sentence sets the tone well for the article well:

"'Microservices' - yet another new term on the crowded streets of software architecture."

A colleague of mine once said that his work (he was a business strategist) was "slowly coming into focus." We're pretty far along today in our understanding of microservices, but at the time the article was written (early 2014) microservices were "slowly coming into focus". Reading what people where thinking when the industry was trying to define them should help in understanding what they are today.

I agree with ian0's perspective, but I am compelled by Neil. I think we can learn a lot of patience about the state of software development from this article.

I appreciate that - thank you.

Yes, Very much so. Could not agree with the author more.

Careful - I know that guy (I am that guy). He hardly knows what he's talking about half the time, and for the other half he makes it up as he goes along.

Seriously though, glad you liked it.

OP vs this response is great example of gaps in "actual usage" vs concepts

I'm not sure this article is really saying too much at all. I started with extreme programming back in 99 (after following the proponents and all the heated discussions on the most brilliant OTUG email list)

Many many times I have seen this kind of question, "Aren't we just making this too complicated?"

and the response being "It's not complicated, if you misunderstand X Y or Z you are probably going wrong"

My observation is, many things of the things we leverage in the software world are answers to problems. Some of those answers seem so good, and work brilliantly when combined with other answers we package a set of answers as a "process / methodology" and advocate it.

But sometimes for some people the answers don't really match the problem, or need adapting to the problems at hand. Now, without good insight into the original problems, then YES it will be "too complicated". Any time you do things for problems you don't really have it is

A phrase that was thrown around in the early days of XP was "Brain Engaged", to try and highlight that you don't blindly do things, you need to be aware why you are doing something and adapt as necessary.

However! If someone says "hey this new new way of approaching design is awesome", and it piques your interest, You should blindly learn/try that something new and debug it till you get an understanding of why it was considered awesome. At that point you can brain engage it as needed.

So back to the original question, yes, we do make things too complicated and we shouldn't do things that don't make sense. It may not make sense because we have the knowledge and experience to know it doesn't, or it doesn't make sense because we are too inexperienced. Both are valid reasons to stop and start asking what is going wrong? why isn't this working? In the world of "Lean" this is the idea of "Stop the line", sort things out, start moving forward again, even if slowly at first.

All that doesn't mean you should keep things really basic without changing, it is very important to know how to adopt ideas for doing things better and that actually contribute to things we care about.

Also important to try not to become a cynic, eg, "tried microservices, they suck, OO sucks, imperative sucks, VB sucks, NoSQl sucks". When we become cynical of something it becomes very hard to be brain engaged in regards to them. Also the opposite is true, becoming enamoured with something can limit our brain engagement also like "Functional program all the things!, unit test all the things!, scale all the things! deep learn all the things! distribute all the things! Reactive all the things! things! the all Forth"

"I'm not sure this article is really saying too much at all" I believe to be accurate, and I'm the author. I wasn't so much as making a statement or a point as addressing the questions in the best best way I knew how.

If "The Dude" wrote a response to Ian0's original question, I would think it would have a similar sentiment to mine, though far more concise: "Yeah man...but, you know - no man."

Value is always found in the honest exchange of ideas, and I found the link to SWEBOKv3. I'm happy for your post bringing it to my attention!

Reply on news.ycombinator.com