Requirements isn't just making lists and tickets in swimlanes. It's actually learning the domain you are building for. Sure you can still build for something you don't really understand, but it'll be garbage. In fact, writing this garbage is OK if you accept that it's just a drafting process for learning the domain challenges and potential solutions. There are no shortcuts to quality.
My advice for stakeholders is to make sure you have access to the person(s) building your application and see if they understand your domain (you can do this without constantly pestering them!). Beware bullshitters using buzzwords.
My advice for designers / programmers is to make sure your stakeholders & domain experts are keen and engaged, otherwise getting requirements and understanding is like pulling teeth. Sometime a stakeholder doesn't even want to solve the problem. They may have wanted to go in a different direction entirely or maybe there are politics involved that has them miffed. Nothing sinks a project like lazy or uninterested stakeholders.
I can vouch for this. At the other end of the spectrum, programmers getting thick with their domain experts is like having clairvoyance. I can sometimes spot problems before my PM does!
Fast forward to today, I'm in a different part of the industry, we're building things that aren't quite as complex but they take longer to finish and the experience is very frustrating. I feel like I'm working in teams where many people don't really know what they were doing but we keep getting pushed for higher story point "velocity" and whatnot. Quality is suffering, tech debt is piling up. This whole thing feels broken.
Also he was an engineer. Most programmers these days aren't engineers (despite the title) and that's OK for how things are done. But in the 60s most programmers were engineers, and that process is different.
For the most part, it's those types of issues that lead to a system not working as intended, or being buggy as hell. If you can get a good setup going where the requirements are clear and don't change and the people on the project communicate well and have full control over the process then things will work out fine. That's seemingly what happened in the source article.
> The stakeholders have no idea what they want
And yet, you want to put together a project plan up front based on them telling you exactly what to build without assuming there will be sweeping changes? That makes no sense!
Building iteratively recognizes this reality that people aren't sure what they want or need, rather than peevishly ignoring it. The goal is to figure out what would be valuable in a short period of time; if it isn't valuable at all, that's fine, it was not a huge investment and can just be thrown away; if it is valuable but not quite right or not as useful in its current form as it could be in a different form, but the change needed is large or fundamental, that's still fine, even sweeping changes are no big deal on top of something small; and if it's already valuable just the way it is, that's great, move on to the next iteration.
This seems to make a lot of people uneasy, I guess because it provides no formula for answering the question of where things will stand in a year, or even in six months. But it honestly recognizes that nobody knows, rather than setting the expectation that where things will stand is exactly where they project plan says they will, with everyone inevitably getting mad that reality didn't match that plan, or worse, being exactly where the plan said you'd be, except the thing the plan called for was the wrong thing to build and the whole thing was a useless waste of time.
I never understood why software engineers had such a high tolerance for this shit. Is it because changes are technically possible at any point?
For example, if I order flowers for a wedding, I need to order at least a few months in advance depending on the time of year, because flowers are a seasonal agricultural product. And I can't change my order once it's locked in because they can't go back in time and plant more flowers.
We should treat software engineering more like that. There's no reason we should allow product people to abuse the flexibility of software. You can still be agile, but you don't have to be at the beck and call of people whose preferences change with the direction of the wind.
What I think is that this is right, except the expectation should just be that ideally you keep that iteration going indefinitely, instead of saying, "ok, now that we've done these couple rounds of prototypes, we now definitely know everything about what your preferences are!".
That's more likely to be true after a couple rounds of iteration of prototypes, which is good, but still unlikely to be true.
> For example, if I order flowers for a wedding, I need to order at least a few months in advance depending on the time of year, because flowers are a seasonal agricultural product. And I can't change my order once it's locked in because they can't go back in time and plant more flowers.
Yes, it's because it would be a lot better if you could immediately switch the order. Wedding planning, and many other things, would be a much better if everything didn't require locking in decisions months in advance.
Why advocate for a poorer experience when it's possible to achieve a better one? Sure, if you want to charge less for a process that asks for all requirements up front with no changes allowed later, because that's less valuable, and a higher rate for an iterative process that responds swiftly to changes in direction, because that's more valuable, then that would make sense.
But it's clearly possible to make changes to software without a bunch of lead time - you don't have to wait for seeds to grow or send a manuscript to the printer or blueprints to a manufactures - so why would we artificially mimic those worse experiences?
> But it's clearly possible to make changes to software without a bunch of lead time
My argument is that it's often not possible, at least not in the way that non-programmers seem to think it is.
There is lead time in delivering an updated product after requirements change. There is a positive relationship between the size of the change request and the amount of time/effort needed to adjust to the request.
A complete design overhaul will typically require a substantial code rewrite. That takes time, and taxes the sanity of the people writing the code. At some point, employee morale and the risk of them quitting for other jobs becomes a resource to that you need to manage.
So while there isn't a hard minimum like there is for planting new flowers, you cannot treat software as infinitely malleable unless you also have infinite resources. Nobody has infinite resources.
At some point, design iteration must end. Iteration without convergence means that completion is impossible, and therefore project success is impossible.
And even in the initial prototyping phase, change requests must be triaged (or rejected if necessary) given resource constraints.
But I'll quibble with a couple things. Or rather, I'll just put forward that I have a different perspective on them, while fully understanding where you (and I think most people) are coming from with your differing perspective:
> A complete design overhaul will typically require a substantial code rewrite. That takes time, and taxes the sanity of the people writing the code. At some point, employee morale and the risk of them quitting for other jobs becomes a resource to that you need to manage.
I think people would have their sanity less taxed if they "just" had more realistic expectations. I think what rightly frustrates people is being asked to redo things on unrealistic timelines or without appropriate compensation. But those are their own separate problems. Absent those issues it really should not be frustrating to make foundational changes in light of things that have been learned about what would make the system more useful. It should be expected as a nearly inevitable part of the process.
> At some point, design iteration must end. Iteration without convergence means that completion is impossible, and therefore project success is impossible.
I don't think so. I think the most successful software projects don't reach "completion" but rather iterate indefinitely. The iphone has not reached completion, and is nonetheless very successful.
But I'm guessing you're thinking of fixed term project work as a consultant / contractor. If so, this is one reason why I frankly don't think that's a very good model for building software. At the very least, I think the expectation that iteration will continue to be useful indefinitely should be built into the contract, with some way for a client to decide that they are satisfied and decide to delay or not pursue further iterations. But I think an expectation of "completion" is a recipe for frustration on all sides.
I also think you can do iterative development with having a clear long-term objective (requirements) in mind. I would even argue it helps a lot.
I suggest looking up "What made Apollo a success" (https://ntrs.nasa.gov/citations/19720005243), they explain it quite well.
But I do agree with that commenter that it is usually the case. But people often seem to ascribe that to a failing of a person or group of people - "top leadership" in your comment - but I think it's essentially the same "failing" as predicting the future incorrectly. Of course lots of effort is put into forecasting as well, and effort put into requirement gathering is similarly valuable. But in both cases, investing in flexibility is a useful hedge against the likelihood that your original prediction was wrong.
> I also think you can do iterative development with having a clear long-term objective (requirements) in mind. I would even argue it helps a lot.
Personally, I don't think this is "iterative" in the same sense of the word. I recognize that it is still iterative and that there probably isn't a better word to use. But just chopping up a long list of static requirements into smaller chunks and doing them in some order is what project planners have done time immemorial, and is not the same conceptual idea as setting a vision and discovering detailed requirements toward that vision a small chunk at a time. It's that second approach that is the sense of "iterative" I was using.
I certainly don't think it's the only way to do things, and I don't think it's a great fit for every project, but I wish more people were actually bought into the leap of faith required to let go of detailed up-front top-down planning and work in small iterations.
It's a sticky wicket because of course time in front of the board is precious so you don't want to constantly run things by them and make them micro-manage, but I think the board would have probably chewed them out less if they had brought a tiny MVP (or even just a proof of concept) that hadn't required significant investment, and asked "here's what we have with almost no investment, here's our plan for the next small step, what do you think of this direction?".
What do I mean by that: I work for big corporations mostly and quite often there is the average case that is about 90%-99% of all incoming work.
Then there is a myriad of special cases some of which happen every third year on a blood moon, if an eclipse is happening at the same time and the witches chant in the woods...
Instead of managing these unicorn-cases by hand, they have to be implemented in code and lead to bugs and a ton more code to review and maintain.
Speaking of maintenance... oh, don't get me started on that one, it's a sore spot!
In particular, these edge cases tend to be more complex, require more knowledge and have less margin of errors than the standard errors happening 90% of the time. Leaving them as a gift for the future maintainers is a special kind of dick move.
by whom exactly? I mean, I agree that it's shitty (I was such a maintainer already), but who is responsible (the whole machinery or a specific role?) and how can we change it?
I personally try to add meaningful comments to code that may seem "strange". Like code that if I read it and it was written by someone else I would ask myself "Why so complicated?" - that way I hope to improve the situation a bit for future maintainers.
If there is specific finger pointing needed IMHO it would land either on the manager or the product owner for miscalculating the impact of that decision.
PS: To your point on "weird" code, imagine a code that just asserts for some conditions with a "if these conditions are true call XXXX team for help" message on it. That's how I'm representing an unhandled edge case.
Sometimes these things are worth it because the manual process is error prone and perhaps frustrating/stressful (these things can't be measured as easily). But sometimes they are not examined critically at all.
How often? How many? How important? Are there simpler solutions?
Kind of depends on the receiving end of the questions in my experience. Some people are happy if you push back and keep things simple, others have problems with that.
I wonder how much does scale affect these basic principles. I would assume that with scale you already have a lot of problems that push even harder towards not implementing every single thing.
This is where trade-offs in engineering and product are important.
Is the edge case safety critical or poses a safety risk? If so, it definitely should be considered and handled.
Does the edge case block a critical user flow (eg. purchase/checkout step)? If so, it should probably be considered.
Does the edge case result in some non-critical piece of UX having the wrong padding in some lesser trodden user flow? Possibly acceptable.
If you think "agile" is the reason requirements change...I don't know what to tell you. Requirements will always change, full stop. It's like a force of nature, there is no universe in which everyone just "knows" what to build up front and has a fully spec'd out API that you can go off into a cave and implement. Real life never works that way, and software engineering is not the right profession for you if you need that.
But it's not that binary. "Agile" is the reason requirements are allowed to change constantly. At worst, it can be like trying to steer down the freeway by slamming the steering wheel from one extreme to the other. And pure waterfall is also blatantly unworkable.
The real question is, at what rate do you allow changes to be made to the specification/requirements? How much dampening do you apply? And maybe under that, there's another question: How fast can you respond to the real world, and still maintain a coherent direction? The faster the better, but don't try to respond faster than you can maintain coherence.
And agile doesn't make you respond to change quicker, it makes it slower since it is done in 2 weeks sprints. Normally a team could adapt the moment new information comes up, strict adherence to scrum agile would push that for the next sprint.
Agile does make scheduling new changes effortless though, encouraging new changes to be made all the time, but it doesn't make the team react quickly to those changes and nor does it remove the total cost of a change. I don't think that is a good thing to encourage, in such a system no wonder people get used to changing things all the time so nobody really knows what things are supposed to be.
But that doesn't mean you have to stop what you're doing and go chase those requirements.
Why did the requirements change? Is it mandatory that the change happen right now? What research was done to support the requirements change? Was the original requirement bad in the first place? Was insufficient alignment and understanding achieved at the start of the project? Do you actually talk to the stakeholders at all, or does your PM just forward you their emails and expect you to do whatever is in there?
There's a lot of room between "design everything up-front and never change anything" and "allow requirements to change arbitrarily".
The arbiter of the changing requirements is the one paying for the work.
If it weren't for users my software would be perfect ;)
Started as a Windows line of business software developer almost 20 years ago. At first we got clear specifications with UI mockups and a description what each and every button should do. When there were questions, these specs where updated and we implemented and tested and fixed until our boss was happy.
Over the years we got more and more customers yet less and less time for tests and fixes. So we switched to "agile" and dropped the specs, instead wrote quick notes about what must (roughly) be done. At first all were happy. But now we have a huge amount of features that not one dev knows. You have to ask and search around until you find all the lose specs.
Now I'm managing such a team of developers and have the same issue. I don't have the time to write a clean specification, yet alone discuss it with the actual customer, which anyway doesn't really understand all the implications. So they start coding by adding more if and else blocks to an already bloated code base.
Pity the days we would start with a class diagram or just some quick drawing about how the components would interact and be testable.
We are inventing the field as we go. Have been for the last decades.
Also it's one of those fields where the expert in the topic must build a system for something he is not an expert with.
Again and again.
And most project are custom.
Agile has nothing to do with it.
It's the nature of IT right now.
If you've got that - why do you need the developer? Genuine question. If the technology requirements are that well defined why on earth do i need to pay engineer level salaries for such basic work? As you said, and i agree:
>> Give a dev a clear API and a well defined set of criteria, and most will write code that works very well
At that point of having clear criteria, there are only 2 tasks left:
1. write some code that satisfies this outcome
2. write it in a way such that it can be cheaply changed in future
But but but performance! durability! ... <many more of the -ilities of software that an experienced engineer will claim to just take in their stride...>
As a customer of this software developement process, these are just the next cycle of requirements to feed into the machine - hey take this thing you did, make it faster while obeying rule #2 above.
It would not surprise me if ChatGPT were able to cover 80%+ of a problem that well defined.
I totally get the attraction to focus on the fun bit, but it's not how the business world is designed to work in most cases.
For what it's worth i think you nailed the actual value opportunity for a strong dev:
>> no one actually knows what they're building
If you can nail that, you're always gonna be valuable. We talk about programming and coding but actually, this is more of the role. The coding part of the puzzle is easy in comparison.
As for:
>> and even if they do it keeps changing because "agile"
If your understanding of why requirements are in constant flux amounts to pinning it on "agile", you're not even on the field yet never mind winning the game.
Also, indeed, not all dev work is equally costly / hard / basic. It’s a crazy thought, but maybe that’s partially the reason why different devs can have different pay grades. :O
Someone still needs to do this work, and sometimes that's the developer. Isn't that what he was doing in this anecdote?
I'm probably romanticising it, but it was a time where software engineers and their skills, time and attention span were still respected.
But in actual professional coding practice over 10+ years, I can count on one hand the occasions where I had clean enough reqs and enough uninterrupted time to design software like that, and still have enough fingers left over to hold a pencil.
In particular, management was tech-savvy and everyone liked and respected each other. Most were also talented.
Agile is precisely the response to the fact that requirements keep changing, not the cause of it.
He did "invent" internal and external APIs for his service, just like developers do today: that's the part requiring developers to be like "engineers".
Now, the (common) version of this that I don't like is when there is no way to know whether a sequence of steps ended up in a better place than where it began. So feedback is very important to me, but there are lots of ways to get it; there are quantitative measures of growth in usage or revenue and there are qualitative measures like surveys or for things like internal platforms, seeing the roadmaps of other teams be unblocked or accelerated by your work.
But I find significantly less joy in being told "we need this exact specific thing, please go build it and report back", and I have rarely seen it be the case that what they needed was actually that exact specific thing.
If you don’t understand a solution well enough to write it down, you don’t understand it well enough to implement it. (Much less get someone else to implement it.)
An overwhelming number of engineers I have worked with seem to think any kind of written design document is busy work, and not actually a part of the process of making something worth using. My experience shows it is not that at all: design docs are a tool to externalize your own thinking, and reflect on its quality without the overhead of keeping it in your mind. That’s their first and foremost purpose. To explain your rationale to others is a secondary objective.
I’m not talking about lengthy requirements documents or specifications, either. I mean a two or three page white paper outlining a problem or a proposed solution in plain English. This is something you can bang out in a half hour or hour if you’re diligent.
Many of the folks I work with never even seem to bother.
Or at least spend the time to tell us why we are doing this nonsense and maybe I can come up with a less asinine solution.