And it's ignored by everyone, the stakeholders don't realize it's a problem, the programmers know but think it's not their problem, and nobody includes it in estimates or plans the required time and meetings for it to happen.
I disagree. We know it's our problem. But we quite often don't get a, or too little, say in the decisions made to manage that problem properly.
Because yes, it would be nice if we would "explore the whole problem space with stakeholders", and it would be great if we could "decide together".
What ends up happening quite often however, is some "stakeholder" saying yes and thank you to whatever he-who-holds-the-money (or his career-advancement) says, with little to no regard whether the promised addon/feature/whatever is feasible and/or accomodated by the existing architecture. Suddenly, the space that needs to be explored may be galaxy-sized, or moebius-strip-shaped.
What also tends to happen: The "exploration of the space" is outsourced to some consultant or architecture astronaut, who wont write a single line of code, and certainly won't have to deal with any unforseen difficulties.
It would also be nice if, once the "exploration of the space" is done, devs get to implement based on what was explored. Which, given the aforementioned unforeseen properties of the problem domain, is laden with unforeseen change as it is. What tends to happen quite often is that in addition to that, requirements, and thus the problem domain change mid project...sometimes multiple times.
The key insight is that it is incredibly hard to capture a system that has the flexibility of handling every past, present, and future edge case while providing consistency for the happy path and preventing "bad" things from being done.
XP and the proto-agilists said, as such, that it was futile to try and capture the whole problem because it turns into contract negotiation and pitting stakeholders against each other. Further, they found out that what stakeholders said and what they really needed ended up being not-quite-right and it wasn't exposed until people actually used the software.
(Why? Because the stakeholders are representing a team, and sometimes they themselves are playing a game of telephone. And sometimes they describe a problem in a way that doesn't capture the real problem.)
So, the idea was to have the business representative in the same room. The problem was that those representatives still had to do their jobs, so they ended up burning out and quitting.
And that's how we got all of these product owners lying around - because we just punted the problem and went back to business analysts and gave them a slightly different mission. And that's why people talk about "dark agile." But I think it's easy to understand why the misalignment exists but very very hard to fix.
For internal development, I think the real answer is that the development team and any analyists should be required to do the job as if they were an intern. If there are multiple stakeholders, they have to do all of the jobs, at least for a few weeks, and meet the people who they are building for. Then, they can at least understand the full complexity and why all of these contradictory requirements exist.
Requirements gathering gets short shrift, if it's mentioned at all. But really, it's the overwhelming majority of the job.
Tell that to managers - oh wait - hire on algorithms and performance review on BS such as commit count.
It doesn't, and it always baffles them when I don't even give them a whiteboard problem to solve. I don't need people who have memorized all graph search algorithms, I don't need people who can code them up in 6 languages without looking anything up. People who can google competently (or use a textbooks index) can do the former just as well, and LLMs can do the latter.
What I need are people who can take requirements, anticipate as much as humanly possible what else the client will do over time, and find a maintainable architecture that implements those and maybe is resilient when facing further changes down the road.
And that is a HARD skill, that leetcode and most of what CS curricula will not teach.
Nah, its systems analysis, a field which is largely either forgotten (in the “cross-functional means everyone is a nobdifferentiated developer” school of Agile teams) or replaced with “business analysis”, whose practitioners have less relevant skills (particularly, systems analysts were experienced programmers with additional process engineering skills, while BAs are nonprogrammers in a role that is often structured more as stenography than engineering) and “software architects” (who have a closer skill set to systems analysts, but have a role with a simulta!eously more abstract and narrower, implementation-focused orientation.)
Systems analysis requires (or at least benefits from) an understanding of programming, but its not the same skill
[0] http://zimmer.csufresno.edu/~sasanr/Teaching-Material/SAD/JE... (first live link besides a non-downloadable Google Drive link I could find, the primary distribution was Yourdon’s personal Structured Analysis website which became defunct not long after Yourdon passed away.)
Of course I don't expect you to debate all this in one HN thread. I'll browse through "Just Enough Structured Analysis" -- thanks for the recommendation -- but that will be in order to to make myself a better Software Engineer, not to become whatever a "Systems Analyst" is supposed to be. I guess if you expect a Systems Analyst to know how to write code as well as the other programmers, you are simply describing what I call a Software Engineer, and what you're thinking of as a "non-analyst programmer" is really just a typist that should be replaced by the compiler.
Possibly: I'd certainly agree that what a Systems Analyst traditionally does is exactly software engineering, whereas whether “software engineers” do that, or are selected for the skills to do it, is rather mixed across the industry. The explosion of the Software Engineer title for programmers did correspond with the downswing in Systems Analyst as a role, but it often was not accompanied by the role and expected skillset of the retitled programmers changing, if anything, it was accompanied by a greater focus in selection in narrow coding skills.
> Someone whose job is to simply type out English words is a typist. Someone whose job is to simply type out code is also a typist,
A programmer who is not a system analyst is not just typing out code someone else is written the way a typist types out natural language that someone else has written. They are still solving problems — sometimes fairly complex ones — with code. And there are probably environments where you need many more of them than people doing the rest of what Systems Analysis entails, and in a world where instead coding skills were devalued that would be a problem.
Software engineers are mostly know-nothing cowboys. The fact that you haven't previously heard of "Just Enough Software Analysis" confirms me in my impression that there are no standards and no regulation.
If you want to git gud, also read "Quality Before Design: Exploring Requirements" by Don Gause and Gerald Weinberg.
strongly agree, but I've found a lot of developers want to make a 6-figure salary and not have to do that part.
Furthermore, often times the stakeholders themselves don't have a concrete idea of what they want, it's nebulous in their head because they've identified a problem and a general direction and are relying on conversations with the technical team to help nail down the details.
That back and forth is the meat and potatoes of software development and its what separates senior from junior.
I think the situations that the parent comment is addressing don't really fit the description of "exploring the whole problem space with the stakeholders and deciding together what the software should do"; the programmer is often only looped in _after_ the stakeholders decided the requirements, and if there was any "exploration" done, it was deemed complete before the programmer had a chance to give input. If management actively chooses to work in a way that doesn't facilitate programming being done well, that's not programming being hard; it's just management being bad.