Hacker News Reader

But all that is part of programming. It's not made harder than it needs to be, it is that hard. Because exploring the whole problem space with the stakeholders and deciding together what the software should do is a central part of the problem of programming.

And it's ignored by everyone, the stakeholders don't realize it's a problem, the programmers know but think it's not their problem, and nobody includes it in estimates or plans the required time and meetings for it to happen.

> the programmers know but think it's not their problem

I disagree. We know it's our problem. But we quite often don't get a, or too little, say in the decisions made to manage that problem properly.

Because yes, it would be nice if we would "explore the whole problem space with stakeholders", and it would be great if we could "decide together".

What ends up happening quite often however, is some "stakeholder" saying yes and thank you to whatever he-who-holds-the-money (or his career-advancement) says, with little to no regard whether the promised addon/feature/whatever is feasible and/or accomodated by the existing architecture. Suddenly, the space that needs to be explored may be galaxy-sized, or moebius-strip-shaped.

What also tends to happen: The "exploration of the space" is outsourced to some consultant or architecture astronaut, who wont write a single line of code, and certainly won't have to deal with any unforseen difficulties.

It would also be nice if, once the "exploration of the space" is done, devs get to implement based on what was explored. Which, given the aforementioned unforeseen properties of the problem domain, is laden with unforeseen change as it is. What tends to happen quite often is that in addition to that, requirements, and thus the problem domain change mid project...sometimes multiple times.

Congratulations, you just rediscovered the pains that led to agile.

The key insight is that it is incredibly hard to capture a system that has the flexibility of handling every past, present, and future edge case while providing consistency for the happy path and preventing "bad" things from being done.

XP and the proto-agilists said, as such, that it was futile to try and capture the whole problem because it turns into contract negotiation and pitting stakeholders against each other. Further, they found out that what stakeholders said and what they really needed ended up being not-quite-right and it wasn't exposed until people actually used the software.

(Why? Because the stakeholders are representing a team, and sometimes they themselves are playing a game of telephone. And sometimes they describe a problem in a way that doesn't capture the real problem.)

So, the idea was to have the business representative in the same room. The problem was that those representatives still had to do their jobs, so they ended up burning out and quitting.

And that's how we got all of these product owners lying around - because we just punted the problem and went back to business analysts and gave them a slightly different mission. And that's why people talk about "dark agile." But I think it's easy to understand why the misalignment exists but very very hard to fix.

For internal development, I think the real answer is that the development team and any analyists should be required to do the job as if they were an intern. If there are multiple stakeholders, they have to do all of the jobs, at least for a few weeks, and meet the people who they are building for. Then, they can at least understand the full complexity and why all of these contradictory requirements exist.

An awful lot of programmers seem come out of college thinking that their problem is to find the fastest algorithm that solves some well-defined spec.

Requirements gathering gets short shrift, if it's mentioned at all. But really, it's the overwhelming majority of the job.

That's something that continues to baffle me whenever I interview people fresh from college. So many of them think their leetcode scores actually help them understand whats required in this job.

It doesn't, and it always baffles them when I don't even give them a whiteboard problem to solve. I don't need people who have memorized all graph search algorithms, I don't need people who can code them up in 6 languages without looking anything up. People who can google competently (or use a textbooks index) can do the former just as well, and LLMs can do the latter.

What I need are people who can take requirements, anticipate as much as humanly possible what else the client will do over time, and find a maintainable architecture that implements those and maybe is resilient when facing further changes down the road.

And that is a HARD skill, that leetcode and most of what CS curricula will not teach.

> Requirements gathering gets short shrift, if it's mentioned at all. But really, it's the overwhelming majority of the job.

Tell that to managers - oh wait - hire on algorithms and performance review on BS such as commit count.

> But all that is part of programming.

Nah, its systems analysis, a field which is largely either forgotten (in the “cross-functional means everyone is a nobdifferentiated developer” school of Agile teams) or replaced with “business analysis”, whose practitioners have less relevant skills (particularly, systems analysts were experienced programmers with additional process engineering skills, while BAs are nonprogrammers in a role that is often structured more as stenography than engineering) and “software architects” (who have a closer skill set to systems analysts, but have a role with a simulta!eously more abstract and narrower, implementation-focused orientation.)

Systems analysis requires (or at least benefits from) an understanding of programming, but its not the same skill

My bias is that "systems analysis" is a fake bullshit job, but I'd be happy to be wrong on this one. My guess is that if I asked you to define what a Systems Analyst is supposed to be doing, you'd tell me what I think a Software Engineer should be doing. Whereas you may think that a Software Engineer's job is to "write code". No. Writing code is like writing English. Someone whose job is to simply type out English words is a typist. Someone whose job is to simply type out code is also a typist, albeit one that needs to understand what a "for" loop does. The actual determination of exactly what the program should be doing, and why, and how that fits into all the other things it's doing, is Software Engineering. Someone who just wants to write "for" loops all day without doing that kind of analysis is not a Software Engineer, they are dead weight. Nothing. Doing what the compiler should be doing. But some business analyst who can't code is not a Software Engineer either; they are illiterate, like an novelist who "does the thinking" to figure out what all the characters should be doing, but expects someone else to do the actual typing. They are also dead weight.

Of course I don't expect you to debate all this in one HN thread. I'll browse through "Just Enough Structured Analysis" -- thanks for the recommendation -- but that will be in order to to make myself a better Software Engineer, not to become whatever a "Systems Analyst" is supposed to be. I guess if you expect a Systems Analyst to know how to write code as well as the other programmers, you are simply describing what I call a Software Engineer, and what you're thinking of as a "non-analyst programmer" is really just a typist that should be replaced by the compiler.

> My guess is that if I asked you to define what a Systems Analyst is supposed to be doing, you'd tell me what I think a Software Engineer should be doing.

Possibly: I'd certainly agree that what a Systems Analyst traditionally does is exactly software engineering, whereas whether “software engineers” do that, or are selected for the skills to do it, is rather mixed across the industry. The explosion of the Software Engineer title for programmers did correspond with the downswing in Systems Analyst as a role, but it often was not accompanied by the role and expected skillset of the retitled programmers changing, if anything, it was accompanied by a greater focus in selection in narrow coding skills.

> Someone whose job is to simply type out English words is a typist. Someone whose job is to simply type out code is also a typist,

A programmer who is not a system analyst is not just typing out code someone else is written the way a typist types out natural language that someone else has written. They are still solving problems — sometimes fairly complex ones — with code. And there are probably environments where you need many more of them than people doing the rest of what Systems Analysis entails, and in a world where instead coding skills were devalued that would be a problem.

Software Engineers are misnamed. They do not register with a professional body and commit to a code of ethics; nor can they be disbarred from working if the regulating Society of Software Engineers decides they have failed to work to the standards required.

Software engineers are mostly know-nothing cowboys. The fact that you haven't previously heard of "Just Enough Software Analysis" confirms me in my impression that there are no standards and no regulation.

If you want to git gud, also read "Quality Before Design: Exploring Requirements" by Don Gause and Gerald Weinberg.

There actually used to be a job description "Programmer/Analyst" or "Analyst/Programmer" - in the old days you were expected to do it all yourself.

Of course: commonly abbreviated as "Analrammer" ... on second thought, let's go with "Programalyst"

Good BAs that have domain specific knowledge are really valuable.

Sure, my issue is with the culture and idea of what is needed that has driven the structure of the workforce and the way many orgs treat the role.

Sorry - I was trying to agree with you :-)

If you're in the area, what books are some good guides on this topic ?

Because of when the culture shift happened that started devaluing systems analysis, a lot of the works are extremely dated in terms of the context into which they see the field fitting (written in the 1970s-1980s); Just Enough Structured Analysis (Ed Yourdon, 2006) [0] is a free mammoth PDF in which Yourdon updated his earlier influential work in the field; it is still quite dated when it discusses (e.g.) particular automated tooling, but it covers the scope of the field and a fairly comprehensive (but highly modular) approach to it in depth.

[0] http://zimmer.csufresno.edu/~sasanr/Teaching-Material/SAD/JE... (first live link besides a non-downloadable Google Drive link I could find, the primary distribution was Yourdon’s personal Structured Analysis website which became defunct not long after Yourdon passed away.)

> But all that is part of programming. It's not made harder than it needs to be, it is that hard. Because exploring the whole problem space with the stakeholders and deciding together what the software should do is a central part of the problem of programming.

strongly agree, but I've found a lot of developers want to make a 6-figure salary and not have to do that part.

Furthermore, often times the stakeholders themselves don't have a concrete idea of what they want, it's nebulous in their head because they've identified a problem and a general direction and are relying on conversations with the technical team to help nail down the details.

That back and forth is the meat and potatoes of software development and its what separates senior from junior.

> exploring the whole problem space with the stakeholders and deciding together what the software should do is a central part of the problem of programming

I think the situations that the parent comment is addressing don't really fit the description of "exploring the whole problem space with the stakeholders and deciding together what the software should do"; the programmer is often only looped in _after_ the stakeholders decided the requirements, and if there was any "exploration" done, it was deemed complete before the programmer had a chance to give input. If management actively chooses to work in a way that doesn't facilitate programming being done well, that's not programming being hard; it's just management being bad.

Sure, but this is a degenerate case of bad management. Leaving aside bad management and rigid role definitions aside for the moment, I think it’s fair to say that getting the requirements and tradeoffs right is the hardest part of Product/Biz System engineering. It definitely requires close engagement from technical folks. No corporate environment is perfect, but if that’s not happening at all then it severely hampers your skill development and upward mobility as a programmer, and may be time to look for greener pastures.

that's nice but if the "stakeholders" tell you a particular set of requirements at one point, and then a different one later that's not really anything to do with exploring the problem space.

The problem space is the business, and business needs change with the market

Yep, if your state passes a law between the time you start the project and now it doesn't matter, you still need to adapt the software to be compliant.

People are allowed to change their minds, even the ones labelled "stakeholders".

sure, but the thing introducing complexity at that point is not that you did not explore the problem space.

yeah, it's an unfortunate reality. Odd analogy, but it's like breaking up with someone and it hurting them. There are things you can do to help mitigate, but it is what it is.

Programming is a lot like design

- the customer tells you they need a design for their product, but they don't know what design they need, just what design they want. Now of course it falls back on you if you give them what they want and it doesn't work well.

- there are many ways to tackle the same problem, many of them work okay, some of them work well and look nice, some of them work well, look nice and are easy to manufacture and maintain. But it also has to be the solution they need and one where you can convince them they want it. Finding a good solution is a lot like finding the biggest common denominator of a large set of numbers.

As someone who worked in design before I can say that the most work and stress for me was bringing the clients up to speed, so I can communicate to them about the designs on one level. That meant explaining to them why their primary-color-powerpoint-comic-sans-poster communicates things they do not intend communicated.

I suspect that this sort of problem is present is a _lot_ of fields; it's not a programming problem or a design problem specifically, it's a human communication problem, and those can make just about anything more difficult.

Exactly, just think about any construction site on earth where throughout the construction they realize that they actually need windows or electricity.

I've just spent the last two days building out a system based on an API created by our clients internal dev team, only to have both the project manager and client throw in a massive curveball they knew about but neglected to mention - the API's been rewritten and replaced with a new one with a vastly different architecture.

That's ~12 hours of work down the drain and one demoralized developer.

It baffles me how this kind of thing continuously happens everywhere. Part of it seems to be that despite how many times you explain it to stakeholders, they don't seem to grasp that development takes time, and altering the plan mid way through is not only incredibly distracting but comes as a mental blow to the developer.

I think it is more that they don't respect developer time.

Any project of any size with any degree of coordination ends up with considerable wasted motion and "do what I mean not what I literally said." Maybe this wouldn't happen in an ideal world. We don't live in an ideal world. Even rigorous efforts to specify requirements (e.g. waterfall) didn't work very well and it's pretty much the definition of iterative development of anything that you throw a lot of stuff out but hopefully not big chunks (which 12 hours is not). That would be spending a month of something to find you were on the wrong track.

I think the point is that nothing about this is really that specific to programming; what we're talking about here isn't an instance of programming being hard but effective communication being hard. The parts of software development that are time consuming aren't very intuitive to non-programmers because there isn't an obvious way to show tangible progress, which makes it very hard to track externally, so it relies more heavily on verbally communicating the current state of things, and that introduces the potential for a lot of misunderstandings.

Designing in all sorts of fields involve long stages without much in the way of visible process as people work problems out. Developers aren't nearly as special as they like to think they are.

which is why you get in contact with their development team directly.

I've seen internal teams that thought the communication channels couldn't go dev to dev and it was a nightmare.

If you're experiencing this, consider that you have options if you're willing to take them.

I feel like communication of requirements is difficult too. I'm currently a business end-user, with developer experience, who is maintaining legacy VBA technology, trying to get an official system produced by a 3rd party contractor.

The requirements started out truly quite simple - we want a database and a nodejs web server.

But Technology are gate-keepers. We're not being trusted with what we want, or with that level of control over the system. Which then starts this cat chasing mouse thing of what the Technology will allow and what the business wants.

I've always vouched for "Internal IT" - specialised developers who work closely with the team, even doing the jobs users do, to self-identify what technology is required and kickstart these projects.

Edit: in this particular project, I handed over numerous diagrams of how the system should be architectured, and Technology BA's came back with a proposed set of technologies (SASS) that simply wouldn't be technically feasible... It's really made me lose hope.

This actually touches on one of SV's durable advantages in the software world: the realization that if you want good software your engineers must be involved in the requirements gathering, design, and scoping phases. Critically, the business understands that engineering is the most authoritative voice on cost and since all business activities should pass through an ROI analysis filter engineering needs to be in the room.

Keeping engineers in the room then adds a positive feedback loop: as they understand the business better they start to offer up new opportunities to the business that have surprisingly low cost.

(I will caveat that for sufficiently solved problems you can generally involve software engineers less)

I think there's something to this. When I write programs to solve my own problems I find the process easy and enjoyable.

How big are the programs, how often do your requirements change, and how many years or decades do you develop and support them?

Because tiny one time programs are indeed fun for a while until they “grow up”. :)

Small, rarely, and usually for one or more decades :)