Hacker News Reader

Comment by wpietri | Hacker News Reader

He's not wrong. Having been involved in the Agile movement since before the term Agile was coined, I think of Scrum as the least interesting of Agile processes, but also the most successful in terms of adoption.

I used to think that was a contradiction. Now I think it's almost inevitable. I wrote more about it elsewhere [1], but the basic deal is that most companies have other priorities than being effective, so the processes that dominate will reflect those priorities.

On average, the first priority of managers and execs is maintaining the power structures that make them a big deal. But true Agile processes are about empowering teams to self-organize around serving users. Ergo, Sturgeon's Law [2] applies here too: 90% of company software process is crap.

I think reason why Scrum in specific won out is that it combined a process framework unthreatening to executives with the multilevel marketing scheme of Scrum certification. Executives got to keep doing what they were doing, the workers got a "didn't stop breathing" certificate, and coaches got paid. Win, win, win!

[1] http://williampietri.com/writing/2011/agiles-second-chasm-an...

[2] https://en.wikipedia.org/wiki/Sturgeon%27s_law

View on HN

Scrum is badly misunderstood and maybe that's a failing in and of itself but it's really a victim of the developers who sold it as a magical process. Scrum simply can't be implemented as a purely developer process. The backlog exists as a rolling contract between dev and product owners and sponsors.

The most common failure I see is when project leadership agrees to a fixed scope and timeline then tries to execute in an agile way. Agile is the contract. But it's also a tough sell. It requires a high degree of trust to say "we'll pay you $XM in exchange for X sprints of whatever we prioritize with no fixed end state" but that's what you need to be agile. If you layer sprints on top of a fixed timeline, you're just adding needless complexity since your end state is predetermined and how you get there is irrelevant.

Have you seen any methodology work consistently when dealing with fixed deadlines? The reality when dealing with large contracts, as you mention, is that there are almost always timetables with expectations. This poses an inherant problem due to the unreliability of estimates, so either quality or features must be sacrificed if the timeline is in jeopardy. My only experience in such an environment was using some hybrid waterfall/sprint methodology that mostly worked fine, but i suspect that was more to with the general culture and people involved than anything else.

I think no industry producing anything new ever found a working process to predict and keep the timeline. (If omniscience really existed, it would have more lucrative applications.)

Developing new aircraft, building a new ship, building a custom-designed bridge (most of them are) are processes that often run out of time and / or budget.

If you want predictability, you want repeatability. But in software all reliably repeatable parts become automated away.

Absolutely. And I'd add that predictability is dependent upon not learning anything over the course of the project. To be predictable either you know everything that matters up front (which is only true for trivial projects with no competitors) or you refuse to learn anything along the way (with, e.g., a big-bang release at the end).

But I think good projects release early and often precisely so that they can learn as they go. At which point predictability goes out the window.

? All the specified above endavours - usually run into what one could call the end of the map. Here be science! If you run into e.g. new material science because your goal stretched beyond the scope of what has been done previously, you can blame the engineer for not telling you that you will leave the boundary of the knowledge of a field- you can not blame him for the estimates beeing wrong.

AFAIC, the only thing that works with a fixed deadline is a variable scope. Which is basically a tenet of Agile.

The alternative is some sort of sub-iteration hybrid where you attempt to fix deadlines for much smaller units of work and constantly revise.

This is OK for some industries. But 2/3rds of a car doesn't quite sell...

One thing about agile is that usually companies not only set fixed deadlines but also fixed feature set; which usually means lower quality.

That in turn means technical debt; which puts future deadlines and feature sets at risk.

Since in SCRUM as a team you try to not compromise on quality the only way out of this is to push back on deadlines or feature set (or both).

So instead of blindly executing orders; the development teams can push back on deadlines.

Most of the time these are actually negotiable. Even the "hard" deadlines....

It doesnt mean the dev teams always win however they are better equipped to inform "management" about the consequence of the deadline.

Exactly. Probably 90% of the burden on a successful agile project is in setting expectations with stakeholders for an iterative delivery without a fixed scope. It can be a tough sell but it shouldn't be. If you've ever done a top line estimate on a fixed scope project, you know it's just absolutely not possible. Why set a deadline you know with 100% certainty you're not going to hit? Just bake it into the contract and upfront expectations.

Nope, never.

I have seen more success in cases where teams estimate for 90% percent confidence instead of 50% confidence. By that I mean "it should almost never take longer than that" vs. "it will probably take that long." Unfortunately, to do that, you need enlightened business management that appreciates the difference between an estimate and a promise, as opposed to business management that pays lip service to the distinction.

> "it should almost never take longer than that" vs. "it will probably take that long."

This sounds like a much better way to estimate. Do you have any links to content discussing this?

Here's an article that explains the phenomenon much better than I can. It made its rounds in HN a week or two ago: https://erikbern.com/2019/04/15/why-software-projects-take-l...

I haven't seen a good way to turn the concept of "it should almost never take longer than that" into a concrete process. I've always seen it as a gut check, often implemented as "take your initial estimate and double/triple it." What I actually see is that a lot of teams implement that but walk back the doubling when the business/PM delegation persistently asks for more in less time. Only in really engineer driven cultures have I seen engineering teams successfully push back.

On the actual use and calibration of confidence intervals - https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-mea...

DHH has a nice methodology that works with fixed deadlines: his basic rule is that features can be dropped/simplified, but deadlines must be met.

https://www.amazon.com/Doesnt-Have-Be-Crazy-Work/dp/00628747...

He's applying an old concept. The Iron Triangle has been in use since the 1950s.

https://en.wikipedia.org/wiki/Project_management_triangle

If time is fixed, then scope and/or cost must change.

This is true, but fixing the deadline and not committing to the scope requires clear communication by the project management / sales teams with the clients about it, and this is rare.

Yes, absolutely. Many times. I've done a bunch of projects that were tied to immovable event dates. The only way to deal with it is having very flexible scope. Typically, these have been more "creative" projects and not tied to a lot of critical functionality so we just make sure we keep our MVP quite small and treat everything else as iterative enhancements so we can cut off whenever we're out of time and still have something presentable.

> Scrum is badly misunderstood and maybe that's a failing in and of itself but it's really a victim of the developers who sold it as a magical process.

Even if so, that would be in itself a failure of Scrum. A process that works only sporadically and only with exceptionally well organised groups ain't really all that much.

> It requires a high degree of trust to say "we'll pay you $XM in exchange for X sprints of whatever we prioritize with no fixed end state" but that's what you need to be agile.

...no? It requires a well-defined end-state, and that includes not defining the irrelevant. It's not really "give me $XM and we'll deliver anything from accounting software to a really nice puppet."

I have almost never seen Scrum sold as magical by developers. Executives? Yes. Consultants? Yes. But developers?

I agree that fixed-scope contacts cause a lot of problems for Agile approaches. However, they also cause a lot of problems for non-Agile approaches. If I have to deal with supposedly fixed-scope situation, I'm going with and Agile approach.

There are two basic cases. One is that scope is truly fixed (which is rare). In that case, having a new releasable version every week with highest-priority features first is excellent risk management. When the date comes, you'll have something to ship. You also get to continually validate and improve internal processes, so you're more likely to be using the time available effectively.

The other is that scope was fixed in the contract but is in practice variable. So every week you deliver something to the customer. Every week you build trust. And every week you encourage them to do tests, deploy early, anything so they start getting value. At that point they get user and business feedback, and come to you with changes. That's the point where you start shifting from fixed-bid to time-and-materials. Maybe you keep to the same budget, but now they're getting more for their money. Maybe they look at the backlog and say, "Wow, there's still lots more value to deliver, let's keep going."

>I have almost never seen Scrum sold as magical by developers. Executives? Yes. Consultants? Yes. But developers?

In many places, especially smaller ones, project managers are also developers (or lead developers etc), and they often drink the Scrum kool-aid.

Seconded. I've also seen it promoted by developers who are in what one could call a "honeymoon period" of their careers. First or second job, probably learned programming at university so every task is an interesting challenge, SCRUM is their first agile methodology, they don't have enough broad knowledge about programming and the industry to become disillusioned and cynical. I've had such people evangelize SCRUM to me, with pride in their eyes, like it was the best thing since sliced bread.

I’ve noticed this, too. For some reason a junior developer on my team 2 years our of school was made into scrum master, and if we so much as sneeze there’s no JIRA ticket for it he starts whining.

It was heavily hyped by developers about 10 years ago. Now they've all given up because they didn't learn the right lesson.

There are in fact two kinds of Scrum (or rather, Scrum implementations), both of them compatible with the Scrum guide: ‘Left to Right’ Scrum (backlog-driven, implementation-focussed) and ‘Right to Left’ Scrum (goal-oriented, iterative).

Unfortunately Scrum is too often explained and implemented that first way, leading to the anti-Agile feedback we see on HN with some regularity. The second way is much more compatible with complementary tools such as Lean Startup and Kanban and I wonder if this (to me very welcome) non-exclusivity explains why it is less talked about.

[1] https://blog.agendashift.com/2018/07/04/righttoleft-works-fo...

"Right-to-left scrum" seems like a hollow phrase made up by that consulting firm you linked.

From what I can tell, it has all the meaninglessness of an empty buzzword used primarily as a method of trying to engage people and give an opening on selling their services by stating "oh if you don't understand the difference sit down with us and let us show you how different and better it is."

It looks like if scrum doesn't work they call it "falling left to right scrum", and if it works it's their "brilliant right to left scrum".

Even by their own definition they both have a backlog that gets prioritized and selected each 2 weeks into a sprint backlog, and executed during the sprint.

The rest is just hand waving. "One is goal focused vs the other is backlog focused. Oh yeah but we do put our goals in the backlog." So they're both backlog focused then? The only thing they're really sayings is "When prioritizing tasks, make sure they accrue to something and aren't just random work." Which is both obvious, and of course too simple to write and sell a book about, so instead this whole other terminology is made for it.

Thanks for sharing the link though.

This logical fallacy is sometimes called No True Agile.

it's really a victim of the developers who sold it as a magical process

Developers may advocate small-a agile, but I have never heard of an actual developer pushing for Scrum, SAFe, or any of the name-brand "agile" methodologies, that come with expensive consultants and certifications and "coaches" and conferences and a whole ecosystem around them. These are things for project managers for the benefit of project managers who see project management the main deliverable l in and of itself. The same with JIRA actually, it's not a software development tool, it's a project management tool, sold to the same people who previously bought MS Project. These things are all overheads in the software development process, not enablers, as another poster says, the goal of most organisations is not to be effective at their stated mission, but to maintain their internal power structures.

>Scrum simply can't be implemented as a purely developer process.

aka, WaterAgileFall.

aka WaterFail

>> developers who sold it as a magical process

If I ever find a developer who can sell, selling is not a developer's USP in general, there are exceptions and I would hire those exceptions instantly!

Scrum won out because it was waterfall in disguise. Commitments and sprints become terribly destructive over time.

> Commitments and sprints become terribly destructive over time.

Could you elaborate?

You're asked to deliver a feature that would take 4 weeks to implement without tech debt in two weeks. This requires sacrifices like ensuring network connections remain alive, handling exceptions, optimizing queries, abstracting functions for re-use, handling all the data input scenarios, etc.

This creates a confusing code-base with lots of IF statements, copy/pasted code, data modifications from several areas of the system, etc.

You'll fix a problem with data getting into the database incorrectly and realize a couple sprints later, fixing that bug caused other bugs elsewhere.

Automated testing doesn't catch it because automated testing only handled certain scenarios.

It's a mess.

>You're asked to deliver a feature that would take 4 weeks to implement without tech debt in two weeks.

Well there's your problem. How is that inherent to scrum?

Because it's a sprint. You've got a finish line and you're racing towards it. Businesses don't function without prediction, and refactoring gets cut before roadmapped features. If feature development is a bottleneck in company growth, debt will grow, quickly.

“Sprints” are, pretty much by definition, a non-sustainable pace.

Agreed. I regularly ask "Why are we running a marathon in a thousand sprints?".

Besides tech debt, a concern I have that I don't see brought up is burn out. With Scrum, every action you perform is micromanaged and with a push for "high velocity". There is no proverbial breathing room in this where the pressure lets up. At least with waterfall (for how we did it before Scrum), the windows of high pressure times were shorter. During the beginning of our 6 month waterfall, in parallel to spec work we'd be taking care of tech debt or implementing our pet feature and it was a time of mental recovery.

imho a problem with the implementation of SCRUM, but not necessarily restricted to SCRUM.

In a perfect SCRUM world you could not be forced to implement this in a time not of your own estimate. A feature that is so big that it takes more than a few days to implement has to be split...but well we all know how this plays our in reality most of the times...

>You're asked to deliver a feature that would take 4 weeks to implement without tech debt in two weeks.

There is something odd with this, if you did the estimation. Shouldn't the story points/sizing/estimates preclude this?

Doesn't the team pick the items for the sprint that they think they can complete?

You can never take a 4 week story in a two week sprint. When would you ever be able to complete it?

By breaking it down into smaller stories?

My experience shows that proper testing and documentation is the first thing that management wants taken out of the story, often with the excuse "We can handle that in a later sprint." But since your life is a neverending series of sprints (note: that's actually an ultramarathon), and management gets to pick priorities, you may never return to the technical debt.

I have not worked in a scrum environment in a couple of years, but when we did it those items were not included in the story. They were part of a definition of done that the team did not really talk about publicly.

Refactoring stories were also rare, it was just sort implicit in the task, unspoken as part of doing it.

I'd imagine that is why they did not get cut.

But yes, management picked feature priorities, but left the implementation up to the team. So at any given time were were working on the "most important" thing from a business perspective.

The most successful (but still crappy) solution to this that I have seen is developers undertaking guerilla tech debt work. This can be done by bundling maintenance work into an existing task where the two are actually not that related or by simply carving out bits of time between official tasks.

While it "works" it is certainly not ideal that people have to go off the reservation to ensure that the project doesn't implode in the future due to the accumulation of buggy code, performance problems and likely security vulnerabilities.

> proper testing and documentation is the first thing that management wants taken out of the story

That's not breaking down a story into smaller stories though, that's simply not doing some of the tasks necessary for a story to be complete. "Story" is not a synonym for "task"; a story is user-centric not about internals. "Write the documentation for this feature" is not a story and "write the tests for this feature" is not a story, they should be part of your definition of "done".

That would be ideal, but we are devs, we don’t get to touch the stories other than to mark them as done...

You're not doing Scrum then. One of the few meetings that Scrum dictates is "backlog refinement" where the dev team works with the PO to get the stories into a workable state.

Yeah, we do waterfall and scrum with the worst of both.

If you can't touch the stories, you can ask whoever does to break it down. If they don't then you shouldn't ever pick it up into your sprint. If they insist and it doesn't work out then you should call out the incident in your retrospective and hopefully get everyone to agree to do things differently in the future.

Sadly, it’s more like the dev team points out that we really cannot work with these stories then management telling us to deal with it while the BA’s suck their teeth.

Then they ask us why there isn’t any progress.

Yep. :)

https://www.brightball.com/articles/reality-driven-developme...

Scrum is iterated waterfall. By iterating faster, inaccurate estimation is shown up sooner. On the other hand, developers are treated like cogs in a feature factory, munching through backlog items fed to them by product managers.

I think it works well enough, for a few years. I don't think it's sustainable - the blinkers of "sprints" encourage growth of tech debt because nobody has an eye on the future and Product won't prioritize refactorings, and sprints are too short for devs to sneak refactoring into the schedule.

I was explicitly told not to write unit tests because they took too much time, which required me to spend entire days retesting almost 100 scenarios when the business logic changed.

Of course the business didn't know all the scenarios at the beginning of the feature development and didn't care because: iterative development means we'll figure it out later.

As you've discovered, good automated testing is mandatory for iterative development. If execs tell you to do something unprofessional, it's ok to say no. Indeed, if we're going to think of ourselves as professionals, I think it's mandatory.

> if we're going to think of ourselves as professionals

We need to unify around these principles, possibly revoke memberships for unprofessional behavior. One dev getting himself fired changes nothing..

I'd love to see that kind of organizing happen.

But one developer quitting rather than do bad work does change something. At the very least, it means that the developer can find another, better job. But it's also an opportunity for companies to learn. And for those who don't, good people refusing to prop up bad companies is a step forward.

I'd also add there's a a lot that can happen between refusing to do bad work and quitting/getting fired. Saying no is the beginning of a negotiation. A negotiation that won't happen if we just say yes all the time.

All straight out of the TDD playbook.

A lot of devs trap themselves by insisting they can write the tests after. Once managers know the code exists they want to use it, or move on to the next thing people are breathing down their necks for. And code written without tests is difficult to test so becomes a self fulfilling prophecy.

In fact it’s such a reliable mechanism for self sabotage that I look at carefully at people who bring this on themselves and try to figure out if it’s naïveté, learned helplessness, or malice.

The biggest problem with Scrum is it lacks any sort of design phase. You do the minimum. Oh, it doesn't work quite right? We'll fix it in the next sprint...

The "spiral" model is closer to a true iterated waterfall. I've seen it used successfully in more mature companies.

It's like "build one to throw away", in that the goal is to rapidly explore the feature space and discover useful features, but what actually happens is that it's a trap and you won't throw away something that works no matter how poorly implemented.

absolutely. If you ever see a group afraid to produce a prototype, it's because they're afraid management will try to ship it as a final product.

The effective strategy is to build an oblique, fatally limited prototype, which can not possibly be mistaken for a shippable product. The bad prototype can only be used to test the hardest parts of an idea and must have major holes in it with no way of filling them. It should also have a largely fixed timeline to ensure it is put to bed before it gets "hamstered" into the shipping product.

absolutely. If you ever see a group afraid to produce a prototype, it's because they're afraid management will try to ship it as a final product.

I once had a manager who, on the subject of upper management wanting to ship the prototype, would say “the difference between dev and prod is one letter.”

I completely agree here. Instead of understanding the volatility of the requirement(s) and designing for that, we refactor and react.

You're confusing "investing in design" with "having a design phase". I agree many Agile teams underinvest in design. But so do many non-Agile teams. The problem isn't the lack of a formal phase. The problem is not taking it seriously.

Yes, you're correct... I didn't mean a "formal" phase, so much.

You get what you measure and Scrum has no measure at all for design. I’m now curious to see some projects that Ken Schwaber worked on.

I'm certainly not defending Scrum. But I think one of Scrum's big problems is relying on proximate measures rather than actual results. It values output much more than outcome. So I think the way to get good design isn't to try to measure design. It's to release early and often, making sure that products fulfill their purpose.

I think our whole industry is full of proxies for the things we actually care about, which is for instance why we rant so often about the hiring process.

For another example, look at what we talk about with Clean Code or SOLID principles. We have that old joke about the two hardest things are naming things, cache invalidation and off by one errors. I think the fourth hardest one is resolving merge conflicts.

You work carefully on code for weeks or months and then in one fell swoop it all gets broken when you aren't looking at it anymore. It's really bad for morale.

If you read through all of these Best Practices with the notion that merge conflicts are dangerous, then you start to see them as designed to keep conflicts from happening. Putting related code together isn't just about reading comprehension. It's also about keeping unrelated code apart. You and I can work on unrelated stories without ever doing a 3 way merge.

And if you look at merges as the problem, you start doing things like alphabetizing data structures that don't have a strong cohesiveness to them because then when you and I add a feature to the same data structure, we aren't both modifying the bottom of the file. I put something under T and you put it under G.

To put this another way, if in some perfect future someone invented a merge tool that never screws up, so that we only get merge conflicts when two people are accidentally working on the same feature/bug, we might start to question all of these practices as overwrought.

It doesn't lack a design phase. Where do you get this information from? It isn't explicitly prescribed, but no one prohibits you to make designs.

Experience. I've been developing software for 20+ years. In practice, due to the nature of agile / scrum, design is very minimal or thrown out the window entirely. It encourages short term thinking. Rarely is anything beyond the "current sprint" considered.

Scrum is unsuited to actually building a decent framework for you application.

We tend towards 4 or 5 sprints in which nothing is completed, only to suddenly have a flood of completed things the next.

Then that's not an Agile process. That sounds more like mini-waterfall.

It's perfectly possible to deliver small units of work weekly. One still gets frameworks while doing that. It's just that the work of making the framework happens over time, not up front. Which is good, because good designs are based on good information, and the beginning of the project is when you have the least information.

>I think it works well enough, for a few years.

"Fortunately," hardly anyone in Silicon Valley plans to be working on the same codebase in a few years. There's a good chance the problem space won't be relevant anymore by then, and on the off chance it's still funded, the new team will rewrite it whether or not it's good.

>sprints are too short for devs to sneak refactoring into the schedule.

As I've gotten more senior, I've just gotten more brazen about doing this less sneakily.

Is the Silicon Valley representative of software development though ? Because on the other hand we have professors telling us that the average (surviving?) software lifetime is 20 years...

This is just intuition but I suspect that the lifetime for software is u shaped. Much of it is very short lived but software that lasts more than 1-2 years is very likely to live for a decade or more.

A lot of 6 month old code gets thrown away either because its been rewritten or because it didn't achieve its stated objective. Meanwhile a bunch of companies are relying on systems that were first created in the 90s because that software achieves its objectives and the cost justification for a rewrite isn't there.

There are many 20 years and older code bases currently. But if you start a new code base today it might get rewritten many times in the next few years because software users no longer have the power to stay on an old version due to the SaaS paradigm.

> Scrum is iterated waterfall.

No, it isn't. Iterated waterfall is at least as old as the first paper discussing waterfall, but while scrum mandates interations, it doesn't mandate much about how work is done in the iterations, and specifically does not mandate the process steps associated with waterfall; further, it emphatically rejects the role separations and handoffs associated with waterfall during the iterations.

> munching through backlog items fed to them by product managers.

That's...not actually Scrum, as it implies that either the role of Product Owner is taken by a PM outside of the Scrum Team or that that the Scrum Team is not self-organizing, either of which is a significant (even if common) deviation from Scrum.

> the blinkers of "sprints" encourage growth of tech debt because nobody has an eye on the future

Tech debt should manifest in reduced velocity which should be noticed, taken as a signal of a process defect, and addressed in the Scrub Team’s various inspection and process adjustment points.

OTOH, if the Scrum Team is properly cross-functional and self-organizing instead of having a non-team-member imposed as Product Owner, then including appropriate restarting as components of completion of relevant backlog items shouldn't be a problem.

What percentage of Scrum teams do you believe are "properly cross-functional and self-organizing"? And could you point me to examples of people losing their Scrum certifications for not living up to that standard?

This is the great con of scrum certification — you can always point to some team and say “well that’s not the official version of scrum”

Which is a really impressive con. Generally the point of certification is to make clear you're getting the official version. E.g., doctors and lawyers self-police because they know it's harmful to have quacks running loose. But somehow Scrum has been able to keep making money despite not bothering with that.

> What percentage of Scrum teams do you believe are "properly cross-functional and self-organizing"?

About the same percentage as that of “Agile” software development shops that put people and interactions above processes and tools.

OTOH, at any place that is considering implementing either, there are decision makers who can influence (or in the case of Scrum more than Agile, authoritatively direct) whether or not that's the case, so for them, at least, it's worth distinguishing between problems with Scrum as prescribed and problems which often occur because decision-makers decided to ignore key parts of Scrum-as-prescribed.

Most nominally "Agile" shops are in effect doing Scrum, so I'm not sure that answer helps your case. Either way, it sounds like we agree the "good Scrum" shop is at best rare.

Given that, I think it's worth considering that the problem is Scrum. Especially given that Scrum is not just a process, but an organization and an army of "certified" people that sell services.

When something generally doesn't work for its stated purpose but keeps making money, I think it's worth asking what its real purpose is. E.g., things like crystals and psychics. As Eric Hoffer wrote, “Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket.”

> Most nominally "Agile" shops are in effect doing Scrum

No, most nominally Agile shops are also nominally doing Scrum, but actually doing not-Scrum in a not-Agile context, largely due to sabotage of both core principles of Agile and foundational elements of Scrum by management.

It doesn't matter what name and superficial ritual you put on the process if it's all window dressing over top-down disempowering command-and-control by persons who are neither doing the work nor experts on the work.

Which isn't to say that the Agile and Scrum bodies of work aren't part of the problem: neither really addresses as a key point how the team effectuates ownership of process and how interaction with management works, which means those gaps get filled in (or rendered moot, in the first case) in ways which compromise what those bodies of work do prescribe because.

Lean, which comes from basically the same perspective (while they don't cite exactly the same values as expressed in the Agile Manifesto, being Lean essentially implies being Agile and vice versa) and is a good body of knowledge to draw from alongside Agile, is better in this regard, and so places nominally drawing on Lean seen to be more likely to be doing what they say, because Lean doesn't leave as much of the core vital parts without good guidance.

This. I've run programs that purport to be agile but the work input stream was waterfall and disconnected from the means of output so it was really timeboxed waterfall, unfortunately my management did not agree to re label. You know it is real Scrum if it is painful for the business team and they come to realise that now it's software all the way down.

You can scream "That's not actually Scrum" untill the day you die. It doesn't change the fact that that's how it is perceived and done in almost every place that says they do Scrum.

A sibling comment to yours made essentially the same point, and I had already addressed it, before you posted your comment: https://news.ycombinator.com/item?id=20020206

> Tech debt should manifest in reduced velocity which should be noticed, taken as a signal of a process defect, and addressed in the Scrub Team’s various inspection and process adjustment points.

This process adjustment usually means "sorry about your vacation" and/or "you aren't doing enough overtime".

Yes, if external actors rather than the self-organizing teams own the process, that is probably the first response, but that won't actually stop the buildup of tech debt and the velocity impacts, though it may produce a one-time, non-repeatable improvement, so even in that case if someone (even the wrong actor) is monitoring velocity and doing process improvement, there will be an impetus for further change.

(Of course, if the wrong actor is monitoring velocity and controlling process, then the team has an incentive to mask the effect of tech debt but continuously adjusting estimates to maintain the illusion of constant velocity, a d avoid the first bad response, or, if that opportunity is missed, to avoid the subsequent external interventions.

Waterfall was coined by a paper discussing why waterfall wasn't a good way to do software projects.

You aren't wrong, either. I found Scrum to be a series of waterfalls that weren't well thought out. "Iterative Design" essentially meant, "We aren't sure what the button should do exactly, but we know we need it there and it kinda has to do this and we'll figure the rest out for the next iteration."

That caused so many problems with tech debt.

Stories began to take longer due to the increasingly large and discombobulated code base, but the expectation was that we continue to deliver the same number of story points each sprint.

Then we got blasted for losing pace. "Why are you under-performing? We are just adding little features. You've already implemented this button on another form before? Why does it take longer the second time? You already know how to do it!"

It became absurd and no one could hear or understand what was happening was easily predictable and in fact -- was predicted by several members of the team many months prior.

When the project management was informed of these predictions now having come true, the development team was accused of intentionally causing the delays.

This mirrors my experiences to a tee.

The most upsetting experience in my career thus far was being belittled by an incompetent project manager.

"How could it possibly take that long? How do you not know how long it will take?"

The happiest times of my career have been when we're lacking a PM -- coincidentally the most successful!

Best PM I ever had dropped by every week or so to ask if we needed anything.

That's what the best ones do.

> "How could it possibly take that long? How do you not know how long it will take?"

Well, if they really said that, you can just dismiss the comment as incompetent. A witty retort may be in order if a non-technical manager is in earshot, e.g.:

"How long would it take for you to earn a green belt in karate?"

"Hmm, what does a green belt entail?"

"Exactly, you don't even know what you don't know, yet."

> A witty retort may be in order

I don't think this will score you a lot of points.

Depends. Are we talking self-esteem/dignity points, or good doggie/loyalty points?

I'd guess career-progression points, or even keep-getting-a-paycheck points at worst.

This low-stakes exchange won't affect much, it's about putting Mr. Incomp in his place. Remember it's a PM we're talking about, not "the Boss."

They did, and I replied

"We don't know because we've never done this before, and we can only compare to the most similar work we've done, and any existing data."

Before I could say more, they I interrupted: "You have to know, it's your job, or you don't know what your doing," or something along those lines. It was the most heated I've felt. I know what I'd reply with today, but I was younger and more naive then.

What would you reply today?

Not OP, but I will answer with the second option and ask him what I should do?

"Yeah, well I had sex with your wife!"

Or just offer over-inflated estimates since they won't accept something realistic anyway.

I commented above that the initial questioning was fine.

> "or you don't know what your doing,"

That though was unprofessional, and should be addressed as such.

In these situations I shift the conversation to what is the PM really after. I explain that I can tell them any estimate they want (or they can make it up on their own - no need to even discuss), but does that really do them any good when it is missed? There is usually some pushback at first, but when items are delivered within estimate a few times in a row they realize it's better to take whatever hit up front than it is to miss and adjust.

Also, someone asking how could something take so long is not a belittling question. I don't see any problem with either question.

Personally I believe this is one of the major gaps in Scrum, and there is no prescripted way to solve this problem. XP's focus on technical aspects is much better in this regard, but the possibility of runaway technical debt still exists.

There are so many places where this can sneak in and become an issue, too:

- The development team can groupthink, underestimate and/or fail to account for technical debt payback in planning poker. They may estimate the cost for the new feature, but not include any time for refactoring to keep the codebase healthy

- The Scrum Master may not shield the development team from pressure to deliver coming from the product owner, which may cause the team to cut the above corners

- Codebase health may be split out from vertical stories into its own tasks that get deprioritised by the product owner

- Even if the development team identifies the need for codebase health, this may not be communicated back to the product owner in a way that makes business sense ("but we can still get this feature out the door quickly and fix the mess later, right?" - yes, once or twice, but the downside is a slowing of velocity)

- Even if the team started well with ground rules the Scrum Master is supposed to enforce surrounding technical quality, the Scrum Master may not be effective in enforcing the process - either because they lack authority or because they lack the ability or knowledge to do so

Among several others.

In your situation, the big red flag is that the team predicted the issue many months prior. The process has failed to extract that information from the team members and deliver a consensus between them and the product owner.

Though ultimately, as with any project, if the product owner has more organisational authority than the development team and chooses to exert this authority, no process will be able to save you.

It's easy to say something is shit.

The tricky bit is having a solution and implementing it.

Having something well thought out is hard work. Is hard work rewarded? Is initiative to even try rewarded? I'd say the answer is no.

The answer becomes yes in small teams that are given time, freedom and resources, or in times of crisis (same thing as small team really, minus the time bit), as far as I can tell. Otherwise, most people's natural tendency (including me) is to phone it in. We'd be wise to let people in software development and many other professions that are not manual labor, to switch careers after 30-35 if they haven't done anything worthwhile by then.

Working with people who have to phone it in for the next 30 years and know it is modern day hell for anyone with an iota of ability.

It's permanent compromise and insincerity, or being an outcast that's temporarily tolerated for doing 5-10x the work of the person next to you.

Because Tech debt and I reading messy code base never happened in waterfall...

The problem is always : features and customer request over bug fixing or design fixes.

The system, project type does not matter. That's why some PO recommend 10 percent allocation to devops / ops, 10 percent to improving code base . And "no bug survives the Sprint" philosophy. And what is left,is you next feature or customer request.

For every example of scrum operating badly there are examples of scrum working well. The common denominator in all of those situations - good and bad - is the project and middle/upper management.

In business process analysis, when a process is dependent on individual people, you don't have a mature process but an ad hoc process.

In other words, Scrum methodology in itself is not mature as a methodology and depends on individual fiat, just like any project that doesn't have any process at all.

The only tangible benefit to Scrum might be paying lip service to development departments while firmly retaining the status quo.

Scrum is no more or less dependant on individual people as any other methodology.

The rest of your post is just warping your initial conjecture to arrive at conclusions you'd already decided upon. I neither agree with those conclusions nor the chain of twisted logic you used to arrive that.

I'm not going to stand on a soapbox and sing for the glory of scrums. People - like yourself it seems - can get very tribal when talking about such things. Which is as weird to read as it is pointless for you to argue. In my experience leading different teams using different methodologies, the thing that really makes the big noticeable impact is people and not methodologies. If you work in a blame culture or have colleagues in your team who don't follow process - then whatever process you put in place will be undermined at every opportunity regardless of it's methodology. However if you work in an environment where people respect one another and want to collaborate in getting work done, then you pick a methodology that works best for the day to day work (eg project work or support operations) and for the way people like their work organised. All the rest of the arguments are superfluous.

> good and bad - is the project and middle/upper management.

Yep. People do not realize that the whole company has to adopt agile/scrum for it to work. It's a shift that many companies can not or will not make. They are tied to fixed deadline, fixed scope for various reasons.

I mostly agree with you but I wouldn’t go so far as to say the whole company has to adopt it. However you do certainly need the support from the wider company. I’ve seen scrum work really well despite waterfall being the predominant methodology in that business - there people were happy to embrace the preferred working styles of whichever team they had to liaise with. I’ve also worked in places where scrum failed badly despite the CEO pushing for it. Those places usually suffered from a blame culture that was also the CEOs doing and that blame culture meant that everyone was spending their time working against the best interest of any of the other teams. By “teams” I mean project managers, sales guys, support or operations teams (if they differ from your developers and/or DevOps), finance staff, directors/upper management and even your own paying clients (if you’re a hired service) etc.

I’ve noticed comments for and against scrum are often so focused on the technical aspects of the methodology that they overlook the human aspect. Which matters more in my opinion.

Waterfall with little or no modelling. If you write the wrong system, you are left with nothing. With waterfall you could at least start over with your design artefacts.

Scrumfall is, sadly, widespread

I love these commitments. When my Scrum leaders asks me why we didn’t finish what we ‘committed’ to doing these two weeks I want to strangle him (nothing personal, I do otherwise like the guy :P).

The terminology is so incredibly developer hostile.

Which is why the scrum handbook went with the term forecast instead. Many years ago.

Scrum wins out at companies that did not exist in the waterfall era, staffed by developers and engineering managers who have only ever worked in startups. Similarity to existing process may explain a large part of the adoption in places that had existing process, but a lot of Scrum shops didn't.

I agree. The last "waterfall" company I worked for was Novell. In my opinion, all the Novells of the world glommed onto Scrum because it was shiny and new and allowed them to call all their bloated and unnecessary project managers "Scrum Masters" and all their wasteful status meetings "Stand-ups." It was just the same old garbage with a different name, but hey, We're Scrumming!

Scrum won out IMO because it's an awesome word and conjures up subliminal images of brute force pushing obstacles out of the way, bringing success to the team. Execs can relate to it without even knowing what it is.

I actually think scrum won because of its multiple rituals that allowed most of middle managers and project managers to keep their job by recycling it.

I think it won out because the number one thing any kind of project management framework has to provide is ambiguous blame-avoiding fungibility to management. If a framework does not offer that, managers will politically subvert the policy-making process to rule out that framework.

It’s the same reason why demonstrably value-additive, cost-effective prediction markets are rejected by managers despite evidence in their favor: it removes their ability to create ambiguous Dutch books out of different project deliverables and resort to exploiting subjective psychological biases of superiors to win money & status increases. Anything that holds them accountable to the actual measured result status of a deliverable can’t be tolerated.

Scrum is just the by-product of corporate status-seeking evolutionary war games against in-house scientific thinking.

Scrum won out because it's basically feudalism. It's great for those in the upper echelon.

Scrum won because in reality not everyone is a wizz bang developer like your guys so make yourself out to be.

Scrum (and other methods that try in vain to make developers replaceable cogs) won because it enabled bad developers to hide among the good ones, and avoid personal responsibility.

When I hear the word scrum I am always reminded that my school stopped playing Rugby because a kid in the next town got his back broken, in a scrum. So from then on we played football (soccer) in Rugby kit, until eventually all the kids had grown out of it, or left. Similarly, if you sprint and sprint and sprint you will collapse from exhaustion.

I often wonder if they picked these words because they knew nothing at all about sports, of if there is a coded warning in them...

Agreed, it feeds the existing power structures. And it did hit at the right time when over-communication is not only appreciated but everything else is seen as intransparent and potentially suspicious. "You don't even know what your team is doing?" - is the best way for middle-management to pick on each other. So in order to never get into that situation daily standups, slack, jira, basecamp all these "transparency tools" are welcome as they offer defense for this manager.

Does anyone know if there is an anti-movement to that over-communication trend? Something along the lines: "The only way to go fast is to go well", * Plan * Honor the flow * Adjust

yes, I too was involved with XP before it was under umbrella of Agile which got consumed by the Scrum borg.

I feel we kind of lost something where creating code/designs using strong feedback loops was a "first class citizen with royal honors" of the software development process. Scrum kind of makes it a sideline issue, a peon of middle management

Totally. Although this has long been a problem.

My dad started developing in the late 1960s. He did a lot of in-house software, mainly manufacturing and insurance systems. His approach was to go out and see something in use, talk to the people using it, go back to his office, write something, release it, and go talk to the people again. He'd do that loop in as little as a couple of days.

Today we'd call that obviously agile. But both back then and now, higher-ups wanted more "control", by which they meant feelings and appearance of control. Which caused longer feedback loops. Which made things more out of control, which required more process and even longer feedback loops.

I think we have the same struggle today, and I have no idea how to get out of it.

Agile has a problem with introducing Processes and Tools (by definition, as the blog post describes). However, Scrum offers a solution to this problem by creating an Agile space or bubble within those organizations that are still stuck with the traditional methodologies (Waterfall, etc.). So Scrum is a nice way to introduce organizations to Agile concepts but it is certainly not the final goal.

IMHO, the idea is that over time the organization will adopt new Agile practices and not depend on Scrum anymore. In practice, however, many organizations think they are Agile after they introduced Scrum, which is a simple fallacy. After all, becoming Agile is about a culture change and not about the introduction of a new process framework.

I don't find much constructive value in your comment. To me it seems like a politely worded screed about the incompetence of management and organizations.

To drive the point home I could easily transform "[o]n average, the first priority of managers and execs is maintaining the power structures that make them a big deal" into a derogatory comment about engineers, fad chasing, and resume padding by a simple word substitution. It would be neither accurate nor fair to many (perhaps most) engineers: just as your comment isn't really accurate or fair to managers or organizations.

I have been both a manager and an executive. As well as a consultant in many companies both small and large. I'm not saying that those people are necessarily incompetent. I'm saying that, per POSIWID, the purpose of most software organizations is not effectiveness in making software (or in serving users via software).

That's not constructive in the sense of providing ways to solve the problem. But that wasn't my goal. My goal was to support and confirm the article's thesis. I have also written plenty of constructive things on software development, but I think it's absurd to suggest that every HN comment must contain that kind of constructive feedback. (And if that were a reasonable standard, you aren't living up to it.)

> On average, the first priority of managers and execs is maintaining the power structures that make them a big deal. But true Agile processes are about empowering teams to self-organize around serving users.

But that doesn't explain the success of Scrum, because Scrum does away with the power structure of managers, and should empower the teams. "We're doing Scrum now" is the big stick I see scrum masters use to beat back Business managers to keep them from interfering with and micromanaging the team.

At least, that's what they should be doing when management interferes too much. And then address this higher up with whoever decided that we're doing scrum now, and explain to them what that means, if necessary.

There's no excuse to tolerate micromanaging managers in a Scrum process. They don't belong there. Kick them out.

Reply on news.ycombinator.com