How big tech runs tech projects and the curious absence of Scrum
newsletter.pragmaticengineer.com
newsletter.pragmaticengineer.com
I went on a scrum course and the takeaway was basically that feedback is a big deal, and you should try to get some repeatedly and quickly. It's also common sense that you should have tasks written down somewhere, and that some of them are more important than others.
You certainly need to think about how long tasks will take, but there's no reason why you need to do planning poker, that just seems to be one among many ways to think about how long something might take. Tracking velocity is another one of these things that seems replaceable.
If you have a team of people that more or less adheres to a few principles, there's no reason you can't get things done.
When you bring a prescriptive implementation, you can then leverage learnings from many orgs over many many years to handle all the edge cases instead of reinventing the wheel.
If you don't, or do "Scrum but not quite", then when things don't quite work out, you're on your own.
I'm no fan of Scrum, and rather use almost anything else, but that doesn't change that there's significant value in using systems that are mostly figured out and "work". Most of Scrum's bad rep comes from people who think they know better, tweak it to suit their needs, then realize they made a ton of holes in the system and their Frankenscrum doesn't really work. At that point you're better off just doing your own things. Which is what folks do, they abandon Scrum and do agile their way.
I see no reason to believe that Scrum is a tightly tuned machine that must be precisely run to work at all, and any deviation from it makes it fall apart. In fact, if that is the case for Scrum I'd consider it a fatal objection. I don't believe the effectiveness landscape for project management techniques is shaped like that, I don't believe that the effectiveness landscape is a black pit everywhere just around Scrum only for a sudden spike of effectiveness to exist precisely where Scrum is located, and if it were shaped like that, I don't believe in the capability of anyone to have found and proved that spike. Since the only practical algorithm to explore this landscape is gradient descent, anyone carefully and thoughtfully exploring this landscape should never have found Scrum.
Or, to put it in less mathematical language, the idea that Scrum is super effective but if you deviate from it just a bit it all falls apart is complete garbage.
Do real agile, all the time. Scrum's only useful for raiding for ideas, and I'm not particularly impressed with it for that purpose, either. But it's not quite literally bereft of good ideas. I wouldn't give its ideas any particular credence for coming from "Scrum!", through. I don't see a lot of evidence that it is specially deserving of respect. It seems to work for some people, yes, I don't deny that, but the rate at which it works for people is sufficiently low that I don't think it has any sort of special claim.
That's why an algorithm was not used. Effectiveness has not been sufficiently analysed to be reduceable to a line on a graph.
> Do real agile, all the time.
People who call something "real" without defining what "real" is may be well-practised at sounding nice and emphatic in meetings, but without any detail it's hard to see it as anything more.
I remember when Scrum came out. It's a very specific method for a very specific company type that was sold to work with any company. Before scrum, each company kinda figured out their own processes based on their size, team composition, output requirements, and their risk tolerance. Ticket systems existed before scrum, feedback loops existed before scrum, bug tracking existed before scrum, prioritization existed before scrum. Scrum just took all the things that already existed and put them in a very specific and somewhat restrictive process. It was eye candy for the execs who had no idea how their department worked, it was a cash cow for the consultants selling it. It's ok if you don't have anything else or you're dealing with high turnover or something like that, but if you have a mature department already, it's probably more restrictive than not.
Scrum avoids the eighteen-month waterfall death marches that kill a department or a company. Scrum gets you thinking in terms of managing scope, of pacing deliverables, of thinking of a list of prioritized "wants" as you say rather than a dictated set of commitments. Scrum makes you approach development as it has to actually happen in the real world, rather than as a dictated plan that inevitably won't survive contact with adversity.
Of course it's completely possible to still mess that up, with overzealous adherence to process and ceremony and metrics. It's very possible to run Scrum as a set of perpetual two-week waterfalls -- and very possible that that's still a great improvement over two-year waterfalls.
Having said that, I'm glad Scrum worked for you.
The main difference here is that Scrum is not nearly as complicated to implement. Though the issue is the same. There's always someone somewhere who thinks the process doesn't apply to them and the whole thing falls apart.
I did work in one company that had a fairly well implemented Scrum, and it did work pretty well. I've never seen it replicated though, so for all practical purpose, it's exactly as you say: it's not worth trying because it only works when done perfectly, and that almost never happen.
> If you don't, or do "Scrum but not quite", then when things don't quite work out, you're on your own.
So, if you do exactly as Scrum prescribes and do not find success, in what way aren't you "on your own"?
Scrum "by the book" has a lot of material on the edge cases. How you handle almost every case that can happen in a software development team. So you can follow it like a workflow. Something unusual happens, you look in the book, do what it says. It's flaws are fairly well documented and understood, too. That's what I mean by being "on your own" if you don't follow it. At that point you can't just use a book to figure out next steps. You have to actually talk it out and use your brain to figure out what to do, because its unlikely your internal process has as much documentation around every single "paved paths".
It's also just a process. What does it mean not to find success here? If you ship broken software, it's unlikely to be because you didn't move the right ticket in Jira.
If you implemented Scrum "by the book", the likely "failure cases" are more things like people refusing to actually follow the process because they find it to be a waste of time, the overhead is too high, people are sick of looking at the book, etc.
Don't get me wrong: I very much dislike Scrum and you'll never see me push for it in a team. My point was merely that if you take an "On rails" experience like Scrum, and tweak a few little things, you lose on almost everything. The books no longer apply. When the entire point of this particular process is basically the books and documentations, doing "Scrum but not quite" really kills the point.
Yeah this is the nonsense propaganda I'm talking about. "If it didn't work, it's because you needed to do it more", which I think is absolute nonsense.
It's like baking a cake without adding sugar; it's not really a cake anymore.
Sometimes teams don't understand practices and will discard them.
As an example - retrospectives. I've worked on teams where we inspected and adapted our process on demand. We didn't wait till the end of a sprint.
I've managed individuals on teams where they're discarded retrospectives. They've complained to me about certain processes (not scrum ones) and when I've asked, "why didn't you raise this in the retrospective?". "We stopped them, we didn't see any value".
No doubt they weren't getting value out of them, but that doesn't mean they were doing things well and they lost opportunity to improve as a group.
For context, currently, I manage a team of 60ish without Scrum. I see Scrum as a bit dated now.
Dogma.
> It's like baking a cake without adding sugar; it's not really a cake anymore.
Nonsense.
> No doubt they weren't getting value out of them
This piece goes against this piece:
> , but that doesn't mean they were doing things well and they lost opportunity to improve as a group.
Why would you continue to follow a process you don't find value in? If you didn't like a process, why would you feel like bringing that process up in a process-oriented equally-useless meeting? This is the dogmatism. "Well even though the meeting wasn't valuable you still should've tried". To what end?
> For context, currently, I manage a team of 60ish without Scrum. I see Scrum as a bit dated now.
Less interested in size, more interested in throughput/attrition.
> Nonsense
You may think Scrum is dogma, but my statements are factually true by reasonable definitions of the terms.
>Why would you continue to follow a process you don't find value in? If you didn't like a process, why would you feel like bringing that process up in a process-oriented equally-useless meeting? This is the dogmatism. "Well even though the meeting wasn't valuable you still should've tried". To what end?
Strawman.
I'm afraid you'll have to take my word for it that the team where just bad at it. They got better with coaching and even managed to see some value.
The values behind the process are more important. This team just weren't talking to each other about their own performance.
> throughput/attrition
Of staff or customers? Attrition is very low of both. Not sure it's that relevant though. 3 engineers have left in the last 2 years and one of those three is returning. Is that a good thing?
As for throughput, even the slowest of teams ship on a daily basis. Not sure what other context you could mean.
Your statements are dogma. "You must've just done it wrong" is scrum marketing. If it works it's scrum, if it didn't work it must not've been.
> Strawman.
Go on then, explain? Your gripe was that your team, that you admitted didn't find value in retrospectives, didn't bring up process issues in retrospectives. Why would they? They didn't find them valuable. Probably because the things brought up in retrospectives never got addressed. Agile solution? Add more process. Again, this is dogma.
> Of staff or customers? Attrition is very low of both. Not sure it's that relevant though.
If I employ 500 engineers and they're all miserable and constantly leaving, and we're never adding value to our clients, should I be proud that I employ 500 engineers? What does team size matter? Value-creation and employee happiness are the metrics that determine the health of an engineering org. Which I chose to label as "throughput", or how much value is created, and "attrition", a proxy for employee happiness.
Same reason for PMP, it's to turn the procedure into "paint by numbers". Of course it rarely works that easily.
But of course you can't justify the multiple middle-managers and why can't the people at the bottom just "turn the wheel faster" without it so there's a bit of purpose in that.
In the end, it was just so obviously unworkable to everyone it had to be stopped.
However, you are absolutely right, we were not in meetings 100% of the time - it just felt like it. Many people need good swaths of uninterrupted time (perhaps 1.5-2 hours minimum) in order to be productive. Rapid context switching between various meetings and development work can be expensive. It is unfortunate when this time can only be found outside of working hours.
My suggestion is to start with no process and no regular meetings and carefully add process/meetings as required. If you need to talk with one or more people... call them right now (or arrange a meeting with a clear agenda - right now). All overhead should be carefully evaluated in terms of its actual ROI.
This is not just a scrum problem. Any agile process eventually reports to a layer of management that doesn't think in agile terms, and wants their PERT charts or whatever. The impedance mismatch at that boundary is one of the biggest difficulties with implementing agile.
Having a strict framework à la Scrum is very helpful for new developpers or new teams, where they don't yet have their marks and need to get a feel of what agility feels like. Being explained principles is not enough to know how to apply them: Scrum is not a bad starting point to be in the right-ish track without really knowing what you're doing. As you gain experience, it's natural to want to shake off constraints, that's Ha and Ri
It's not like FAANG isn't hiring boatloads of new grads every year. Why don't those people need scrum to get on the right track, but half the rest of the industry seems to?
Anything that is reasonably unpredictable in terms of requirements but needs to be reasonably predictable in terms of feature delivery speed is a good candidate.
We use it because we need to release versions of our software a chunk at a time, and it's the closest we can get to continuous delivery given that constraint.
Maybe people are tired of the nonsense that systems like Scrum bring with them? Even the name is a bit silly. And when you start naming roles and inundating people with rituals the eyerolls really get going. Why add abstraction to common sense?
Or maybe Scrum is really just an attempt to turn bad team members into good ones?
Good team members...
- Provide good estimates
- Cooperate to create clarity around requirements
- Work to divide big problems into small chunks
- Keep good track of their work in a shared format
Bad team members shun all of this and expect someone else to do it.
My current job is the opposite. Some tasks take 15 minutes of discussion (no, they aren't complex tasks) with people debating whether it is worth 5 points or 8. It is just tiring and pointless. And retro - gawd, I hate those. There are all kinds of stupid shit (people using references from music, movies etc, trying to make it "fun" and "hip"). I can't bring other tasks into the sprint, even if I finished all my current tasks, without my manager's permission. And on and on.
I was thinking the other day why this is so painful and awkward. I realized that it all comes down to "metrics" - end of every sprint, my manager has to present it to his bosses, with pretty graphs describing "velocity" and other buzzwords. So he has to do all kinds of jugglery to appear competent to his bosses and not alienate the team at the same time.
All of this could be avoided by treating the team as adults, instead of trying to "processify" or quantify everything.
However hard we try, human beings can't be slotted neatly into buckets nor can they be precisely quantified. However hard we try, estimating software projects will never be an exact science.
A cause of this problem is when people are worried about what happens when they say "erm, I don't know how to do this cleanly" and so they want the "how" to be defined far far in advance. Solving this requires a mix of
A. Identifying those activities that will genuinely make people's lives easier if they have a process and designing those processes to be meaningful and low-burden. Those activities will vary based on team members individual peeves and affinities.
B. Creating clear lines of communication for how and whom to ask for help. This often means more clear naming of responsibilities.
C. Ensuring there is enough slack time for people to be able to help each other out.
D. Creating trust in the team that asking for help won't lead people to question if you are fit to do your job.
> quantify everything
Many times, this is the https://en.wikipedia.org/wiki/McNamara_fallacy
It's also a way for our team to get together and talk BS, have a little bit of fun etc.
I have teams that respond well to it. People like me, who work in metaphors and abstractions, respond well to it.
That sounds like fundamentalist Scrum.
I'm baffled by this behavior. If you know it is the next most important thing, why do you care? What would it change in your behavior if it is a 5 vs and 8? If you are looking at a task that you would choose not to do if it were an 8 but choose to do if it were a 5, then you probably look for something that is more important to work on.
I think you are right that much of that is driven by managers looking for metrics, but unless they understand what the metrics mean it is pointless.
Again, this is all within reason, and if it takes 15 minutes on every ticket, the team should probably work on their communication skills.
Or that the assignment of points is arbitrary and imprecise and that different people have different ways of making up numbers.
At least that is the idea.
Second, the numbers _are_ arbitrary. Completely, 100% arbitrary. It's a _relative_ difficulty scale. Say you've got 3 tasks - A, B, and C. A and B are approximately as hard as each other, they're 1 story point. C is more difficult than either one - it gets 2 points. That's it. Story points are not, and should not be used as, a unit of measurement. The biggest utility is to identify big, scary tasks with lots of unknown factors.
The fact that they are _numbers_ is what tricks so many teams/PMs/management/etc into thinking that story points are more meaningful than they were ever supposed to be. Incidentally, this is also why some planning poker teams use t-shirt sizing (S,M,L,XL,XXL, etc). No numbers means people are less tempted to punch them into a spreadsheet while deluding themselves into believing that showing numbers going down is the same thing as "showing progress".
As for its usefulness: You of course don't have to necessarily poker to get these discrepancies, but it's a pretty effective way of going about it, I'd say.
Instead of worrying about 5 vs 8 though, the team should be discussing relative difficulty: is story B easier/more difficult/complex than story A? And then ranking stories based on the relative difficulty of each other.
Story points can then be derived based on that ranking, if the team chooses. They're useful for being applied to velocity calculation, and also helpful when picking up a story to work on (maybe a bad idea to start an 8 pointer on a Friday).
(I have a SM cert, working as a TPM. I applying Agile principles to teams, but modified to how they want to work and be most effective. No militant Scrum here.)
The conversation is meaningful when it's about 3 vs 8 because exactly as you say, there probably is some hidden complexity not everyone knows about, or sometimes some work has already been done or there's a framework feature that makes it easier than it seems.
But 2 vs 3 or 5 vs 8... this is explicitly designed to be an imprecise number, so let's not waste our time debating which one to choose.
Quick discussion of the differences is useful, but 15 minutes is ridiculous. Just take the higher value and move on. Eventually you’ll baseline at slightly more points per sprint on average, but they are imaginary numbers anyway and not really comparable across teams.
That's why there's even a velocity expressed in points, like the tasks - it's the PM's budget to work with. So 5 points or 8 doesn't change the work you have to do as an engineer, but it changes the number of things a PM will put on the team's plate in the next sprint, and the points force them to make trade-offs, otherwise they'll insist that the bug, the small feature and the large feature are all high priority and have to be delivered in the next sprint.
I'm not justifying the 15 minute time investment for sizing, that is clearly way too much time, I'm only highlighting that in scrum, as an engineer, you don't necessarily know or decide what the next most important thing to work on is. That's the PM's job.
Well anyway, managers in my current company are dumber than second coat of paint and I inflate points.
It's not a coincidence the consultancy sector uses scrum. They get paid for their output, usually by the hour, and scrum measures output.
If a consultant implements a load of useless features that make a product worse, but do so very quickly, then that's a great success for them. It's not fundamentally different from rewarding developers for the number of lines of code they write.
1. Sometimes your team needs to give an estimate of when it will deliver a feature. The idea of pointing in Scrum is that you establish a velocity, which lets you get a feel for how long it's going to take to deliver a given piece of work.
2. Sometimes your team needs to make trade-offs, and often the right lens here is ROI. You can't estimate ROI if you don't estimate the I. (Although I believe agile tends to somewhat overrate the usefulness of the ROI lens.)
3. In the absence of 1 or 2, the act of pointing can help to catch the case where you miss a piece of complexity; if someone thinks it's 8 and everyone else is a 5, then maybe they are aware of some gnarly code that everyone else has forgotten?
I'm not arguing in favor of discussing each story to death, just making a more general justification for the practice. If you never need to do 1 or 2, then maybe pointing isn't going to be useful for your team at this time -- and that's OK too.
I'm sure in some cases this is just driven by managers needing metrics to measure their teams by, but the above is an attempt to suggest some reasons that the teams themselves might find the practice useful.
I'm just giving some examples of ways that some Scrum teams do get value from the practice.
I think we just were part of a software factory that is not necessarily bad, it's just a different approach to work. It's much more convenient for a manager to just apply "standard" principles (scrum, 2 weeks, story points, velocity, etc. like a mantra) rather than innovating at this risk of being wrong, difficult, etc.
Too much bureaucracy is not good, maybe that was the purpose of this article. The more you have it, the more difficult it becomes to get out of it, to innovate, etc.
"Metrics" though drive when something might be delivered which triggers marketing spend, hiring requirements and so on. It says "we're working on this now and that next" which drives all kinds of internal storytelling about priorities and cascades down (or up?) into quarterly calls.
How do you do it otherwise without having ipod summers?
I work in an environment that had no project management tools or methods being used other than emails and I've finally gotten buy in on Kanban and I want to make sure I guide my team properly.
https://www.amazon.com/gp/product/B00A32O00Q/ref=dbs_a_def_r...
Although I think he ended up with something more Scrum-like later at Spotify.
He also has lots of good videos about broader team organization (Tribes etc.) that I found useful for conveying some of these ideas to my non-technical peers.
Not only that, but if anything else comes up during the "sprint", you're expected to address it while still finishing everything that you (involuntarily) "committed" to during sprint planning.
Thanks, I hate it.
Now you do need don't to anything illegal policies in place, and you might have a [unwritten] moral code of what you can't do. However the purpose is to game the metrics. For all the discussion that follows I'm going to assume we are within these limits.
If the boss wants profit margin as a metric, then you need to figure out how to do things cheaper, and how to get more sales, or higher value sales or some such, and sometimes get something off the books. Note that this last is why smart investors always ask about the bottom line - because the rules of what you can hide from the bottom line are strict enough that the metrics cannot be gamed to your disadvantage. That said off the books is sometimes a useful thing to have - sometimes you know you need to do something hard and so you need to leave an approved way to hide specific things in plain sight.
The problem is most developer metrics are things where gaming the metric isn't good for the company.
This is a lesson more managers need to learn. You don't give engineers a metric to game, they'll run circles around your silly numbers.
My favourite anecdote about this:
I was a consultant for a multinational corp. The Natives had their bonus targets tied to "amount of open bugs at $date".
What did they do? At $date-1 the natives assigned all their open issues to me, a poor lonely consultant - who was outside of the bonus structure.
...and at $date+1 they grabbed their issues back, bonus objectives met :D
That was my last gig. The whole organization eventually just collapsed under the increasing load of being "Agile".
If you're not personally responsible for deadlines on the project, it could seem pointless. But the difference might mean pushing the commitment for a feature out two weeks, which in a commercial project could be a big deal. Planning is hard: you've got to try to estimate something with incomplete information, and then reconcile differing opinions. But it's definitely not pointless.
Velocity is probably one of the most misunderstood aspects of scrum. It's a key metric for long-term planning, but it's not intended to be manageable. It's also unique to the team, and not intended to be compared between teams. Many managers are not used to a metric that they don't manage, and that causes a lot (maybe most) of the bad experiences people have with poorly practiced scrum.
This is a thing that's pretty key to Scrum: velocity can't be used to compare teams. It is easy to misuse though. I would start every meeting I were in with "these points measures cannot be used to measure performance between teams."
I mean, just the difference in an iteration with a couple of people on PTO, or a holiday or two intruding into it, will throw the numbers off. Now factor in all of the other fuzziness that's inherent in the process... yeah, no. Don't bother trying to calculate or track "velocity". You'd probably get better results from Tarot cards or animal entrails.
Velocity in physical terms is an instantaneous measure. It's the same in Agile, you've done X things in Y time. But that ignores the context of those X tasks and the context of Y time.
The X+1 task might turn out more difficult or have some other challenge that makes it take longer. Time keeps passing so at time Y+2 the velocity measurement is lower than before. Now all the PMs and other team members jump in demanding answers which now makes the task take even longer.
As you mention, some span of time can see team members out or unavailable. So the velocity is "low" but rarely is that context captured or bubbled up the management chain. So your team has "low velocity" which ends up generating worthless meetings, bad reviews, or all sorts of problems.
It's not part of Agile. It's specific to Scrum.
I've found that most Product Owners really care about predictability and less about actual estimates. They care that if you say you're gonna finish something by the end of the sprint, much more often than not, you will. Estimates and velocity and such things are just means to the end of trying to figure out which items are reasonable to take into a sprint together.
If your Product person knows that they can throw 10 "random" (but currently most wanted by stakeholders) work items into a "sprint" and 9 times out of 10 you will actually finish all of those, they're probably gonna be very very happy as they won't have to explain delays over and over again on their end.
I don't see how this can be the case. If each ticket is estimated to be about the same size, and it sounds as though they should be, then if you're doing say 5 tickets a week, 10 a sprint, and they're all about the same size then each ticket you could say is 5 points, and you're doing 50 points a sprint.
That's probably pointless (hah) if they are all the same size; ticket sizing is more for if you have tasks of different sizes and you're trying to roughly combine them into something you can predictably deliver.
But if you're at a stage when your delivery is already this predictable, you may not need to use story points particularly. If it's mandated for some silly reason then you could use the above method to bat it away quickly!
You've discovered the secret of Agile in the corporate workplace: that the key takeaways, as far as the enterprise is concerned, are not finding better ways to develop software and better customer-developer relations. It's all about trackability, measurement, and metrics, because everything is. Trackability and measurement enable key decision makers to make accurate budgeting and execution plans and there is literally nothing else for key decision makers. Therefore, it makes less of a difference what you output than that it was properly planned for, tracked, and measured and that it hits organizational KPIs. That's why nobody cares that Scrum is a giant productivity suck. Code could be written faster, but that's not writing it properly.
Agile in the enterprise is a game of Mornington Crescent. The goal is not to foster the things advocated in the Agile Manifesto. It's to make it look like the company is fostering those things, while actually promoting the same Taylorist values corporations have always loved (and workers hated).
That's probably the best description of "agile in the enterprise" I've ever read.
"Interspersed with the turns is h̶u̶m̶o̶r̶o̶u̶s̶ 𝗯𝗼𝗿𝗶𝗻𝗴 discussion among the panelists and host regarding the rules and legality of each move, as well as the strategy the panelists are using. The actual aim of the g̶a̶m̶e̶ 𝗰𝗲𝗿𝗲𝗺𝗼𝗻𝘆 is to e̶n̶t̶e̶r̶t̶a̶i̶n̶ 𝘄𝗮𝘀𝘁𝗲 𝘁𝗶𝗺𝗲 𝗮𝗻𝗱 𝗺𝗶𝗰𝗿𝗼𝗺𝗮𝗻𝗮𝗴𝗲 the other participants and listeners with a̶m̶u̶s̶i̶n̶g̶ 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝘃𝗲 discussion of the fictional rules and strategies."
This is exactly how I've seen it run with success at several different companies (including how I run it).
In my mind, the most important value of Scrum is having a cadence to give people reasonable timeframes to think in AND ensure that a variety of different conversations happen frequently enough (discovery, planning, prioritization, execution).
----
> And retro - gawd, I hate those.
It sounds like you hate these because you aren't actually empowered to make meaningful change within your team.
We've found retros are one of our most important rituals. They help us identify risks, process, and team issues; often while they're still small. We quickly work those in.
We also found retrospectives to be pretty useful and each team was encouraged to find what worked for them. The result was that we had 5 teams that had some agile/scrum process, but created mechanics that worked well for them.
This is basically how we use it. Typically, if we have a multi-sprint effort, we'll guarantee delivery in the last or second to last sprint. Anything before that and we're having conversations around scope and trade-offs we need to make to hit targets.
I've had several engineers tell me their 1.5x to 2.0x as effective as their prior roles while having better WLB.
Then they decided they needed the metrics on everything, so they switched to JIRA, started doing retros, setting strict points on tasks(reprimanded in retros if you messed up), using burndown charts to reprimand even more, and giving the product manager the power to dictate what I work on and in what order.
It went from being a great company to work at, to a company I ran away from. I have half a mind to send this thread over to them.
One of the worst parts of scrum in my opinion. Even if it's a "quick" 15 minutes (which in my experience it rarely ever is), the forced context switch ends up wasting more than 15 minutes of productive time every day.
I have do it as a consultant. I have taken a project on life support that they were gonna cancel, regardless of our performance, and resurrected it so hard they actually decided to keep it. We changed their minds
The most effective tool was retros. I dislike agile in all other aspects, and do not use story points. Retros are the best bit! It's the core part of the learning loop.
I learnt retros from Google. Big tech does retros. Its honestly magic when done well.
It might be a force multiplier in other types of companies, but it probably isn't in FAANG style companies. Similar to how FAANG hiring can make sense to them but still be bad for typical companies.
lol, as if their top-down hiring practices are at all driven by concerns for company efficiency.
The author obtained first party data backing up what he wrote. You present nothing.
> For complex projects that span several teams across different offices and time zones, leading such a project is a full-time job for an engineer. Big Tech pulls in Technical Project Managers (TPMs) to manage these complex, and often strategic projects, taking the load off engineers.
> Dedicated Program Managers or Project Managers still exist within Big Tech. They are typically tied to external-facing commitments and customers – such as a Program Manager for the Apple partnership – or to long-running initiatives, like a compliance program. Similar to how TPMs are not allocated to a single engineering team, these Program Managers or Product Managers also don’t tend to have an engineering team, but they work across many teams instead.
Companies that implement agile to the letter typically treat their developers like consultants. You're treated as a mindless drone that is supposed to work on tasks that were predefined for you. You might as well be outsourced.
I found that kanban-style works better imo. A prioritized backlog where you pick the most important task to get done makes for a more relaxed and friendly environment, compared to an environment with a rigid deadline every two weeks that everyone sprints towards. As long as there's progressed made towards the most important items, everybody is typically happy.
It treats developers like consultants because it's really designed for agencies working on one-off short-burst projects ( this is the only setting I've seen it have a positive effect )
by nature it just chews people up if it becomes a day-to-day practice, the whole process revolves around the assumption that there's lack of trust and team cohesion
Where in Scrum is this defined?
This is because nobody can be assed to figure out who needs to communicate with who, on what.
The Sprint gives a defined time period during which the team is allowed to work uninterrupted by these kinds of requests, with the updates, feedback, change requests etc. taking place at the Sprint boundaries.
They take the waterfall approach, what do we need to get done by X to achieve goal Y. Then apply tasks/actions to get it done to people who are at a meeting. Following this step they shift over to scrum sprints, routines and measures to give management comfort on whether its track or not.
The alternate might be, what is the highest value items that we can achieve with this group of people.
Generally speaking, if the thing that is meant to help you get to value becomes more of a focus then you're in a spot of bother
do you know why i do this? cause i'm lazy. i can't be bothered to remember things or talk to people again so i do the one thing people often forget we can do: write it down. know how many times i've annoyed my lead by having to re-ask questions for a typical project? 0. cause i ask it once and write it down. it literally takes no time at all. it saves on having to do more meetings down the road because i have it all documented.
Scrum is anti agile, it's basically processes over people.
Honestly I find it hilarious how agile people try to bring scrum into companies just so they can get a tiny bit of extra power.
In my experience “traditional companies” will often have a bunch of people in cushy “gatekeeping“ jobs whose main function is basically forwarding emails back and forth between devs and the business. If you try to get direct access to the business usually the business is quite happy but the gatekeepers get very upset.
All I can think is "Wow, I bet I could really focus on something difficult in that cubicle!"
Like asking for a 6months dev work to help them save 1 hour annually on an annoying task they had to do.
If I were designing an organization with such a division of labor, those wouldn't be different job descriptions but different activities within a single job description; maybe I'd spend 22 days a month doing product-engineer stuff like adding features, and 3 days a month doing support-engineer stuff like visiting customer sites, diagnosing customer problems, and answering calls, while maybe hatchnyc would prefer to spend 22 days a month doing support-engineer stuff and 3 days a month doing product-engineer stuff. The reason is that, doing either activity, you gain an enormous amount of knowledge that's crucial to the other activity, but very difficult to even put into words, much less into a knowledge base.
Bill Gates found it useful to answer tech support calls as late as 01989: https://www.entrepreneur.com/article/289857
This is good, but it's easier when you're a very extraordinary human, and your own boss!
I see this as a complaint a lot, but... in a private company, they're the ones writing the checks. So if they want to spend hundreds of thousands of dollars to save an hour, that's their prerogative. It's certainly our obligation to point out the cost (including ongoing maintenance) but again in a private company you don't really have the option of saying "no" except with your feet.
I do however agree with leaving any organization that does not allow you to express disagreement.
After that, I really really don't care anymore, that's someone else's problem. I've played that soapbox and it's a waste of my time and energy. I make sure responsibility is passed back, documented and proceed. If you want to throw hundreds of thousands or millions at me and whomever to some wasteful request, go for it. You have the option, you've been warned, and I've given you a valid excuse to proceed with high risk, high uncertainty of ROI option with everyone important involved.
I do a lot of contractual/consulting work so even warning can mean early termination of work forward, so I'm taking a risk even informing you (some people actually listen and work stops early during consulting time and development follow up never occurs, these are the smart people and they're bad for my livelihood)--I could just do whatever you ask and get paid. If I were salaried, there's even less risk of me losing income forward by informing so I say in those cases just do your professional due diligence go let people know it's a bad idea without creating political land mines for anyone, then allow them to step on land mines at their own discretion.
Now imagine a government entity where the users and the dev are both paid by the gov.
It could be something that's tricky. While the optimal case is 1h of work, if something goes wrong then its doing forensic auditing on the books in 3 months and spending lots of time there.
It could be something that needs to be repeatable. Yes, it's 1h, but before this was done Alice did it and before that Bob did it. They did it slightly differently and that cause problems. Having a computer program do it provides change management to the process of doing that task.
It could be something that needs auditing. Tying into the previous two, having a "this is how it's done and all the calculations that go into it along with a test suite" makes it easier to be assured that it is working correctly.
There are lots of things outside of the purview of a developer working on doing whatever task needs to be done to solve a problem. The value of the problem being solved is the issue of the manager or the customer.
As long as they're being otherwise reasonable, I don't have a problem doing something that doesn't seem terribly "important" to me either. However, in my career I've had many frustrating instances where they were demanding something in a timeframe that I couldn't deliver it without ensuring that there wouldn't be any unintended side effects - and then blaming me for the side effects when they came up. I only push back when they don't realize how complex what they're asking for is. Of course, then they play the "tell me how long it's going to take so I can argue with you until you agree that it will take as long as I originally asked for" game.
Far too few developers understand: just because the people writing the checks ask you to do it and you implemented it exact as they ordered you to doesn't mean the people writing the checks will take responsibility for the resulting bad situations that can lead to them not writing you checks anymore. If anything, you'll be used as a scapegoat to prevent them from not getting any more checks written.
In general I think there is a huge advantage if developers know first hand how their product is being used and not through gatekeepers.
It sounds like what you're describing is the lack of such prioritization. It isn't necessary to keep the developers ignorant of new user "demands" in order to prioritize them; they just need to be empowered to follow the priorities the team has chosen instead of accommodating every demand.
Sometimes it's easier to accommodate a demand than to write up a card and estimate it, though. Sometimes a bug fix or layout tweak is obvious enough that doing it takes only a minute or two; then, it's just a question of process overhead whether the actual cost of doing it is five minutes or five hours: potentially multiple forwarded email round trips, a fresh checkout, pair programming or other code review, compilation time, test suites, commit comments, merge requests.
Usually, in my experience, if you have extreme overhead, most of those changes never get requested, because they don't get prioritized the way they would with the order-of-magnitude lower cost a lightweight process can provide. The result is that they never get done, so the product is full of easily observable minor problems, which is experienced as shoddiness.
I believe this is why the author noticed that certain types of companies had engineers say the product owner was one of the reasons they're more satisfied. From when I started at my role, my team went about 2 years without one, and then within months of having one, we have been much more productive and have managed to create much more immediate and future value for the company.
Developers tend to be viewed as something that is not "core" to the business, and generally only necessary on a project-by-project basis. So, projects are initiated, resource needs identified, and, since nobody wants to hire a bunch of full-time staff, most likely a consulting firm or off-the-shelf product is selected and configured.
The downside is now you have a thing that needs to be maintained, and probably it will be maintained by a limited pool of "in house experts", who are also responsible for just about everything else.
The net result is that you've got this centralized resource that is under provisioned and, well dear reader, what does your training as a systems engineer tells you will happen with a centralized resource that is under provisioned? Contention? Rising latencies and queue lengths? Disastrous to recover from failures?
You betcha.
From what I could tell no-one was firing engineers because it was too expensive and we didn't really care since we could get more work within a few hours, and the "bosses" didn't like the lack of power they had in that dynamic.
And Anatomy of a runaway IT project https://brucefwebster.com/2008/06/16/anatomy-of-a-runaway-it...
Those are fun reads too.
It was probably still unprofessional, but probably not as bad as I made it sound...my bad!
“But I’ve got people skills, dammit!”—Tom Smykowski
In traditional (not engineer-first) companies, engineers are subordinates of PMs or business directly without their own agenda, so they just accept the fate and become a feature factory for the company.
Scrum is a parody and pretty much anti-agile.
Traditional organizations are driven by the cost-accounting mindset that creates gate-keepers and makes it impossible to share a common purpose, let alone collaborate across teams, units or vertically.
I find it interesting that they author references Skype circa 2012. It sounds like classic "uppercase A" agile: they reframe success as shipping and hitting other invented agile milestones, while masking shipping lousy and incomplete product that is not succeeding in the market. That was right around the time Skype started being bad. It may succeed on some metrics because MSFT started bundling it, but anyone that actively used it at that time should know what I mean.
The idea of an "agile enterprise" is intrinsically absurd. Teams are agile, not companies, and teams and products should be loosely coupled. That's why the big tech companies in this link have converged to similar answers to this problem (they are not "agile enterprises" but give teams some latitude to solve their own problems, and rather focus on results). Kanban is better in most ways for most teams.
Also, Tuckman is a silly model that is only popular because it rhymes.
PS: Strawberry-Jam-O-Meter and a Wrong-Order-O-Meter and their descriptions could hardly be more cringe inducing. Skype should be a part of the curriculum looking at less than ideal acquisitions and post-acquisition execution. https://www.wired.co.uk/article/skype-coronavirus-pandemic has is a reasonable overview if you're not familiar. They lost the consumer market and failed at enterprise (see Teams) and Teams in turn essentially failed in the general market- where most places that were free to do so went with Slack.
I’m really not sure what you are basing this on. Agile, as I’m familiar with it is framed in terms of lean management which is a well established approach to running business, exemplified by Toyota. Now you can debate the suitabilities of this and the effectiveness til the cows come home, just as you can with agile and scrum etc but it isn’t accurate to say that there aren’t a lot of businesses that aspire to the lean approach.
I think one of the keys to this was the buyin everyone had. It wasn't totally consistent at the beginning, but over time everyone got on board, we did trainings, actually incorporated feedback consistently.
I hear SAFe hate all the time, and when I ask how things went, inevitably it turns out they never actually did SAFe. Somebody made them email a 10 week plan and used the acronym and that was it. If you're gonna do SAFe, do SAFe.
So the commander says "Yes General, we'll do Drills™ to make it happen."
And then the commander writes down "Do drills" on a TODO list, gives presentations to the soldiers about the importance of drills, and maybe even hires a certified Drill Sergeant.
But the commander and soldiers never actually do drills. They clean their rifles, practice shooting, and do other soldiery things, but no drilling.
3 months later, surprise surprise, the soldiers walk all over themselves at the big parade, the general is angry, and the commander is confused why simply talking about drilling didn't work.
And no, "adapting the process for your needs" doesn't work. The problem is having the process mandating meetings and timelines in the first place. If you just do everything freeform as these people wants then it isn't Scrum.
Now, it might be that some people selling Scrum have Very Specific Views about some of these things, but that's not Scrum either.
No it doesn't. Refuting claims about Scrum that are definitionally wrong stops us actually having useful conversations about whether actual Scrum is good or not.
> Individuals and interactions over processes and tools
> Working software over comprehensive documentation
> Customer collaboration over contract negotiation
> Responding to change over following a plan
How many big companies do you know that are organizationally capable of valuing individuals over processes and tools, or responding to change over following a plan? Almost all big companies have the opposite value system on those two points, imposed from the top down by the CEO (with the exception that they do value individuals if those individuals are senior management). Many big companies also have incentive structures that strongly countermand the other points, too. Comprehensive documentation is crucial ammunition for redirecting the blame for bad decisions, for example. (Cf. https://news.ycombinator.com/item?id=28674388: "we not only need very thorough clarity on what feature development will yield the greatest returns, but also deniability that we had good reason for doing what we were doing if it turns south.")
A perfect example of using agile rhetoric without agile practices comes from today's post about what tech managers do—"standup" meetings where people don't stand up! https://news.ycombinator.com/item?id=28677250 This results in the meetings lasting half an hour, doubling or tripling their cost per participant, even before you grapple with the increased number of participants that likely results!
I have my doubts about how effective Scrum could be even if practiced perfectly, but Jeff Sutherland, Mike Beedle, and Ken Schwaber are among the authors of the original Agile Manifesto (not just the signatories!), so at least they endorse those four core values and tried to make Scrum reflect them. So I think it's justifiable to say that if you value processes and tools over individuals and interactions, or following a plan over responding to change, you're not really doing Scrum, or any other agile process.
Not every criticism. Just every criticism that is actually people just not doing it. "Standups become status update meetings, therefore Scrum is stupid" is an example.
Pretty much every process will work successfully if everybody participates in good faith. I don’t think SAFe has special properties that people honestly work together. For example in my company it would just create a new bureaucratic nightmare where we would hire even more managers, project managers and consultants while senior leadership still would never talk to the lower ranks.
Is something actually valuable if it is A.) Inflexible and B.) Rarely goes well?
We do PI planning from SAFe. We would get the last sprint of quarter for slack, improvements, they said.
Instead, we use it to clean up the mess created from waterfall scrum in the previous sprints.
My biggest issue with the whole thing, however, boiled down to the set of incentives that centered around individual performance. You see, the teams weren't _teams_ per se, but a collection of individuals working on their own projects, sometimes related to those of other teammates, sometimes not. A team-oriented process like Scrum or Kanban cannot survive in an environment where every person is optimizing every decision for their advancement/bonus/whatever. There's some exaggeration here, but I definitely saw a lot of this at FB. Having come from a company with high-functioning Scrum and Kanban teams that worked together to achieve a common goal, I'd choose that any day, JIRA or not.
This is obviously a single example and many of the issues were Facebook-specific. But I bet there are other, not so positive, sides to the "no process" story at the rest of the companies.
Note I'm not claiming that the Agile processes are by definition good. I've seen plenty of bad implementations as well. At the end of the day, every group of humans is different and may require a different process (or no process at all) to maximize their success. What I'm suspicious of is the "every team picks their own process" claim, having seen company-set norms exerting enough force to make deviations from the common pattern rather painful and counter-cultural.
Now I use Jira and hate every second of it.
While I was there management was trying to resolve this situation and transmit message that long term projects are impactfull and important (and strategic to company), yet they couldn't answer question of how impact should be calculated every 6 months.
Maybe by now they came up with some formula or something
That said, large cross-org projects certainly do exist. To answer your specific question, there would normally be some sort of a lead on the whole project, tech lead or PM lead or, often, a pair of those. They would meet regularly with leads of various sub-areas, organized similarly, and ensure things line up properly. Now repeat recursively until you get down to team level, where there might be a tech lead, or perhaps just a senior engineer working with a few non-seniors on their portion of the larger initiative. At this point this would get broken down into individual projects that each person owns and is accountable for. Tech lead may or may not contribute code to this project, they (and their team at large) might be involved in a bunch of parallel initiatives that often don't have much to do with each other, other than the product scope the team owns. Things like daily stand-ups don't work when everybody has their own work stream (and often multiple streams at that).
Hope this clarifies. The setup works for certain types of personalities, but I found it to be a very individualistic culture that I didn't particularly enjoy.
Scrum/Agile tells you to split up a project into well defined tasks to be done simultaneously by multiple people.
That's counterproductive if, come perf, you need to show that you completed the project, and distinguish your contribution from others'.
Both in enterprise space or in consumer space, smaller companies are on a much more rushed timeline, for varied reasons including but not limited to financial situation of the company.
Bigger companies can afford to build more slowly. This affects how the company plans and executes as well as rewards performance.
In a smaller company, when a complex thing has to be shipped on a shorter time scale, a lot of divide and conquer and quick coordination across large number of contributors is required. This is a necessity.
In a larger company, deep complex things are built by very small teams or just individuals over a relatively prolonged time. Individual engineers prefer to keep chiseling away at a particular problem until they can showcase a significant impact with high difficulty/complexity of the problem/solution. That's the consequence of individualistic performance measurement culture.
Both cultures have pros/cons and there are rotten extremes in both.
Software is typically quite hard and mostly unknown at the beginning of a project. Most of the hard work is figuring out the things that weren't known. It's a learning/discovery experience. It's essentially impossible to predict how long that activity will take. The further out in time you consider, the less likely you'll have much of any clue about the subtasks to be done. And of course in the meantime the participants likely aren't even working close to 50% of their time on the new project -- instead they're fixing the bugs and technical debt in the last project.
And yet, the "stakeholders" simply wouldn't accept any narrative like the real one (we kind of have an idea how to do this but we're going to figure it out as we go and it'll be difficult and unpredictable). They'd go hire another bunch of folks who are willing to salute the flag. So instead we pretend that what we're doing is predicable and plannable, like building a bridge that's a bit longer or shorter than the last 10 bridges that we built.
The stakeholders probably have a pretty good idea that the developers are making it up as they go. But everyone behaves as if the reactor isn't on fire.
Frankly I have had a much better experience on teams that tracked work in spreadsheets than teams that tracked work in JIRA. Like it's not even close.
In a past life you would be told Agile meant “self organizing teams”. But in practice that was only allowed in a narrow definition of change under the prescribed process being foisted on teams from above.
IMO while you need some consistency to get alignment on goals at a high level and coarse quarter-level goals, at the team level you can more or less let the team decide and then judge them on their effectiveness.
Odd how so many companies want to do “what [successful tech co] does”. Yet those companies innovate their own processes.
We (Bugsnag) are a relatively small engineering team, so have the advantage of low comms overhead, and we do have an overarching approach, but each of the teams works a bit differently. Even from project to project I'll adjust what makes sense based on complexity/risk/size.
My conjecture (please confirm or deny) is that these self-organizing teams are a result of and not a cause of big successful companies. You are probably iterating on a very well-understood and successful business model with a huge cushion for mistakes because you have some much revenue.
By contrast, I have mostly worked in consulting and non-tech orgs. If we don't show that we're spending their budget on high value features, we get layoffs. Hence we not only need very thorough clarity on what feature development will yield the greatest returns, but also deniability that we had good reason for doing what we were doing if it turns south.
If you're in a situation where you need to show you're "spending their budget on high value features" that's a low-trust feature factory being treated as a cost centre by a remote client, not a value-producing unit setting its own terms.
How could you fix that? Would it be sufficient to give the developers more contact with the customers, or would you need to hire different developers, for example domain experts who also knew how to program?
I ask because, when you're writing software, you're constantly making tradeoffs between different aspects of quality: throughput vs. latency, flexibility vs. performance, learnability vs. routine usability, information density vs. skimmability, recoverability from errors vs. security, predictability vs. everything else. The more you know what your customers want, the better you can make those tradeoffs.
Ultimately "how to execute" is the automatable part of the job.
A corollary is these self organizing teams add up to self organizing business units. The teams know what to build because theyve at read & internalized what matters to the business. And senior members should be involved with the 6-12 month planning cycles that happen across that business unit. So their day to day execution is informed by, and happens in the context of, the larger business. Think of something closer to hierarchal federation than directed work silos.
WRT to “lighting money on fire” there are teams that look like that. But thats generally a speculative investment with a medium term (~2-3yr) goal theyre working towards. This is business, so its not free or infinite, and in a sense their efforts are competing with the alternative efforts that could be more profitable.
The key here is that you're using the third person: "their goals", "their money". "Business" is a remote third party, distinct from your teams. It might work, and a lot of people do it that way, it might be the only way your business model can work, but it's a long way from optimal.
If you're not in a situation where you can envisage "trust the team" being something you could live with, then yes, what you're doing probably looks rational to you. "I can't spend my dev hours..." makes me think you're in a fundamentally McKinsey/Taylorist environment, low-trust by design. That's not uncommon, but it doesn't make other approaches wrong either.
In terms of content, teams of experienced developers have always worked in the spirit of Agile (of course there are exceptions). This informal understanding was and is superior to a formal horizontal Scrum implementation. For example, it preserves seniority and true accountability - two things that Scrum pretty systematically destroys in my experience. Initially, I was hoping for additional solutions to problems in the vertical direction - management, customer relations, etc.... But that never materialized, at least in my environment. And since I've been in the industry for more than 20 years, that's not too little.
These days, mandatory Scrum is a contra-indicator for any project that crosses my path as a freelancer.
Yes! Although I don't agree that Agile is a good idea that scrum ruined.
Agile in 2021 means something different than Agile in 2002. Project Management was a different consideration 20 years ago.
Which of these do you disagree with? https://agilemanifesto.org/principles.html
I disagree with 2. Now, perhaps you disagree with 5. These were written in 2001 for problems in 2001. Now, some of these problems have been solved. Some of them are taken for granted today. However, we take them for granted in large part because a group of people got together and said, "we can do better."
I personally think it's time for another group (not the old agile luminaries but people with new ideas) to take a look at today's problems and take a fresh stab at it. Maybe it looks like "Plan-Build-Ship." But maybe it looks different.
But first, I think we need a clarifying question: What in the principles or manifesto do you disagree with?
Edit:
> How long have you been in the industry?
32 years.
Rather than moral principles, I prefer to base all discourse on scientific principles.
And those moral principles, deemed to be good, can be taken as assumptions. Assumptions can be questioned. Improved. Even replaced.
A manifesto and the holy writings like the Scrum Guide are unquestionable and give rise to dogma, tribalism, and all sorts of psychotic reality distortion perspectives.
If you're tired of Scrum, Manifestos and Holy Guides, there are better alternatives which foster much more humane, sustainable and collaborative working environments. I invite you to check out my TameFlow Approach.
Ask anything if you want to know more.
Hmm. Scrum seems like a relatively good means of aligning two-parties with low-context.
I understand where you're coming from, a lot of freelance work these days is just code grinding for a short or maybe longer period of time.
My projects are not of this type. Most of the time they are a mixture of specialized mathematics, software design and implementation.
According to the article, what the big tech companies have in common is engineer-led products. It's not the besuited MBA's making product decisions, it's us, the engineers.
And the big tech companies are also united in having evil products. With the possible exception of Shopify and Datadog, every company on that list of "big tech" is doing things that actively harm society.
Cue the "are we the baddies?" meme [0]. If engineers are left to build products by ourselves, do we build things because we can and not because we should?
I don't think following incentives makes someone bad though, it's only human nature, and of course there are plenty of people queued up to replace you. The system that creates those incentives might be bad, but it can be impossible to untangle where the incentives stem from.
Creating a sports car, or paying someone else to do so, is a lot more expensive than just stealing one, if you can get away with it. Robbing a bank is cheaper than building a bank. If you can do this kind of thing at scale, well, to the extent that you only have to spend money on capturing value instead of both creating it and capturing it, you'll be vastly more profitable than the "suckers" you're capturing it from.
Same thing is true of Fecebutt: creating the mountains of data their users go to Fecebutt for (restaurant recommendations, family photos, thoughtful essays, sexy bikini shots, social-network data) took perhaps a trillion person-hours, but Fecebutt gained control over it at a cost of under a trillion dollars. Ballparking, they had to spend less than 5% of the cost of those assets to acquire control over them. Now it can leverage them to organize genocides, rig elections, and extort photos of government IDs and use them to out promiscuous women and closeted gay people—not to mention selling you tens of billions of dollars a year of stuff you don't need through advertising.
It's hard for positive-social-value companies like Shopify and Datadog to compete on salaries with that 20:1 value multiplier. The fact that they manage it at all is not only a testament to the enormous value they're producing but also the enormous destruction of value implicit in Facebook's business model.
However, no process can remove actual randomness (did anyone get the necessary ideas, for example) or uncertainty (is it even feasible, for instance).
That said, in larger - and I'd say mostly non-software companies - changing processes to things that are more inclusive, i.e. tech and the business talking directly to each other does help. And one way to sell that is under an agile/Scrum kind of construct. To me, seeing Scrum then more in non-tech/consultancy makes total sense.
Maybe they were missing what I noticed missing from my last Scrum-focused company: view and attention from the higher altitudes.
Scrum makes it easy to break down features and problems into nice little pieces which can be done within one sprint (often in one day). That is a win in many ways, but it doesn't encourage higher level reviews, refactoring, etc. It's the problem of not seeing the forest for the trees.
Of course you can intentionally create epics for these kinds of reviews, but those don't have a good short-term return value for time spent. And so, they get inadvertently or intentionally postponed. Eventually the system is complex and messy, few people understand the system in entirety, and the easiest Scrummy solution is a re-write.
For me, Scrum was a straightjacket, a hundred NOs against my creativity and my attention to broader concerns. There are times when you are implementing a simple ticket, but you keep touching things that need attention... or you see how the entire system has become convoluted, and you see a better way. To do it the right way either means creating new epics and tickets, debating those with product owners who don't understand the value, and generally NOT solving the real problem.
Now back to a personal opinion: Most "software engineering" "methodologies" are intended to appear productive without actually requiring productivity. Or, as I sometimes put it, "software engineering is about making people who are fundamentally not very good at writing code look as if they were producing code." Look at the number of managers, leads, coaches, trainers, and consultants required for every methodology and compare that with the number of such "overhead" positions in, say, open-source projects. (Note: I'm not trying to start a "which is better" discussion; open-source is just a convenient example since their development techniques are transparent, unlike everyone else's.)
Further, many medium- and large-scale initiatives in large, non-tech enterprises fail spectacularly, in spite of using the then-popular methodologies. I would go further and suggest that they all fail at some point, even if they eventually deliver a usable product.
The only solid generalization I can see is that good developers can develop good software with any (or no) methodology; poor developers cannot, and most formal methodologies are built around trying to get them to do so.
Seriously, though, having your software enforce too much prescriptive workflow on you is dramatically worse than having it enforce too little. Historically it was common for every software team to write their own bug tracker for this reason.
It is a fuck-ton of work to set Jira up so it gets out of the way, but it has all of the customization I need to serve most of everyone’s needs.
If anyone so much as thinks about enforcing workflows on our scrum board, they are going to hear my polite, but firm, rejection of such meddling. It’s our board and we let other people see it so they don’t have to bug us with requests for status updates.
Jira isn’t a hard no for me because I’ve been an admin for it at both this job and my previous one. Let my team manage our stuff, and I’ll happily set up Jira to keep everyone else’s grimy paws off my team.
I'm still employed there so I guess my methodology (or lack thereof) isn't that bad ;)
We planned what we were going to do the next two weeks. During that time urgent work would also be added and other things wouldn't be finished, but they just moved to the next sprint. Releases were quarterly (customers didn't like frequent releases) so not related to the "sprints".
Some managers described this as "Scrum light", fine with me.
First of all, I work at Amazon (and posting this under our social media policy, this is just my experience, and I don't speak for the company), and Scrum is pervasive at Amazon. The article is right that many big projects start with a 6 pager, but once teams start executing, many teams run some form of Scrum (Kanban is also popular).
The second thing I found strange in this article was the idea of Scrum being heavy-weight. Early in my career I worked at a small company where our process guru was a big fan of the Unified Process. The point of Scrum is that it is lightweight (without being zero process). There really isn't a lot to it. You sync with your teammates daily, to make sure you're all focused on the most important thing today. You check in every 1-4 weeks by (a) demoing your software to figure out where you are, (b) you do a retrospective to look at what can be done better, (c) you figure out what is the most important thing to be doing the next N weeks. The teams that I've been on that have tried to not do these things either in actuality, did these things, but did them subconsciously, and therefore unintentionally. When I hear people say "Oh, I don't like Scrum because my old team did Scrum and I didn't like how we..." what follows is almost inevitably something that if you asked a professional scrum coach "Should we do that?" the answer would be no.
> Competent, autonomous people need less structure to produce reliable, high-quality output. Big Tech is able to attract, afford and hire these people.
I don't like this way of thinking. Start-ups have lots of competent, autonomous people. Big Tech companies have lots of people early in their careers that are still learning to be autonomous. In fact, when I was at start-ups I often heard the opposite (and equally untrue) claim: that start-ups required people with more autonomy, because the product is new and the team is involved in discovering the product, whereas in larger companies the products are more defined, so people needed less autonomy.
I appreciate the author surveying people and trying to make sense of the results, but in this case I think he just over-simplified a much more complex reality in a way that isn't particularly helpful.
The big tech version of scrum is just sprints, stand ups and shipping often. There’s still a waterfall process, mounting technical debt, huge backlogs where everything is high priority, and over committing to slipping deadlines.
I think people should think twice before looking at big tech as the place to learn good management practices from.
Cloud-delivered CRUD apps are the easy mode of SW development and people who have never done anything else really end up quite stunted IMHO.
I actually agree with your first point too, I do think big tech devops and security practices should be copied.
The point I was trying to make is that I don’t think big tech management is actually that good. Big tech is optimizing for staying big (reliability, security, etc.) the famous projects are usually the most uninteresting to engineers cause they ship a handful of new features a year and you spend most of your time working on bug fixes, infrastructure updates and security patches so those projects end up having the most churn in engineers and management. Because of that churn these teams tend to adopt the practices most suited to oversight (stand ups, sprints, retrospectives, etc.) instead of the ones best fit for improving the customer experience (grooming the backlog, collecting user feedback often, etc.). So that’s why I say we shouldn’t be using big tech management practices as an example imo, cause big tech still hasn’t decided on a solution to the problem of slipping deadlines, exploding backlogs and insurmountable technical debt.
Perhaps it's just a difference in the scope and type of projects (boutique hardware vs. large scale online stuff).
Looking back at old codebases that I kept around, we managed to build quite complicated largish systems in a reasonable amount of time using straight waterfall/ad hoc approaches. Simple source control. Simple spreadsheets for bug lists.
Admittedly it's hard to avoid attaching particular people to particular blocks of a system, so perhaps making development capable of absorbing random Engineering Resource Units (humans) is the point of all this.
..or maybe it's just that earlier generations of programmers were stupid. I can accept that.
· nightly builds or continuous integration;
· pair programming or code review;
· automated builds;
· comprehensive test suites or test-first programming;
· fixing bugs before adding features;
· source code version tracking software;
· keeping a list of bugs and planned features instead of not keeping one;
· prioritizing planned features instead of not prioritizing them;
· merciless refactoring to keep the design simple;
· DRY, as a criterion for what "simple" means;
· "YAGNI" (not writing code to provide functionality that I'm not implementing right now);
· a sustainable pace (no death marches);
· exploratory "spikes" to explore unknown features;
· taking regular breaks;
· informal usability tests with prototypes;
· standup meetings;
· frequent releases;
· quiet working conditions; and
· retrospectives.
I'd add, though these usually go without saying these days:
· high-level languages;
· interactive development environments, as opposed to the batch-mode approach where you submit a batch job and get back your program output an hour or a week later;
· enough testing hardware that you never have to wait for a testing machine to become available in order to get your work done;
· using available software libraries instead of not using them.
I'm pretty confident that each and every one of these helps me build better software faster, though a few of them may vary somewhat depending on circumstances—I know a guy who can concentrate to write code better in a noisy café than in a private office, frequent releases are at best minimally useful for pacemaker firmware, and you couldn't have written qmail using existing libraries.
Also, though, I'm guessing that the old codebases you're looking at with simple spreadsheets for bug lists practiced "working software over comprehensive documentation", "customer collaboration over contract negotiation", "responding to change over following a plan", and especially "individuals and interactions over processes and tools". And those are the core aspects of agile development (though not the Fake Agile we so often see).
Some of these are things earlier generations of programmers could only rarely do. If you're working at a company that values processes and tools over individuals and interactions, there's not that much you can do to fix that if you couldn't start your own company; https://news.ycombinator.com/item?id=28670326 suggests one reason this is so widespread. Similarly for fixing bugs before adding features, using an interactive computer, having adequate testing hardware, and having quiet working conditions. Source control systems, automated builds, and automated test suites provided substantial benefits to teams that adopted them, but many didn't. That doesn't mean they weren't valuable at the time, just that many people did without.
By contrast, I'm less convinced of the value of private offices, written spec documents, schedules, dedicated low-paid testers, scrum masters, planning poker, splitting and merging product backlog items, quantitative task estimation, and even coding interviews.
I occasionally see statements like this floating around - does he mean:
* Devs make their changes locally, commit and push directly to main, and then the CI/CD either notifies them that tests failed or deploys to prod
or
* Devs make their changes locally, commit and push on a branch, and rapidly (ideally automatically) merge to main if tests pass. This happens on a cycle of minimum coherent set of code changes, not a whole feature at a time or anything like that.
The first seems like it would be very frustrating if code with failing tests if pushed with any frequency, but seems to be literally what the phase "work on main" means. I also don't really see a drawback to the second one in comparison.
Yep, that's the point - it's to discourage devs from checking in failing code (because then they'll be swarmed by annoyed coworkers). It's fairly common to see developer A make a breaking change to some low-level library and then see developer B push a rollback of A's changes a few minutes later (with or without A's permission).
And presumably there is no need for a staging env, since your code is deployed to production as soon as it has passed tests.
Edit: Just to clarify, it isn't ok to break downstream projects, if you do the change gets rolled back almost immediately. And you can run all tests for the change before submitting, but it isn't default since it is expensive for many core projects to run so many tests.
Not saying I have a better solution off the top of my head, but "just fix it with another commit" is a big red flag IME.
Fundamentally this all seems like a misreading of trunk-based development, though. The whole idea was to get away from old Subversion-style branching, where there would be long-lived branches running for sometimes several month. If you were a company that moved from svn -> git, trunk-based development was a way of communicating that branches were cheap and disposable, and merging was quick and easy.
So unless you have a ton of dependencies you wont notice much breakages from other teams.
No, in trunk-based development, you push to the trunk, aka master.
The difference is perhaps not that big; with TBD, you have a branch, but only on your machine, which you merge to master before pushing, whereas with the second workflow, you push your master, then merge somehow.
> their intermediate setup seems impractical technically for most setups - how do you run standard tests without committing your code, unless you do everything locally?
I don't understand this. Of course you can and should run all tests locally.
If you follow the ops link there are multiple diagrams where that is not the case.
And
"The short-lived feature branch should only last a day or two and never diverge from the trunk enough so that a merge back is problematic. After the seal of approval from code reviewers and CI daemons, it should be merged back into the trunk. It should be deleted, as proof of convergence. The developer in question may then go ahead and make the next short-lived feature branch for the next story/task they’re doing" https://trunkbaseddevelopment.com/youre-doing-it-wrong/#dura...
For larger features, these changes could be stacked in a list. After each change in the list had been approved, they’d be landed on main as an all or nothing.
So yes, you work locally on a fork of the repo but not the main/master repo. I'd surprised if all devs have access to push to main/master.
I think having branches leads to a lot of merging issues. Because technically, the branch will then be used by multiple devs. So it's like dealing with multiple masters, kind of.
Personally, I prefer the TF approach. Oh I forgot to mention, the release tags are branches I think, perhaps read only branches.
https://docs.github.com/en/repositories/configuring-branches...
A fork is still a lot less friction. Otherwise, technically yes it is a branch.
For the rest the merging issues are identical. You can still have a merge conflict between downstream/master and origin/master if someone merged something conflicting into origin/master since the fork happened.
It's very unusual to merge more than one commit to master at a time.
You can think of it as a branch and merge approach, but the set of changes is small - a 100-line diff would be a large change (automated changes and deleting unused code excepted). The process and tooling optimizes for small changes. Refactoring is done incrementally. Big changes are gated with flags so they can be built incrementally and rolled out incrementally and turned off at the first sign of trouble.
At my company top management often comes from a sales or medical background and you can clearly see that in how they treat sales people and medical people. They get promoted more and receive way more attention from management because they understand each other.
IT and software are a big, scary mystery to management so they are prone to take advice from snake oil salespeople, consultants, or middle managers who tell them what they want to hear.
I think in general I think you are better off in companies where the output of your work is the product and top management understands and respects your work. Better to be viewed as someone who contributes to the profits of the company and not just as a cost center.
My role (as I see it) is to keep devs motivated, translate this mystery of development for them and present some kind of roadmap. As much as I would love to only work with empowered teams, the reality is that with these big nontech companies are not ready to change anytime soon.
Maybe we worked at the same place?:)
Seems we are at different companies and my company is a step ahead. After Accenture then comes TCS where nobody knows anything :).
Sure Scrum has some ceremonies, but these are basically planning and retrospective for improvements, so again, not really any different.
At the end of the day, most of the ways I have worked end up with Tickets/PBIs/Stories and you work through them. The only major difference is how much in-advance these are specified and to what detail.
I do not usually track my current task with a ticket or story. I have once week meeting with my team to discuss what I did last week and what I’ll work on next week. There is no 2 week sprint/constant stand ups/individual ticket estimates/retro. A typical plan is for a project that will take a month on short side or 2 quarters on long side. Deadlines tend to be soft with it culturally accepted that most deadlines are rough estimates and a week or two so deviation being fine. Process feels very light for me and I’m pretty happy with that.
All that being said, I can easily see how a dedicated team with a very clear product strategy iterating on a hugely successful business model can just roll along without a lot of oversight. I'd still be a bit surprised that anyone working on a customer-facing product isn't running their design decisions through product experts. Idk what that world is like, but I've spent a lot of energy duking out product debates where nobody is really sure how a feature should work or it's even beneficial and developers usually have the worst instincts.
So how do you fit in this new requirement ? You are already in the middle of a feature built on day 3 or 4. Devs are working on their forks and have uncommitted code. Now you have a new feature that has to be given priority, assuming there are no dependencies on the existing cycle. Do you just keep a copy of your old work, and start working on this new stuff ? Feels like a mess. Any better approaches ?
The best thing you can do, in my experience, is make sure that the devs are fully bought in on why X is so much more important than A,B,C. Ideally, you want to present it to them as 'X enables us to get this contract/solve an uptime issue/whatever' and then let them decide, as a group, that maybe we continue on B while the two working on A and C instead focus on X.
I haven't really seen a better option. Urgent work is disruptive to schedule by definition.
Maybe the larger, successful tech companies avoid doing this. I believe that most of them define work at a quarterly level. So a team is unlikely to have the sudden changes in direction.
They are also unlikely to be resource-constrained, so when the odd high-priority feature emerges from the ether, they can put together a new team to tackle it.
Smaller companies need the discipline to know not to suddenly change everyone's priority, and instead, work emergent features into the current backlog at a pace appropriate for the size and scope of their teams.
I think you want to have a culture where abrupt new features are incredibly rare.
As someone who has spent most of their 7 year career at big companies that follow "plan build ship" cycles, I don't understand this post.
What abrupt new feature with urgent deadlines? Why can't this hypothetical feature wait?
Sure, for Coronavirus teams spun up and other work stopped, but that's a pretty big outlier. Most features are planned a quarter or so ahead.
> Do you just keep a copy of your old work, and start working on this new stuff
Sure why not? I don't understand how any "abrupt new feature with urgent deadlines" wouldn't require stopping what you're currently doing.
Besides, it's not like I sit on massive code changes for weeks. I submit code incrementally.
The Swiss Franc is no longer pegged to the Euro, so our trading software needs to track a separate currency, and also we need to stop trading EUR/CHF forex pairs as if they were still pegged; we lost $2 million before I turned the system off this morning. The new version of IOS doesn't support the old SNMP MIB, and our customer has already installed it. The vid.me domain has been bought by a porn spammer and now we have hardcore porn ads on all our news articles. YouTube revved their AJAX and our video metadata scraper stopped working. We're being hit by an 0.1Mbps DoS because this one page takes a whole CPU second to render. The new satellite downlink site came online six months earlier than expected, so our comms software needs that multisite feature next week instead of in six months, or we're paying a shitload of rent for nothing. Modi demonetized large bills last night, so we need to stop accepting them at our ATMs. Apple told us yesterday they won't approve our update to the iOS app unless we add in-application purchases (and disable every other payment method). We signed a contract to provide security for a new film, but they're filming onsite at an Air Force Base, so we need a field in the database for which security guards are cleared to go on the base, and filming starts on Monday. Turkey just ran all their ISPs through CGNAT, which means we need to fix our NAT traversal strategy so it works through two layers of NAT, or we lose half our Turkish users. Russia is fining us $100,000 a day until we move all the data about Russian users onto our Russian server. Our popular webcam software just doesn't work with this popular new webcam. We've identified a new signal in the market that we think will give us 0.7¢ per trade, but someone else will probably exploit it within the next two weeks, and then it won't be profitable anymore. It looks like Berkshire Hathaway is going to break US$671088.64 per share this month, and we need to be able to handle that. Apple just announced that in iOS 11 the default video codec is HEVC, so unless we tell our users how to reconfigure the Camera app (and they reshoot their videos) we need to be able to handle HEVC uploads before people upgrade their phones to iOS 11. NCMEC just gave us the new version of their damn fuzzy hashing algorithm; they'll start giving us hashes in that format next month, and if we don't support it by then, we're kiddy porn purveyors. Oh, Argentina canceled daylight savings time with 11 days of warning, so all those calendar entries our users scheduled two weeks from now, they now have the wrong UTC time and need to be fixed, plus we need to roll a timezone update. We finally signed the deal with Nicki Minaj to make her a character in the game we were about to ship, but we can't slip the ship date past Black Friday or our sales will take a huge hit.
(Names and details have been changed to protect the guilty. Some events haven't happened yet.)
1. Rare actual urgent issues
2. Shit leadership adding last minute requirements (that no scrum or waterfall or agile whatever would help with)
3. A bunch of config changes (I worked on payments systems in the past. We had some contractors who would basically mess with config files all day as countries changed laws, etc).
4. Stuff that only quant firms care about
Regardless, you didn't address the second part of my comment: What does the fact that we'd have to stop other work have to do with anything? That'd be the case for any workflow structure.
> “Kitchen sink teams” which have everything thrown at them, typically find that managing stakeholders with a heavyweight process like Scrum is a win. Stakeholders are educated to understand that an ongoing sprint cannot be interrupted and that new feature requests need to be groomed.
I have repeatedly see teams that run like this fail. They're happy in their own right, they've got their process, they get what is in front of them done, but the process makes them entirely unable (actually more often just unwilling) to respond to urgent issues or adapt to a changing environment.
It's absolutely great for a team that has a large number of external stakeholders with diverse needs to be able to point at some process and say "Oh well here's the process we follow, here's what you need to do". But in reality what they're saying is "We're entirely unable to respond to the actual requirements of our stakeholders, so instead we've decided there are only a subset of requirements we're going to fulfill" and anything outside those requirements you just have to work around the team. It very often also works to provide barriers between the stakeholder and the engineer, so by the time they tackle they're either doing the wrong thing, or it's no longer needed or they fail to actually deliver it to the stakeholder. It's a great system to build a well functioning team that entirely fails to deliver what the larger organisation needs. /rant over I guess
Also what happens is you end up with a never ending dev cycle where business expedites an item making the cycle a little longer, business sees the cycle is longer and thus adds in more items to be included further lengething the cycle... I think you get where this is going.
At my startup company we have suffered from just this in past, where we would have never ending dev cycles while our maintenance back logs filled up and never got worked on properly.
We trade the story of one Agile Founder wanting to kill his monster baby, but the others have been making bank.
Why make up an easily debunked conspiracy theory?
Scrum wasn't invented at big tech..
Agile development in general became a thing in the mid-90ties.
The point was that whatever the origins and intent, Scrum benefits Big Tech (who largely avoid Scrum as per OP) by hobbling smaller competitors who adopt it.
You need to have rules (however dumb you might find them), you even need babysitters. There's no going around that. Someone making 1/4h of what a SV engineer makes is highly likely to be less competent.
The whole idea that you can be agile while having delivery dates for things several months in advance is a nonsense. Agile is about doing what is important now, not doing what we thought was important a few months ago.
The best you could probably hope for is Kanban or the never-ending backlog but some people struggle when there is no finish line, just an endless sea of work!
Not sure where the idea of "only look one sprint into the future thing" came from.
"The whole idea that you can be agile while having delivery dates for things several months in advance is a nonsense. Agile is about doing what is important now, not doing what we thought was important a few months ago."
I agree with you - i dont think its worth trying to set a date a year out in stone. but you have to do exactly what you say. keep the trajectory in your mind and update it with new information. and and some point turn down the rate of change so you can get some cook time.
maybe I reacted too strongly - but I've certainly been in shops where I try to talk about overall project arc and am firmly told to keep my gaze fixed at exactly two weeks.
If Agile truly rejects even the possibility of an estimation (something I'm sure many proponents will disagree with), that would be a problem with Agile and not a problem with the idea of estimation. Companies will de facto pivot away to methods that will give an estimation, even if they're 'bad Agile' and even if software engineers are unhappy about it.
Because there are more useful things to optimize for than efficiency of software delivery or what software engineers like. People really need to know when the works on their street will be done, companies and countries need to plan ahead, and so on.
The flexibility of scrum is being able to say what is done represents a working though not fully featured product. You can generally send red flags a lot faster in scrum, but you still need management that believes and understands those flags.
The thing I think that fails to be recognized about Agile (though there is a lot adjacent to it, without outright calling it out, in the central documents) too often is that Agile has fundamental tension with the project-oriented, and even to a lesser extend product-orient d, mindset, and really calls for a mindset in which technology is an integral component of continuous end-to-end business process improvement. It can be adapted to product- or project-oriented environments, but only with compromise.
> The flexibility of scrum is being able to say what is done represents a working though not fully featured product.
The flexibility of Agile in general comes in rejecting products as a goal to themselves, and recognizing that technology is part of a broader system of delivering value and any improvement in value delivery is worthwhile according to the value delivered compared to the cost of delivering it.
"Exposure to the business and to business metrics. Engineers are encouraged to interact with the rest of the business and build relationships with non-engineers. In contrast, traditional companies often make it impossible for developers to interact with the rest of the business."
No, that's not at all what it means. Another person who does not understand the bullshit that is NPS. NPS treats people rating at the lowest of the scale just like people rating at the middle of the scale, which is patently ridiculous.
Of course NPS scores overlook some nuances. It's a single estimate of a complicated space. The imprecision is probably to its advantage. The goal is not to be perfect (no single number could do that), but to provide a quick estimate of quality with a single question (nobody wants to fill out a 3 page survey, and the executive team is only going to sit and listen for a single statistic).
As an exercise, consider how you chose a restaurant based on a standard Yelp 1-5 scale: you might find that you are using a similar methodology while reading product/service review. Ratings of 1-3 are considered bad reviews, 4s are decent and 5s are good. American culture as a strong bias for positive affect so a 3 is considered to be unfavorable. Additionally, there is a lot of variance in terms of how people get upset and express their anger. Someone may rate something as a 1 because they are more expressive of their angry, while another person rates it as a 3, but neither is really happy with the service. Try coming up with a better metric based on yelp reviews for restaurants you are familiar with and you might find that NPS does a reasonable approximation of what you have discovered.
The approach is described here https://www.gordiansoftware.com/news/two-weeks-too-slow and in practice it’s even better than anything I was hoping for. I can legitimately say this is the most productive environment I’ve ever seen.
As for solutions to document stuff, specifically issues, then Jira seems to be ok and not lacking in any area, maybe not a leader but not a worst thing to use too.
Ding ding ding!
Seriously though, I think a lot of issues with Jira stem from poor implementations that cause it to perform terribly, which impedes an engineer's ability to quickly get in and out of Jira. For example, the on-prem Jira instance for one of my clients would take upwards of five minutes to render the board, and another 30+ seconds to render the ticket page. Adding comments was another frustrating experience. I'm sure the OOTB experience with modern Jira is probably fine, but it seems that hardly anyone uses Jira without some resource-intensive customization.
Each has benefits and tradeoffs. Like all tools the proper process type for your project will be context sensitive.
For an industry so full of smart, supposedly logical, people ours is still quite immature and tribal in its management theory.
I think the big takeaway is "dont worry about your project managment system if you are playing the wrong game."
The sprint is also a product of its time. A methodology for shipping new features on a monthly basis is great when your competitor is shipping Excel on an annual basis. But the shorter the turnaround time for a feature, the less use there is in planning timelines.
This is why the big tech companies don't have a unified process -- the process that works best for shipping operating systems is different than the one that works for shipping a mobile app, is different than the one that works for shipping a webapp. Any program office that tries to smooth out these wrinkles is hazardous to their wealth.
Data from article here: https://docs.google.com/spreadsheets/d/1Cz7NqDblls_TBJVI4xwD...
Here's some professional advice: The idea that releasing more often is a good idea is total rubbish. Eric S. Raymond said it originally ("release early, release often") about open-source. People have applied it to software in general because of survivorship bias. Web devs tried it, and surprise, web dev can get away with shipping an update multiple times a day. Your users probably won't even notice.
But what if your program isn't a webapp? Updates aren't so seamless. If you update Skype 3 times a day, your users will be prompted to update Skype app 3 times a day. Do you really think that's a good idea?
ESR wrote http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral... in 01997, but Jim McCarthy said "Release Early, Release Often" in his 01995 software-process book https://www.amazon.com/dp/1556158238 (which I haven't read) according to https://devops.stackexchange.com/a/80.
Releasing often isn't always a good idea, but it isn't necessary for every user to upgrade to every release. Here are some advantages of releasing every iteration (once or twice a month):
1. You can never have the software in an uncompilable or unshippable state for an entire iteration, much less three months at a time, so the only time you don't know how long a showstopper bug will take to fix is immediately after you find it, and you're never "integrating, don't know how long this will take," because frequent releases force continuous integration. You never slip a ship date, at least not by more than a week or so; you just cut scope. Slipping can be enormously painful and sometimes fatal to a business.
2. A hacker costs a business US$200k a year. A team of ten hackers costs US$38000 per week. If you release once a quarter, then, on average, at any given moment, you have six weeks of work that you've paid for that haven't reached users and are thus not yet producing any value; that's about a quarter of a million dollars of "inventory"—having a lot of inventory might sound great if you're a survivalist or an RPG player, but for a business, "inventory" is wasted money. If you can cut that to monthly, you only average two weeks of inventory, so you've saved US$170k, almost the cost of a person-year of work.
3. When you ship software to users, they react, which is an opportunity to learn from them about what they value and what they don't; this both gives you ideas for new features and gives you more information about the value of features you'd already thought of. This is true even if only some of your users install the new release. If you ship once a quarter, you get four of these round trips per year; if you ship once a month, you get twelve. So you can learn a lot more from your users. Evidently that wasn't enough to get Skype on track.
4. If you're a business, the business environment changes as you're developing software: for example, your competitors add new features, systems you interoperate with change their behavior in ways that break your software, and new libraries become available. To remain profitable and in particular to remain better than the competition, you must observe these changes, orient yourself with respect to them, decide what to do, and then act. In competitive environments, the competitor who can go through this "OODA" loop faster will almost invariably win. Shipping features that embody the right decisions for this month instead of the right decisions for three months ago can be a decisive advantage—though I think this is what WhatsApp was better at.
5. When your sales guy meets with a potential client on October 10 and finds out about a missing feature that's a showstopper for that client, and the October 28 release now has that feature, it gives you totally different chances of closing the deal than if the January 10 release has the feature. Even if they don't buy then, you've made a totally different impression, and you're more likely to make a sale later or get a lead on another potential client from them.
6. More generally, frequent releases make progress and quality legible to people outside the programming team, circumventing the "thermocline of truth" problem named in https://brucefwebster.com/2008/04/15/the-wetware-crisis-the-.... Having five features "80% done" after four months could mean you'll deliver them in another month, or that you'll deliver them in another 8 months. By contrast, the difference between delivering four of those features in four monthly releases, and delivering one of them, is a difference observable to users, management, sales, and customers. Sometime it's even helpful for keeping the people inside the development team from fooling themselves, despite being below the thermocline of truth.
I'm writing this based on the experience of utterly smashing two established competitors at AirWave, where upgrading one of our clients always involved a field engineer, often doing a site visit. We weren't writing open-source software, either.
> Scrum got in the way of shipping on a daily basis. The whole idea of Scrum revolves around Sprints, of committing to tasks at the beginning of the sprint, working on these during the sprint, and demoing what we did at the end.
What if each team member came up with a few tickets to work on during let's say a 2 week sprint and then these tickets were shipped as they were finished resulting in releasing new code potentially every day or couple of days?
I've worked with a bunch of companies (contract work) and very rarely did they stick with 4-6 week sprints but only shipped everything at once in the end. That would be really dangerous. Almost always things were shipped as they were completed and reviewed.
The review at the end is for all stakeholders to be able to catch up on what's being going on the last 2 weeks.
It wasn't forgotten. Maybe the places I've done contract work for didn't follow scrum to a T. There was no daily planning. It was meeting once for 2 hours every 2 weeks to figure out what to work on and then a few small teams self regulating themselves asynchronously until things were done. This included doing the work, updating the ticket, having at least 1 person review / test the PR and it getting released. Usually there was a 1-3 day turnaround time from the PR being set to review to it being live on the site. Could be longer if the PR needed a lot of changes from things caught in the review process but the ball was always moving.
It never killed a single developer's productivity to wait on a review because every developer had a few tickets to work on during that 2 week process so they could jump to the next one or review another person's PR in the mean time.
There was no manager component once the tickets were chosen for the sprint (which was often a manager + developer group effort). Code could go from a ticket to production with no administrative bottlenecks.
On the other hand, one of my best friends works for the largest insurer in the company. They do about one release per six months and working in two week cycles would be an unimaginable speedup for them. It frequently takes more than a month to get word back from the regulators that their proposed code changes have been approved at all.
Both of the groups ("scrum slows you down"/"scrum is uncomfortably fast") have trouble imagining that the problems of the other group can even exist.
1 day doesn't seem too bad, often times it can be 2-6 hours. It really depends on what lines up and how big the PR is.
This is how most of the code reviews went for me at Google. If you do a bigger change it will take more time, but I see no reason why smaller should take longer than that. After all you are just sending small bits of text back and forth, it really isn't more complicated than a few chat messages.
The delay isn't always around the review itself taking long. It's having a dev working on their own tickets and then having an opportunity to check in and review someone elses code. You could end up getting a review on your code in 10 minutes or 5 hours based on what's going on for everyone at the time you open the PR. It might even take until the next day and for a bigger PR it could involve a few more eyeballs and time commitment for the review itself (outliers but they do happen).
I can't help but feel that people say Kanban because Scrum is so uncool. You can see that from the roasting Scrum's getting here. With these visceral reactions, I can see why people are cautious about saying they do Scrum.
As for Kanban, most teams are no more doing Kanban than they are Scrum. That is, they're probably just following Jira's process. I've asked candidates in interviews what they mean by Kanban (e.g. if mentioned on their CV) and the common answer is, "the process we use in Jira".
Can you imagine Linus, John Cormack or Brian Kerningan having to stand up every morning, providing updates on what bug or feature they are working on? Which one would throw its mouse to the Scrum Master head first, when he would complain about their project "velocity"?
Use the "One Hacker Way" instead: https://youtu.be/FvMuPtuvP5w
The goal was to get shit done and make customers happy.
On the other end of the spectrum, I have worked with teams that were pretending to embrace agile and scrum methodology though could keep standup quick and sharp.
It was an excruciating ritual that would take my most productive hours of the day: every day!
I always try to work with smart people. People who don't need much direction and know how to get things done.
The second is other large companies that deploy huge amounts of tech but have different business models. Banks are an example. Apparently JP Morgan Chase’s annual tech budget is $12 billion!!! That’s a lot of money.
I would say "lack of vision" is one of the key features of SCRUM.
All Scrum really means at a big co is an extra morning meeting that penalizes both people who like to work late and parents who need to get their kids to school. Plus you may occasionally hear the word "sprint" but it doesn't mean anything as you can be given an urgent task or have a feature cancelled at any time.
It is based on Patterns and the Theory of Constraints.
Has anyone here some experience with Shape Up? Love to hear your thoughts. I just read about it so far and would love to hear some real-life stories.
What miss is (maybe its me): How do you get into that execution mode from there? Break it down? High level estimates? How do get started (especially large endeavors that would be more than 6weeks)?
Thanks for any insights
Glad to see the author admit this... unfortunately most scrum cultists can't believe this can be the case.
Agile has mostly been a disaster IMO.. I've worked at big public companies and 2-3 startups during it's run. It works to come up with a "MVP for 1.0 and then it starts dragging everything down because it encourages tech debt.
My experience has always been the successful companies/teams were tongue in cheek telling everyone they were Agile when agile was popular while simultaneously trying to ignore Agile as much as they can because it was mostly ceremony and downsides once a product gets anywhere near mature. There was a lot of "we're agile with a lower case a" at places that could tell the Emperor's new clothes were suspicious.
Some of the old processes have been slowly creeping back in, and we're kind of at the point now where teams no longer need to feel like they have to advertise saying they're scrum/agile to appear competent. But I think you either need some older managers or something else to figure out what to do next and how to bring back the good parts of older process methodologies because it feels like theirs a vacuum.
The relationships between teams and PM have changed so much. Agile elevated PM a huge amount but as Agile moved on PM seemed to abdicate all responsibility to do their part in the whole agile process, and now you're left with a bunch of extra PM just getting in the way.
Agile without developers being included in estimates was one of the worst things.. non-technical PMs constantly shooting down estimates with "No, that is easy" is the worst. But Agile to me always seemed to be about management being able to absolve itself of all responsibility for not being able to figure out what to build or how to build it and to always be able to blame the engineers.
As we go forward teams need to look back at the past and what worked before Agile, not just make something new up out of whole cloth.
One agile team I worked on which probably was the strictest essentially pivoted the entire product to something else 3x because PM never had a clue what to build that would actually sell.
This is Agile’s “Executive Armor Effect”, which turns developers into a sort of ablative armor that can take hits for executives.
Characterizing people as non-linear, first-order components in software development [1999]
https://dl.acm.org/doi/10.1145/1811226.1811241
Found TLDR:
http://www.csci.csusb.edu/dick/biba.php?search=Cockburn00
Sorry, Cockburn's OG website is apparently gone. And I can't quickly find a naked link to this paper.
Am I the only one who thinks Big Tech is slow at shipping?
Also, shipping fast and actually innovating are not the same.
I would love to hear from Big Tech employees if they feel like they ship a lot. To give a baseline, 1k LOC per developer per week is a lot. That's what the highest performing teams can produce.
As soon as you scale scrum to organization wide, you get inter team communication challenges for which Scrum has no clear solutions (because its a team methodology and not company methodology). Once you have multiple teams that need to align their plannings to get stuff done, inter team dependencies become a bottleneck. Add a few external teams to the mix and you get a perfect storm of conflicting interests, miscommunication, politics, etc.
Having a lot of dependencies between scrum teams also means you get a lot of decision making lag. Scrum teams have two issues: they work in sprints with some fixed length and you can't introduce new work during a sprint. If either is not true, you are not doing Scrum. When you have a graph of interdependent scrum teams, the lag you build up to get anything done is basically proportional to the path length through the graph. The bigger the org chart, the slower things get and it adds up quickly. The longer the path, the more planning and implementation sprints are involved in each team along the path and the more chance there is for things to get lost in translation. Beyond 3 vertices, the whole thing starts looking like a very chaotic waterfall style process where it takes months/years to even agree to do a thing.
I speak from experience. The most dysfunctional thing I've seen on this front was Nokia flying in teams from three continents to have a bi-monthly sprint planning about 12 years ago. That worked just about as well as you can imagine (i.e. not at all). You needed to get your requirements scheduled 3-4 months in advance to stand a chance in hell of anything happening under 6 months. I'm talking critical stuff getting de-prioritized because one or more teams involved got other priorities imposed on them and thus had to say no. One of two things happens in such organization (usually both): 1) things get escalated to VP level management, a lot, and there is a lot of politics with lots of egos clashing. 2) teams eliminate dependencies to each other to speed things up (i.e. they reinvent wheels they could have reused just so they don't have to deal with each other). Both are bad for productivity and cost.
The way out is to bypass middle management, ignore the process (on both sides) and establish direct lines of communication with engineers in different teams. Once you have that, pragmatic decision making can actually happen. At this scale, middle management should just not get involved with day to day engineering decisions and instead enable the day to day interactions and smooth out any inter-organizational bottlenecks.
Most of the big companies mentioned in the article are not great at this either. E.g. the reason Google has so many chat tools is because they have teams working against each other rather than with each other.
The product owner is not even tasked with approving the work done! Nor having any real kind of spec. However in my experience they still have veto power and the power to change what the project is at their whim.
This is all done with the idea of “autonomous teams” held firmly in mind. Yeah you’re autonomous... you can decide for yourselves how you will do what the project owner has decided they want today.
But they are still the owner! And will be the face of success when your project ships.
I suppose it’s a good thing they pay well.
Individual teams can use whatever they want to manage themselves (Kanban, Lean, Scrum, etc…up to the team). Estimations planning and commitments for each quarter are done by developers exclusively, including coordinating across teams. Releases can happen at any point. A strong and continuously improving CI/CD pipeline is an expectation.
I really wish more people knew about SAFe. It’s constantly improving and refining the experience and from what I have seen, if there is a better way to do things SAFe will become it.
Also note up in the top left corner, where you've got the customer abstractly feeding work into this treadmill of... stuff. If your development team aren't talking to either the customer, or representatives of the customer, nothing else in agile will work.
> Individual teams can use whatever they want to manage themselves (Kanban, Lean, Scrum, etc…up to the team)
Except that they can't, because they will need to align on PI dates based on sprint durations, publish their burndown statistics and various other measures, and there's no way the poor soul who is reponsible for collating that will go into several different tools
> Estimations planning and commitments for each quarter are done by developers exclusively, including coordinating across teams
No they're not, they're done by the product owners, the system architects, the release train engineers (shhh... don't call them project managers), the product managers etc
> Releases can happen at any point
In my experience, releases can only happen at the end of a program increment, or maybe at the end of each sprint because until then nobody has integrated everything together
> A strong and continuously improving CI/CD pipeline is an expectation.
Fair enough, until the "innovation sprint" becomes the "bugfix spring" and all those good resolutions fly out of the window
> I really wish more people knew about SAFe.
So do I, just like I wish more people knew about colorectal cancer
> It’s constantly improving and refining the experience and from what I have seen, if there is a better way to do things SAFe will become it.
SAFe is constantly evolving, because that's the easiest way to keep the money coming in. It's as if PADI had found a way to change diving equipment every 2 years and ask everyone to recertify
Sorry for the tone, but I still have PTSD
SAFe consultants or not, it’s still up to your company leadership to implement it. A lot of SAFe consultants I have talked to try to advocate for things that follow Scrum because many teams are already familiar with it. This is honestly one of the biggest issues that I have found with the teaching.
80% of SAFe is based on Donald Reniertson’s “2nd Generation Lean Product Development” book, which is fantastic. Once you’ve read that it’s easy to spot the differences.
how dysfunctional must an organisation be for that to be an improvement? And in what way is that agile?
SAFe is about addressing the communication problems that happen as an organization grows. You’re looking at the diagram for everything that would be applied to a very large organization.
SAFe itself has a goal of using 50-150 person “release trains” that are essentially independent units. That’s the core of what’s going on.
Everything outside of that is about managing communication between other “trains”. In the diagram that you’re looking at, the bottom 2 sections where the CD Pipeline sits is where most developers will experience SAFe and you’ll find that it simply advocates for DevOps best practices.
I completely understand how everything else looks like a mess. It’s a lot to digest.
The quarterly PI’s are for planning because the most you can reasonably plan for with any degree of accuracy is about 8-12 weeks.
There are no deadlines imposed. You plan what you’re going to work on and how long it will take. You coordinate with other teams for any dependencies they may need from you and when. But you as a developer are responsible for saying how long it will take and when you can have it done.
Even after that, you’re only allowed to plan 2/3 of your time and the final 2 weeks of the PI are unplanned specifically to build in more buffer for you. If you do finish when you originally planned to, that 2 weeks is equivalent to 20% time where you can work on anything you want for the company, training for yourself, etc. It’s a built in reward for being on schedule.
And then as a group you discuss all of the risks that could prevent one of these items from being hit schedule wise. So if something comes up that keeps a team from being able to finish something, everybody was aware of it in the beginning and had the opportunity to try to help address it.
There’s no waterfall to it unless you consider anything that acknowledges dependencies to be waterfall.
Planning for 8-12 weeks at a time is the ideal window that lets you set expectations, coordinate multiple teams and avoid derailment without the idea that you can plan forever into the future. The farther you get from now, the more any plans become pure imagination which is why year long waterfall plans are a joke. It’s entirely possible to plan 8-12 weeks at a time. Especially when the point of the system is to acknowledge that estimates are at best a guess.
There’s a balance between setting expectations that business people can plan around and working in a box without communicating anything other than “I’ll let you know when it’s ready.” SAFe strikes that balance better than anything I’ve seen.
But everything will always boil down to leadership embracing it. Until leadership can acknowledge the fundamental lack of accuracy from software estimates, you’re screwed no matter what the methodology.
Which amounts to estimating software deliverables beyond the scrum window, which will invariably lead to disappointment, which will eventually fallback upon the shoulders of devs. It is impossible for delivery dates beyond the estimation horizon to not start being used by management as an anchor at some point in the near future. It only takes one manager promising stuff for the rest of the management team to try and emulate the savior.
Teams coordination only requires timely backlog priority management and does not need to be planned top-down for days on end with all members of all teams present.
Yeah, I feel the same about nicotine, lead, mercury, dioxin and asbestos.