Hacker News Reader

Comment by m104 | Hacker News Reader

Oh lordy, the "two crews" bifurcation fully written down. What a fantastic way to ship until it becomes far too expensive to ship anything good.

Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything.

Worse, your feature crews aren't learning anything beyond how to get those lines out the door, which will somehow get slower and more expensive as time goes on. Why? Because you removed the one fitness function on good software development which is to fully re-incorporate the negative feedback back into the source of development.

A real CTO leadership handbook would say clearly "it's your responsibility to help your developers improve, especially while shipping, and they're not always going to be happy about it."

View on HN

> It allows your feature team to remain 100 percent focused on the future, undistracted by customer support work.

AKA "it allows your feature team to be completely oblivious to the horrors they unleash, and keep at it until the ship is solidly planted in the iceberg"

Not talking about the conflicts it creates for merging between sales-supported feature teams and customer rep-supported maintenance teams. Given that the "customer crew" is described as something you grow out of, there's no question who wins arbitrages.

> It provides another career path for individual engineers, especially junior engineers, to learn and level up on your team.

"Senior staff doesn't want to fix shit so we have juniors do it"

Further down the article:

> The Microsoft blog post referenced above recommends swapping some team members between the two crews every week.

This would hopefully mitigate the worst of the effect you describe, since everyone eventually gets exposed to the consequences of poor feature development.

I don't know about you but it's rare that I've neatly wrapped up my tasks at the end of any given week. Single-day tasks are rare, there is always carry-over work including over the weekends.

The only thing worse than a feature that got rushed out the door Friday afternoon because you had a completely different role come Monday is one that was 80% done then passed off to someone else because you had a completely different role come Monday.

In my company, we rotate every 5 months. So every 6th month, I get put into the customer-facing team for 1 month. Every other month, a different team member is on the customer-facing team.

This is still annoying, but gives you enough time to work on features, and enough time to try and crack some customer cases (though I could even see being in the customer-facing team for more than 1 month, as sometimes, this is not enough to debug the issue and provide a fix).

I've got to admit, as much as I dislike being on the customer team, it's certainly less annoying than working on features, and have constant customer issues interruptions though.

Maybe when you know you're due to start support the next week, you stop feature work sometime the previous week and do small maintenance/backlog tasks and or documentation. Like a cooldown period before task switching

Further, I'm not sure what efficiency it provides overall. Is dedicating 20% of your team to support _that_ much different than the entire team spending 20% of their time on support?

We've actually found our quality goes up massively when we force our engineers to deal with the problems in the features they ship, directly with customers. We still have dedicated front line support (that rotates weekly), but they run off a playbook for common support needs then delegate everything else out.

It really sucks when you get pulled into support a feature you launched, but it really makes you want to build your next features better. Better internal documentation, better customer documentation, better UX/requirements, better edge case handling, etc, etc.

Would putting some percentage of the team on 'support' for a week or two help with reducing task switching and help to allow deep work? Maybe everyone in the team would spend 2 weeks per quarter or something like that doing support.

I (n=1) would prefer to be answering support tickets for 2 week blocks, and know when the blocks are in my calendar, so that I can plan work around them, rather than trying to debug something while I am being pinged about unrelated stuff all day.

hand-off needs to be really carefully defined and managed efficiently. Client tickets rarely respect arbitrary calendars, and the context switching alone can be really expensive. The best I've seen is a primary/secondary setup where you move from copilot to pilot, so you're not coming into everything cold

It's pretty hard to be fully hands off of customers. That being said, we don't expect immediate replies unless (1) you're the front line support for the week (2) something is on fire. We also don't expect immediate replies. Generally, within 24 hours is acceptable.

It's a bit of a drag, but most people just deal with their occasional support needs at natural context switches. First thing in the morning, before they head out, in-between meetings, etc, etc

> Is dedicating 20% of your team to support _that_ much different than the entire team spending 20% of their time on support?

Yes, it's [much] worse. Because nobody wants to be the support crew, so you end up with the 20% most junior, least outspoken people. Then the other 80% cares less about what support requirements will come out of the code they're writing because it's not their problem.

It's the perfect scenario for the aggressive prima donna who thinks their code is golden and everyone else's is dogshit.

I feel strongly that your front-line support should be full-time (not rotating) front-line customer support. That should be their job. If I reach out to a company for support I don't want my first contact to be with someone who writes code 95% of the time and this is their one week answering Zendesk tickets. I want it to be someone whose entire job is fielding customer issues and resolving them quickly and efficiently.

That's why you rotate everyone, not just those that "volunteer"... This way, you're spreading knowledge to everyone, e.g. if I'm forced to deal with an issue on code you wrote, I'm forced to learn about it.

Of course, I might have to ping you and get you to help me with it, so it's less efficient. Then again, if you leave the company, I have some knowledge about the feature, so... There's tradeoffs for sure.

I believe you're referencing the Engineering Management principle of "share shit work evenly".

Related topic, but every company I worked at that had a platform team (as in a third-crew support team that manages tools/practices/common-code for a discipline) ends up being infested with over-engineering.

They tend to attract that kinda of people who have disdain about delivering features and fixing bugs and like to over-abstract problems. Instead of fixing bugs they try to create increasingly complex abstractions to prevent the bugs from happening in the first place, with obvious results.

I wonder though if there aren't more forces at play. For instance, the business problems some systems try to solve really are so large and complex, you might need some kind of overseeing function in your company.

Also I have a hunch a team dedicated to providing helper "libraries" more than than "frameworks" could provide a lot of value without so much downside. If you can call a library function without it imposing a whole framework on the rest of your codebase, it's more self-contained and can't spill its abstractions all over the place.

If your org starts a platform team it is really important to have this concept drilled on early. Buffet, not framework.

I clearly remember having some discussions with platform people in my last job and asking them "why should I use your solution instead of getting an open source one that is likely better tested and used by more people" and the answer was usually "we can help if you run into any problems". Well, the "help" is to be planned and prioritized in the next sprint and probably will only come next quarter. So now the devs in my team need to make PRs to the platform people code and beg for reviews, how is that better than using the open source?

argh! PTSD - This was exactly what happened at my last start-up. Two of the engineering team and one from the R&D team started a platform team and it became a pre-PMF product with the slickest pipelines, DevOps, Cloud-cost optimization ready to scale to infinity. But with no customers, a broken front-end, and a under-funded R&D team as all the effort was put into the essential SaaS Platform. Truly put the company back 1 year while burning two.

That is actually usually not that bad (if there is, you know, revenue). What is really bad is when those teams start to roll out a lot of custom code that other teams need to use. If they are just configuring standard tools for everyone else it is usually fine (as long as they are not going to crazy with it).

The "platform" team at my company has rolled out a completely custom query language that we have had to learn and write so they don't have to make new endpoints to access different combinations of data

And they haven't documented anything

"There are integration tests, those are documentation go read those"

Good times

That's really the best, when not even intellisense can help you.

Yes, this is exactly what I mean.

This was the first place I worked at. The platform team became more and more insular and detached and more and more convoluted. As a result, things got harder to add on and soon they were telling the implementation teams that the features that the clients were requesting couldn't possible be needed. Million dollar contracts but no, you don't need to be able to put links into text blocks, that's a stupid feature and the client can't possibly want it.

insular architect waves hands These are not the features you are looking for.

That has been the fate of every platform team I’ve worked with in recent years.

Then they become gatekeepers, refusing to allow anything on their platform unless it conforms to their ideal vision. The catch is that their vision won’t be ready to use for 6-12 months, so you can’t deploy. Now your biggest problems aren’t engineering, it’s constant politicking to get around the platform team.

Add to this the concept of “architects” who don’t code but jump from team to team critiquing their work and you have a recipe for getting nothing done. One half of engineering is coding and trying to ship, and the other half of engineering is gate keeping and trying to prevent anyone from shipping

> Then they become gatekeepers, refusing to allow anything on their platform unless it conforms to their ideal vision

As the owner of a platform team, this very common attitude of platform teams kills me. Yes, we have a long-term vision that we're working towards, but our main goals are two accelerate developers AND produce more robust systems. Outside of totally egregious violations of company standards, my team is expected to focus on how to get things done. That means being flexible, working side-by-side with other teams, etc. to make sure that a) they're able to deliver what they need and b) we help them build it in such a way that it can eventually be aligned with our utopian long-term vision.

That’s exactly how every platform team starts. There is inherent tension between accelerating developers and building their own systems, though.

In my experience, the platform teams developed an idea that their conceptualized system would accelerate everything once it was done, but working with product teams was a distraction from getting it done. They also didn’t like the idea of deploying something now and then having to rework it later when their ideal system was ready. So they defaulted to gate keeping, delaying, and prioritizing internal work over requests from the product teams.

The only way to get things done was to leverage management chains to put pressure on the platform team to prioritize getting your thing deployed. This was constant no matter how much headcount the platform team received because with every new hire they developed new ideas and goals to add to their internal roadmap.

It’s not supposed to work like this, but it plays out this way in many companies.

> It’s not supposed to work like this, but it plays out this way in many companies.

Absolutely, and I've been on both sides. We go much more with a carrot approach than a stick approach, and have no ability to "block" any product team from doing things. Our goal is to ship things that are useful and lower the effort required for product teams to ship their products, which is handling basically everything except product-specific features. However, product teams don't have to use the platform, but then they own the operational burden of whatever custom stuff they're using. When that happens, we still work with them to minimize that or bake that capability into the platform and eventually take it over if it's useful to the wider org.

"Success" of the platform team really depends on serving the product teams, so blocking or being a barrier goes very much against that. We try to provide opinionated golden paths, but also try to build a properly abstracted stack of capabilities so teams can also extend/consume at a lower level if that better suits their needs.

There probably isn't any middle ground in practice then. If the product teams have control, then the tech debt just keeps building as they keep prioritizing new features over longer term maintainability. I see it already happening at my startup where product has a lot more influence than engineers in terms of what goes on the roadmap (there's practically zero time devoted to lowering tech debt.)

That's actually the "premise" of Google's SRE book

i think this is where a large portion of the tech consulting market comes from. Someone in business gets absolutely fed up dealing with IT and trying to get something they need in production. Next, they go find a budget, call a couple firms, get some proposals, pick one, and do it themselves.

>Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything.

This is the PM's job - one or a few people who are deciding the vision of how all of the features fit together based on feedback by working with customers. Customers (esp. non-technical ones) will definitely not have a coherent product vision and only want immediate fixes regardless of what else may be planned. Customers may also not communicate to one another and their feedback can conflict.

If you put this burden on developer shoulders, they now have to manage all of that communication in addition to requiring technical skills to know the code base and maintain it well, on top of every developer needing to have the same coherent vision to make thoughtful decisions. That's now two to three jobs in one depending if your developers also manage infrastructure like many roles are requiring these days.

What you're describing is exactly the opposite of every actually successful team I've seen, and describes every mediocre team I've seen. Silos are death and not just in a code base. Good developers understand the product. Mediocre ones churn out tickets mindlessly.

I'm okay with that knowing those developers are doing two jobs for the pay of one. And most products turn into that once the original developers leave.

It's not like you can't learn the product through the PM either.

I think you're conflating "doing two jobs" with "not being allowed to just type JavaScript into a computer all day in isolation and being expected to actually communicate and think about things other than data structures and algorithms."

If you're a true senior software engineer as most of us claim to be coding is a small part of your job, not your entire job.

You should be learning the product through the PM for sure, and I don't think a senior engineer should be doing first-level support, but especially in small companies talking to customers is good and should be expected from basically everyone who is working on the product.

Let's flip this around and see if it still fits:

"the PM can't be expected to sit in meetings all day, they need to learn the coding side of it too so they know the potential limitations of the features they want to suggest"

But if a PM does have a technical question, they don't need to go google stuff and figure it out - they ask a developer.

Likewise, when a developer has a product question, why can't they rely on a PM to answer that for them? Why must we also be expected to be in customer meetings and putting in extra effort, when PMs definitely won't put in effort to learn the technical side?

Yes, it still fits when you flip it around, speaking as an engineer turned technical PM. PMs should absolutely be technical and have enough depth of understanding about the product they can figure things out for themselves, as well as write code.

That's not going to prevent the PM from asking questions to the developers though. I ask questions all the time, because I want to validate my mental model with others and verify my understanding. Asking questions is a /good/ thing.

The part where you are missing the boat is acting like customers are a distraction or an enemy. Customers are /the point/, the /only/ point, really at the end of the day. Every role in every business is customer-facing to some degree.

> But if a PM does have a technical question, they don't need to go google stuff and figure it out - they ask a developer.

In a good organization they first try to figure it out themselves versus distracting a developer (thus costing possibly hours of productivity due to breaking someone's flow). The same way a developer would first try to answer their own question before they start badgering another developer.

Unless you're working 80 hours a week you're not doing two jobs. You're doing one job.

Giving away flexibility for free is a collectively dumb move on our part. If someone knows you can take on coding tasks and customer interviewing vs. just coding, you are more valuable to them and they should pay more for it.

They've already gotten away with adding infrastructure and architecture (aka system design) rolled into one developer position. And putting it behind long and stressful interview processes. I'm not doing PM stuff on top of all that and not getting the pay and prestige for it.

Why do you assume it's for free? The compensation of a software engineer varies widely. The same experience can get you anywhere from $100k to $1+m.

Perpetually doing less in fear of not getting paid enough for doing more is how you get paid a pittance while complaining about it constantly. Doing more and then finding a way to get paid more is how you get paid more and be happy.

In my experience, both are needed. Product owners and developers who understand the product. It's possible to have both, they're not mutually exclusive.

You're right, I didn't mean there shouldn't be PMs but rather that the PMs shouldn't be the sole people concerned with product.

Yes, but it's the Product Owner's responsibility to clearly understand the requirements from both customers and the business and communicate them clearly to engineering.

Having engineers handle "support calls" doesn't make much sense, they are not equipped to manage product feedback or understand the business implications.

One thing that we did to account for this was to shift the teams every or every few sprints. It allowed folks to get more experience, still get feedback, since if they built a buggy feature they'd have to fix it, etc.

People seemed much happier with that, because they also didn't get tired of 'always fixing bugs' or never getting the feedback, which you insightfully mentioned.

Developers must run and maintain the software they build. It's as simple as that.

And take the calls.

Don't like being paged at 3am? Write robust software and test.

The best development teams WANT this.

They will readily take on the responsibility to get the autonomy. The problem is many companies give the former without the latter...

Well, what's the time horizon? A PE backed outfit, or a CTO looking to move on within a year or so, would be well advised to follow this guidance. Lots of success now, and the problems deferred to later.

The book mentions having a rota:

> Engineers rotate between the crews on a regular basis. The Microsoft blog post referenced above recommends swapping some team members between the two crews every week.

In my experience this works well. With my current and previous client each team had a "hero of the week", whose responsibility was second line support and monitoring. If nothing came up the hero would work on their tasks as usual.

If something does come up the heroes of the week would be tasked with solving it or pulling in someone who knows how to solve it. This leads to engineers both having to accept accountability for writing shoddy code, but it also exposes engineers to the wider codebase when pulling on threads. It also solves the issue where no-one or the same person always takes responsibility for handling bugs.

Isn't this just called on-call? That's very different from a separate team.

Maybe - though I associate being "on-call" with being expected to respond outside of normal business hours which was not the case in the teams I worked in.

This just sounds like having a point developer. The challenge is too many companies expect this without giving up a feature-dev headcount. Any work the get done aside from point is a bonus and unplanned.

This may put me and my peers out of work (in a good way). SRE is a consequence of this function being lost, IMO. Pattern: developers don't like it? Give it to Ops/SRE.

Take away the escape, we will all be better for it.

I call them “shiny team and shitty team”.

It is not a problem if you measure and reward the infra team for their ability to enable the feature team, such as change lead time and deployment frequency, as well as the the stability metrics that the infra team might want to pursue.

Reply on news.ycombinator.com