Operating well – what I learned at Stripe
samgerstenzang.substack.com
samgerstenzang.substack.com
I know of a number of instances where folks left Stripe and took an extended break before doing something else. I've heard a few nightmare on call stories (we all have stories but these sounded really severe/frequent). I've heard folks describe their workload as operational heavy with little opportunity to actually develop software or build things. I've heard a few things that make me wary of past and current middle managers there.
It's funny. I imagine it's possible to have a company full of smart, kind, hardworking people but a culture where operational work is celebrated and embraced rather than solved and reduced.
I've had nothing but good encounters with Stripes and ex-Stripes but something seems off, and this post inadvertently reveals some of that.
I always take engineering management advice in books as a "do what I say not what I do" sort of thing. It's rarely the case that companies actually operate as well as the stories published about how well they operate.
That's pretty true for Will. Too much self-marketing. It doesn’t help his decision to leave Stripe for a failing startup that just laid off 20% of its staff.
- Direct link to book mentioned above:
https://press.stripe.com/an-elegant-puzzle
- And collection of reviews/interviews of the book on author’s website:
> Can we re-frame this in terms of the customer’s problem?
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
Well, that sounds like a recipe for high turnover. I know few engineers who would want to work for this kind of low trust boss and they are all bottom of the barrel consultant types. I thought Stripe was doing better than that.
The point of these questions should be to encourage really creative thinking to break out of "normal" operating modes (take on tech debt, etc) and if that's not well understood, or perceived to be an instruction to crunch, then you haven't set the context appropriately and will burn people out.
Like we were handed a solution to implement that architects and product folks had been talking about for most of this year. But they didn't include anyone that was day in day out hands on with the systems they wanted to change. To do that and then ask the people implementing if we can speed things up would be quite annoying.
It's asking them to contradict the design decisions that people 1-3 levels above them made. Risky business for sure. Even moreso for technical folks that in general aren't as adept at sussing out the difference between someone really asking and someone looking for a rubber stamp on a decision already made.
Providing a vision is the most important task for the management and leadership of a company. It permeates their conversations with other leaders who continue to spread it down.
The open questions mentioned in the article shouldn't be taken literally.
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
> What would we need to do to get twice as many customers? Ten times as many customers?
They strip away all the redundant and nice to have features leaving the core of what is truly necessary to build. That becomes the team's focus.
The questions are a thought experiment exercise that help inform and empower teams to say no to and ignore unnecessary work in the short term in pursuit of the vision.
Other parts of the article are about culture:
> Elevate each cross-functional partner you work with
And what is culture but the manifestation of vision and will.
Leaders and managers who incorporate these principles are the ones I want to work with. They are passionate, discerning, ambitious, and empathetic. They are rare but lead to a culture and work that feels gratifying.
It sounds like this guy is very good at running projects. I would love to work with him.
Is there any place in the world for a developer passionate about creating value for users and less so for fighting leadership-made fires (velocity! faster! quicker!)? I don't care to debate endlessly about velocity and processes. I just want decently thought of processes, to adapt to them and produce value.
As an IC I don't want to be involved in every step of the strategy, to keep a track of all my colleagues and how they work, to keep track of the tasks in the context of the epics and the milestones, and be blamed for when a strategy I was not part of doesn't work. I don't want to answer "how could you do this faster?" for myself because it's a horrible question to have to answer almost every week when you feel like you're burning out running. I don't want to answer "what is the priority of the task you are working on relative to our investor's desires?" without actually knowing who the investors are. I don't want to have feel impostor syndrome every start of the week and be in a team that gets ground down at the end of every week for the shit job we're all allegedly doing and where solutions are expected to come from us.
I just wanna get to work, have a clear roadmap that was developed by CxOs, leadership, consultants, investors, people with actual domain knowledge, deeply thinking about what needs to be done and in what order (in order to produce value for our customers). Then I want to create a technical road-map with my fellow engineers and engineering leaders that takes into account business requirements, technical know-how inside the team, planning and start working on that, in sprints, under the watchful supervision of ... supervisors that don't need me to answer questions that they should have had the answers to about a half a year ago.
I'm feeling my mind closing down to this kind of stuff. I don't know how much longer I can do this. I am now in a position where on Monday I get a pile of unthought of requirements, by Wednesday everything changes. Thursday evening a fire starts burning and if fix is not done by Friday morning, we are already behind and disappointing our bosses and investors. Doing the fix and dropping the current task, for urgency, will of course get the velocity / scope discussion. And the seemingly simple fix for the entire team is apparently that we should just increase velocity, but we're just too retarded to do that.
A number of years ago, I left the corporate world, as horrible as that was, there were some people that had answers. Since joining the startup world, perhaps due to bad luck, I've never had a proper answer. I always get a "I'll get back to you on this" and they never get back. If I push for it, it usually gets deprioritized even though it was considered top priority. If I push even more I have to start investigating it. Talk to stakeholders that are not aligned on answers, and it never seems to be about users. It's always growth and velocity.
</Rant>
The only responsibility for the IC would be stand ups with their manager to discuss progress and agree to future timelines. The ICs wouldn't have to go to any other Agile meetings or write any business/project planning docs. Manager would scope and write all IC tickets.
I think you would very quickly see a culture change with regard to process work. Having sole responsibility for both creating and completing the process work, it would be in the manager's best interests to make it efficient.
I've never seen an org try the above, but it doesn't seem crazy to me. I expect you'd still be able to attract managers through increased compensation and opportunities for career progression.
Note: I'm not saying the IC forfeits all planning discretion, I simply mean it would be the manager's responsibility to write it up into whatever documentation is required by the org, track progress, rewrite the plan as necessary, etc.
I think this would end up causing negative feelings both for developers and managers. I've almost seen this in one place where we were Waterfall and it created a disconnect between the dev's understanding of the product, resulting in frustration and bad decisions (the small ones that you make during the day where it's important to have context of the product).
The other interesting thing I noticed was that managers had a lot less excuses for the higher ups and would do a lot less office politics and a lot more execution, but that ended up also not working amazingly because we've had instances of people playing only politics and winning vs. execution (surprisingly fast to get promoted when instead of defining tasks and taking care of execution scheduling, you grab a coffee with your manager's manager) and that ended up frustrating managers also.
At this point in my life I've given up on believing in both gajillion dollar corporate projects that mostly satisfy egos of stakeholders and startups that are not customer focused. I've also stopped believing in co-workers that are not user focused. I don't care both for how many mountains a manager can move, neither for the sacred 10x developer, if their talents are not put to use in the customer's name.
And, to your post's subject, I've also stopped believing in any sort of process, that is not done to deliver more value to the customer. IMHO, your vision (or any other vision) could work, or not, depending on the people involved in the process.
I wonder is there a rule or a set of rules that would bring out the best in people, make us focus on creating great products while simultaneously caring about sustainable future.
Profit-centric world seems to breed all kinds of pathology, on every level. Even our growing regulations seem not adequate and usually protecting the powerful.
Eventually the product manager was fired, the engineering manager quit, and the team got reorged into something fairly different.
If you try to ask for permission and write up docs, etc etc then it’s a pain, but if you just _do_ it resistance becomes a lot less.
Just heard a comment from Albert Speer on Hitler, saying, paraphrased, that Hitler's ignorancebof basically everything was his greatest strength. And that it turned into a disaster once this ignorant approach failed, the rules of whatever game was played started to matter and things began to fall appart. Ignoring the Nazi angle, this principle is universal it seems. Success in one field used to assume success in unrelated fields, unsuitable first princule thinking, all that works in start-ups as long as VC money can be thrown around. When that money dries up things turn south.
Why is this inherently bad? Growth and velocity can be beneficial for users.
It's not inherently bad, but if you have velocity and growth build on shoddy foundations, that's not velocity and growth. That's speculation.
> Growth and velocity can be beneficial for users.
100%, with the above mentioned caveat. However, as a user of a lot of products, I find it hard to remember any time where I wished for more growth for the company and velocity for the teams involved in building the products.
A lot of people in tech haven't collaborated with experienced product managers either, or owned little of the financial/growth side of launching something, where these kinds of questions get ingrained into you pretty fast. Much more of a dark art imo is how to act on them without giving your team whiplash and crushing stress (beyond their comment of 'keep innovation teams small', which means fewer people will still suffer the same problem)
There is a similar term in DevOps that I forgot. Hopefully someone can chime in.
First time you have to step in and do something, cool, get in there, get it done.
Second time, sounds like that wasn't a one-off, you have toil, and you need a runbook on the company wiki or internal doc site - that way anyone in the team (or organisation or company), can quickly step in and do that thing.
Third time, have somebody else run through the runbook it, validate it, figure out if it can be improved. Perhaps have some better alarms and operational oversight.
Now, start automating parts of the runbook. Some parts will be easy, some will be hard. Do the easy ones first and update the runbook.
Automation will need alarms with the right thresholds. Think about those carefully and remember whenever an alarm fires there are only three things you should be doing: fix the problem; change the threshold; delete the alarm. If you do anything else - especially ignore it and just look at things and wait to see if anything else happens - you have misunderstood and are heading towards alarm fatigue. Don't do this. Ever. Remember the three possible actions.
Once all the parts of the runbook are automated and alarms are working correctly to trigger automatic resolution, delete the runbook.
You have now eliminated toil.
This can apply to any manual operational task that needs to happen anywhere in a business, but is most easily used when improving resiliency and durability in systems that have easy automatic resolution (such as horizontal scaling by adding more instances when loads are high on existing infra). It's exactly what an AWS EC2 ASG is there to do, for example.
I think that most of us appreciate that sometimes it's just necessary to do this, e.g. because Things Are On Fire, but I think that, amongst the numerous other red-flags that other comments have pointed out, this one particularly sticks out to me.
If you need to grab workers to fight fires "late into the night", that's probably an indicator that your "operating" practices are creating fires that shouldn't have been there in the first place, right?
Highlighting this as a "pro-move" for management was probably not their best idea IMHO.
Toxic organisations always find ways to defend their deep issues as "inevitable".
Google SRE book is now decade old. Read it.
"Things break; that’s life."
Let me reiterate: "inevitable" things are rarely inevitable and most commonly just a poor excuse for poor organization and toxic work practices. Even if you put out pithy quotes.
The way my company does this is with a centralized incident management team that follows the sun across 4 timezones, and specific teams will be paged by them depending on what is broken.
I agree what most people consider inevitable isn't, but your position is too extreme if you've ever been in a real company. Not everything can be predicted by an error budget, and in most non-google companies "stop deploying until you got your budget back" isn't going to allow your business to operate fast enough or flexibly enough. Reliability isn't all in business.
Yes, bad stuff happens - but having it once per two years or once per few months as the article implies it a different scale and the MOST important part of it is to stop treating it as normal, required and necessary.
Also, Stripe has 14 offices globally. That's already a size where you can have work-time rotation for those "late night" problems. Which then stop being late night problems.
Again. Not inevitable.
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
Some helpful insight into why their employees are crying and burning out. Has anyone else's impression of the Collisons been tainted by the realization that they seem to have inculcated psychopathy into their corporate culture?
I think those questions are good. One benefit is that it can reveal potential organisation-wide blockers:
*Getting design changes could be slow (and hence the design department may be understaffed, or inefficient communication channels)
*Maybe some processes can be ran in parallel. Someone can build the login form while the other does the database.
*They might require git access for another department that could take a while to come through.
Considering engineers' time is often the greatest cost for a technology startup, you can bet they'd be incentivised to maximise the efficiency of it.
Not only might they reveal potential blockers, but answering it also helps both parties understand they're on the same page regarding what the task entails. If the task is building a login form and the response is 'It will take two days to train the neural network', you can nip that in the bud.
It's often really helpful for someone to ask these questions, especially when they have higher level context you weren't aware of.
Those aren't bad or uncomfortable questions. Having as much clarity as possible helps ensure plans are on track, need adjusting, etc. right? Some folks stress over questions like this, but I think it's important to ask them, and to provide context to the folks on the receiving end.
So, I think you ask these questions with a few basic rules:
(1) Don't send an email/message with a question like that without being clear about how urgent the response is
(2) Send the question or have the conversation during the course of a "normal" work-day - instead of in the middle of someone's night/etc.
(3) Keep the emotions out of the question, hopefully helping to keep them out of the response
(4) Just always think about what might be involved in answering the question. Are you needlessly grinding someone for an answer or do you really need it?
That last one really hits home for me. Was most recently in analytics/product at a large financial services company (insurance) and am now in a start-up. But I see a common thread across the two: senior folks sometimes asking for answers/work without spending much or any time thinking about the time the receiver will spend trying to figure out that answer/work.
You want stable operations with some slack for the cases something goes wrong, you don't want ops that are constantly under stress.
Having slack and not being stressed are important, but they have nothing to do with asking uncomfortable questions.
Stopping bothering me with damn fool questions would be a good start … ;-)
If you make people talk to you despite being uncomfortable - for example, because you are their manager - they will find a job where they are comfortable.
Potentially your business runs on destroying people. I don’t think that’s a reasonable way we want to run society.
The way /normal/ companies handle this is things like agile retrospectives - what went wrong, how can we fix it. Did we have enough people? How did /management/ fail /the people doing the work/?
If I answer “we would need X more people to get this done tomorrow instead of next week”, does that actually change anything, or do you just respond some nonsense about how you can’t afford that and we should just work harder? Basically: are /you/ willing to make sacrifices for /me/, or is it a one way street?
Nearly every confirmation of this shared here on HN is still self-censored to some degree, but to summarize: yes.
On the other hand, if the manager trusts the employee, and they trust their manager, and they both want to deliver something valuable to their customer, those pieces of advice all look pretty good to me.
I jave to agree on the strategy part so, strategy is such a pleasant way to not do any meaningful work.
It is important to discuss, because many youngsters read these forums and don't know the ways of the world yet.
Not a red flag, just different vocabulary for a different audience.
Hey, I’m just asking questions! We’re just discussing! You’re allowed to disagree!
It’s so funny, and hard to describe, the intentional slight malice in such moments. Like it’s absolutely there, and absolutely veiled and intentionally done in such a way to lend plausible deniability.
In a given situation, everyone knows the “correct” answer, it’s already been telegraphed in any number of ways. Sure, you can disagree. We have an open environment here after all. We value discussion.
Thing is, you’re probably not gonna have a fun time if you do. You’re probably gonna end up wishing you had kept your big yap shut. Strangely enough. “Disagree and commit”
There’s a very disturbing misunderstanding about the difference between consensus and unanimity, made doubly so on projects that actually embrace distributed computing concepts like consensus and unanimity.
- decide what success and failure are
- record your successes and failures
- adjust your priors based on your failures
I have a "what have you changed your mind about" interview question I use on both sides of the table that's really illuminating. The specifics aren't important, but what I'm looking for are the above: what were you aiming for, what did you get instead, how did that experience change you. Hopefully the other party has had lots of these and they've built up an immune system against the various things that caused them to fail, stuff like "test hypotheses", "there is no silver bullet", "everything is a tradeoff" and so on.
Of course, if you then just override everyone every time, they'll soon stop having opinions because they'll realize they don't matter.
instead of asking: "Is there, you think, anything we're doing wrong?"
[1]: https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell
In other words, four words to make people feel good about their role at a company. Middle management is littered with these obscure definitions.
EDIT: Also gone is the value of charitable reading. This is not specific to HN, but I'm just used to holding HN to higher standards than other fora.