How to move fast and not break things as a remote company
jam.dev
jam.dev
> When developers pick up tickets that don't have all the information they need, they need to spend more cycles investigating and waiting for follow up information from the broader team.
This could also be interpreted as adding layers of friction between the people experiencing problems and the people fixing those problems. This generally leads to worse outcomes.
> After product spec and kick-off, the next step is for engineering to spend a little bit of time writing a technical plan.
Oh good, a spec tossed over the fence in yet another meeting, and then another spec before any lines of code are written. No thanks.
> Having milestones put on the calendar also gives us the flexibility to move them when needed, but makes it so that we consciously need to decide to move them, rather than letting dates slip by.
The dates slip by because software estimation is akin to the Halting Problem. You aren't fixing that, just making your managers feel better.
> Having a daily check in on the project status helps us recognize when we are drifting off track and need to make decisions about whether to change the date or re-prioritize what's in scope.
Even more meetings, hooray!
> All of the tips above are things we started doing as a direct result of something one of us suggested in a retro. Retros are awesome, and are so important.
And more meetings about what went wrong in the prior meetings.
FAANG doesn't do this. Hire great people, trust them, communicate async frequently, coordinate planning infrequently. This reads like a manifesto for "we hired a bunch of subpar developers and don't give them any ownership or empowerment". Pass.
I work at a high performing remote-first company. This is how we do it. Ownership where I work includes an engineer, a TPM, sometimes an exec or manager. The gist is that there's individual ownership over:
- Technical outcomes
- Business outcomes
- Customer outcomes
- Integration outcomes
Sometimes it can take us a while to agree on direction, but that's okay. For the (potential) slowness with respect to direction, it makes identifying, prioritizing, and fixing bugs much more simple. It also speeds up incident response quite a bit.
FAANG doesn't do what? All of these are what I've done at FAANG companies and on multiple teams.
Hoo boy. Reality is so much more complicated.
When you hire someone, you think that they're great, you hope that your estimation of them proves out, but it's not really possible to know. Even people who come with great portfolios, great recommendations, the fact of the matter is that your company has its own culture and its own people in different roles, and it's simply not possible to know with 100% certainty that people will truly fit into your organization until you place them into their new role and see how things play out.
Furthermore, nobody comes in with full expertise on every level of the stack. Nobody is an expert also in UX, also in your frontend stack, also in your backend stack, also in your datastores, also in security, also in compliance concerns, also in observability. You must build a collaborative culture if you want to ensure that all concerns are handled.
You can build a culture where you trust the people you hire in general. But "trust" is not a euphemism for "let people go play cowboy from their desk in the corner and it doesn't matter that we don't know what they're doing because we trust them." The key foundation of that trust is knowing in which area the organization trusts you to be proficient and, in turn, trusting others in the organization with expertise in other areas, at least until a "great" hire proves to be unworthy of that trust.
> more meetings... don't give them any ownership or empowerment
You can do both. You can build a culture where you split a typical eight hour workday for individual contributors into no more than 2 hours (i.e. usually less) in meetings, 4 hours for planned issues discussed in those meetings, at least 2 hours (i.e. when meetings take less than 2 hours) for unplanned concerns which may include any number of the following: planned issues taking longer than planned, production outages, talking to customers, working on stuff the individual contributor thinks is important ("ownership/empowerment"), etc.
Hire people who are arrogant enough in their area of expertise that they will loudly and energetically share that strength with others, and humble enough to listen to others with expertise in other areas.
within 1 month tops its completely obvious who is pulling their weight and who isn't. Even without micromanagement or any kind of management at all really, it just becomes obvious for newhires after a few weeks.
Most successful companies don't have fire-fast policies. Companies with a fire-fast culture tend to be cut-throat, where nobody remembers your wins, only your screw-ups; employees are incentivized to blame others and cover their ass in order to avoid the Angel of Death. Nobody wants to work at a place like that, at least over the long term.
They also have ridiculous interview practices that have nothing to do with the work you will be doing.
It's not automatically good just because FAANG does it.
If you have a random spec tossed over to your side you absolutely have to translate all the garbage to technical terms first, not doing that just leads to all your developers constantly asking WTH they’re actually supposed to build.
That said, this whole story does sound exactly like what we’re doing at our F500 enterprise. I very much doubt it makes anything more agile.
This is so hard to get teams to do. I am almost always the slowest to implement a change simply because I don't conflate design and implementation.
At first glance it looks like I get less done, but when you account for the fact that nobody has to clean up after me or shoehorn a work around for an edge case in I become one of the most efficient.
Assigning cost to subsequent follow up tasks is difficult as well. Both technically in terms of aggregating/grouping as well as interpersonally because people get defensive or discouraged when you're consistently pointing out shoddy work.
That's so awesome.
The Jam dev team recently shipped a major refactor of all the state handling in the app a few weeks ahead of schedule and that's because the team spent so much time upfront documenting and planning the changes. Thanks to Tony Stuck for showing us how important doing upfront planning is on the Jam engineering team!
To me that doesn't sound like a thing to be proud of.
1. You approved a plan to have a refactor that lasts several weeks. Even for big companies that's usually an anti-pattern. It even contradicts your own advice of "make things better as you go".
2. You tremendously overestimated the required effort (i.e. absolutely failed at planning)
3. You got lucky this time and two wrongs made a right.
Also, your example suffers from survivorship bias. Of course everybody has examples where planning paid off. But there are also plenty of examples where planning was a waste of time, or where people were following outdated plan for the sake of following the plan. "More planning" is a bad advice. A good advice helps distinguish areas that benefit from "more planning" from areas that will suffer from it. But that is hard and nuanced.
Sounds more like engineers managed to communicate why it was worth doing a refactor to the stakeholders, made a rough estimate, and then through a solid round of planning managed to find out how to reign it in a bit. Sounds like something to be proud of to me.
I've been working at an Agile(TM) "prefer communication to processes" shop for a while now.
One thing that they absolutely do _not_ understand is that sometimes the smallest unit of work cannot and can never fit into a one (or even two) week Sprint.
If someone comes and says "This project will take a long time.", yeah, sure, sit down and have them explain why. Do _not_ shoot them down as a matter of course because "Well, nothing should take more than a week, and _definitely_ never than two." and tell them "Go mutilate your plan so that you have a Real Deliverable every five working days.".
> You tremendously overestimated the required effort...
a) Neither of us know how long the entire project ran, so "A few weeks" could be a small overestimate, and evidence that a sane amount of cushion was baked into the planning.
b) It's way, way, way better to overestimate a fair bit and come in under that estimate than it is to underestimate and _require_ either rescheduling a whole bunch of shit, or mutilating a solid plan in order to shove it out in the original timeframe.
If we're talking about web development - yeah, pretty much. IME, most projects where team lead was not willing to "mutilate" were a disaster.
> Neither of us know how long the entire project ran, so "A few weeks" could be a small overestimate
Are you saying changing state handling (whatever this means) in a young startup could take several months? If that's the case, they definitely should not be teaching how to move fast.
It seems like you agree that my core statement ("sometimes the smallest unit of work cannot and can never fit into a one (or even two) week Sprint") is correct, but are unwilling to explicitly say so.
> Are you saying changing state handling...
I'm saying that neither you nor I have the details required to understand the minimum time required for the project.
You've said yourself that you have experience with tasks that have justifiably run longer than a work-week. So, it's clear that -given more information about the task- you could be convinced that the time estimate is a reasonable estimate of the time required.
Move fast and break things
Way easier to achieve in a small organization.
edit: Did a bit of digging and I may be wrong here. In an interview in 2014 [0], when Facebook was changing their motto to "move fast with stable infrastructure", Zuckerberg said
"As developers, moving quickly was so important, we would even tolerate a few bugs to do it..."
0: https://www.cnet.com/tech/mobile/zuckerberg-move-fast-and-br...
Too many orgs don't move fast because of the fear of breaking things.
2. Bug fixing has diminishing returns and exponential costs.
If you accept these two axioms, Zuck's quote makes perfect sense.
And certain bugs are simply not worth fixing because cost of fixing it is higher than cost of having it.
Just another google spam blog post to drive traffic.
Btw, if your team goal is "ship to production" (as per screenshot), you're probably not moving fast enough.
It also doesn't tell us what "fast" is at Jam. We only know that it's calendar-driven development with quality gates.
I find this to rarely be true in practice. It's not that comments are intrinsically useless (though they do suffer from the intrinsic limitations that are not statically checked or tested), but that most developers tend to write comments for things that benefit from them. Or, when they do need them, they are poorly written or too terse to be useful to anyone else. I would venture to say I've seen maybe or two comments in my 15 years of software engineering that I saw and actually thought "wow, well this helps me understand this a lot better."
In the last year alone I’d estimate comments written by my past self have saved me dozens of hours of effort tracking down the reason why code was implemented a certain way. A couple times I’ve undertaken some refactoring that would’ve taken hours, only to find 10 minutes into the process a comment to the effect of “don’t try to refactor in that way, for these reasons” and it was correct.
Maybe my memory is just particularly bad. But if the comments are helpful to me, they’d also be helpful to someone else who didn’t write that code.
It does take practice to know when and how to write those comments though.
> But if the comments are helpful to me, they’d also be helpful to someone else who didn’t write that code.
this does not necessarily follow unless you've been extremely thoughtful in how you wrote those comments (which you may very well have been). That is, in fact, the issue most of the time — that comments are written in such a way that they serve to jog the memory of the person who wrote them, but are rarely of general use to anyone else.
But for the rest, it's exactly as you described. There is usually some workaround/shortcut being taken or there is interfacing with an external dependency that does not work as expected. In those cases comments make aware of things that the code doesn't show and can save a looot of time.
Does daily project status seem excessive? Development can be non-linear, so I wonder what decisions are needed to made on a daily basis.
But there are definitely teams where members are poor written communicators, and in such cases, there's nearly always something new that comes up in a daily.
Expect 1 year to migrate to new services.
But that's another story, because "If it works in production, don't touch it".
Not breaking things is only half of story.
* Make your software observable. Your feature is not finished until you've verified it is being used as expected. Ideally you should have all of the data you need to know if things are going well without having to wait for customers to email you with complaints. Be careful not to develop a culture of developers throwing code "over the fence", either to QA staff or a release team - developers own their code from IDE to production.
If you don't develop this culture of ownership you end up with longer feedback loops between customer support, QA and dev.
100%, that's great
In terms of implementation, if it isn't on a dashboard or doesn't fire off an alert, it doesn't exist. So put your telemetry and logs on a dashboard, and set up alerts based on your SLA's/SLO's.
Also, make it easy for developers to understand, create and edit dashboards/alerts. If it's hard to understand or hard to do, it won't get done.
That may well change during development, but new work doesn't get approved without at least a first stab at how we're going to monitor it.
That was an important implied Principle of Steve McConnell's famous Software Quality At Top Speed article[0]. In fact, if you read that article, a number of these things resonate.
Not everything good is new. I just think that we forgot them, along the way.
For myself, I generally operate as a one-man shop, so I have the following tips, that work for me:
* Modular Design
I write my stuff in discrete modules, if possible. Sometimes, the modules are source code files, organized by functional realms, other times, they are complete, soup-to-nuts, published and supported SPM modules. The iOS app that I'm writing now, has this current dependency manifest:
* KeychainSwift: 20.0.0
* LGV_Cleantime: 1.3.6
* LGV_MeetingSDK: 2.3.0
* LGV_UICleantime: 1.1.4
* RVS_AutofillTextField: 1.3.1
* RVS_BasicGCDTimer: 1.5.2
* RVS_Checkbox: 1.2.2
* RVS_GeneralObserver 1.1.1
* RVS_Generic_Swift_Toolbox: 1.11.0
* RVS_MaskButton: 1.2.3
* RVS_Persistent_Prefs: 1.3.2"
* RVS_UIKit_Toolbox: 1.3.2
* White Dragon SDK: 3.4.2
I also wrote a couple of backend servers, that the app leverages.
* Layered Design
I write stuff in layers; especially server code. Each layer has a loose coupling with the other layers, and also maintains a fairly strict functional domain. This makes it very easy (and less bug prone) to pivot and fix.
* Document As I Go
I write my documentation into the code. I've learned to keep it mostly to headerdoc-style documentation. I write about my process here[1].
I also practice "Forensic Design Documentation"[2], and "Evolutionary Design Specifications"[3]. These help me to move quite quickly, yet maintain a well-structured, well-tested project.
* Test Harnesses over Unit Tests
I find that test harnesses[4] allow me the greatest flexibility, and fastest path to integration testing, which is where I want to be.
* Reduce "Concrete Galoshes"[5]
This is pretty important, and many of the above practices go a long way towards this goal.
I know that the way I do things won't work for many folks; especially teams of less-experienced developers. My experience, coupled with an OCD nature, make it work for me.
WFM. YMMV.
[0] https://stevemcconnell.com/articles/software-quality-at-top-... (It would be great, if he fixed the images in that article, one day).
[1] https://littlegreenviper.com/miscellany/leaving-a-legacy/
[2] https://littlegreenviper.com/miscellany/forensic-design-docu...
[3] https://littlegreenviper.com/miscellany/evolutionary-design-...
[4] https://littlegreenviper.com/miscellany/testing-harness-vs-u...
[5] https://littlegreenviper.com/miscellany/concrete-galoshes/