“Yes, if”: Iterating on our RFC Process
engineering.squarespace.com
engineering.squarespace.com
I've worked at two companies with RFC processes. At both companies, the processes were -- and I hate to use such a strong word -- a total sham.
I suppose the processes were crafted to look like some egalitarian meritocracy. In reality, it was just a test of who had the most political pull. If you had management on your side, your "RFC" was effectively law and dissenting voices were effectively career suicide.
Which, you know... fine. I understand that choices are made based on cliques and political capital rather than anything else. Cool. That's okay! Sucks sometimes, but that's how the world works. Hopefully your org structure itself is at least something of a meritocracy, so that edicts from on high are of a generally high-enough quality. If they get it right-ish often enough, your org will be okay... probably.
But what really rankled me was the fact that the RFC processes amounted to some kind of elaborate cosplay so that, I guess, folks could pretend that there was some sort of healthy collaborative process. If you're going to let a few "popular and management-blessed" engineers make all the decisions, fine, but don't add insult to injury by pretending otherwise.
I've heard similar things from others, elsewhere. I hope there are at least a few companies dedicated to being something better.
Of course now that the project is behind there's no architect in sight to help us out.
If someone here is a junior and gets "figure it out" from some strategic level engineer, ask professional for clarification or a team-up until you understand. Involve management if the other engineer is uncooperative or fails to help. It's their job to explain slightly more than it's your job to understand.
If all else fails, find an organization that expects the architecture astronauts to be explainers in chief as well.
Then again, if the project is falling behind,
then that is probably more indicative of a
poorly-run product team and/or a bad SDLC process.
Architecture choices directly or indirectly affect every line of code, every minute, every meeting, everything. As you said there are plenty of other places where things can go bad. But it's galling to hear an architect say, "Probably not the architecture, must be something else."
a group of architects that aren't responsible for
delivering anything
The code itself is often somewhat trivial. Effective delivery is all about working about quirks and potholes and traps in the toolchain and architecture inefficiencies.
As we all know software moves fast, and once you've divorced yourself from the reality of actually delivering anything at all, your knowledge quickly decays. You know abstract concepts and how to draw the rectangles on the whiteboard, but before very long you don't know how anything at all actually happens and soon you are unqualified to tell anybody else how to do it.
There can be exceptions to this rule, like projects with an extraordinarily stable toolchain and environment. If you've been writing COBOL at a bank for 40 years you can probably step back into an advisory role and benefit the company by getting yourself out of the trenches and guiding others instead.
Architecture isn't something you can just do on the side when you've got some free time, while still crushing out feature requests and bugfixes 40-50 hours/week.
But.
Architects can't be so detached from shipping code that they have no idea how things are actually made and shipped.
Otherwise you get situations like aloof architecture astronauts casually dreaming up complex distributed or microservice architectures, which look great on the whiteboard, but entail an enormous shift in thinking, tooling, and processes. As well as lots of overhead in general compared to a monolithic architecture. Sometimes that is the right move, but it is a costly one and not a shift to make lightly.
Had another "architect" 20 years ago dream up what was probably the world's dumbest architecture. He demanded that all application layer code be stripped out. We were going to build an entire interactive website with nothing but XML and XSLT. Where conditional logic was required, we could use conditional XSLT <xslt:if> statements. He tried to have us implement a social networking site with that. Had he tried to ship a single bit of code he would have realized it was insane.
Also recently had architecture astronauts push an entirely new primary language on the team. The old one was deprecated. This change was largely championed by people who had never written a line of code in either language with no thought to the tooling changes, process changes, or the overhead of throwing away 10+ years of company-wide language expertise in Language X or the ramp-up time and bruises that would be required to get good at Language Y.
After all, what did they care? They're architects.
One way to avoid/mitigate this is to have architects build and ship proofs-of-concept and reference implementations using the proposed shiny new architecture bits before the rest of the team is expected use and master them. Learn as many of the speed bumps and potholes as possible. They should continue to do this and work closely with scrum teams as those teams build and ship. (That may sound obvious, but lol @ this industry)
But, when the role is poorly constructed, it's a nearly guaranteed mess.
One could say the opposite extreme is just as bad of course. Without architects, you have a bunch of unherded cats. That's also true.
And then you conveniently avoid the architect title!
In general merits (rationality?) can bubble up if it’s framed in correct way for upper levels (eg. risk, profit, time horizon), and can then sway the position.
But yes on average what’s decide at upper levels is gospel. But that’s how corporations function, and that’s ok.
I also think it’s ok for a monarchy as opposed to
democracy ruling what gets approved, assuming merit
based is not an option.
In general merits (rationality?) can bubble up if
it’s framed in correct way for upper levels
Ultimately, business folks must always trust someone with technical expertise, trust is a necessary component of the system. Competent organizations ensure that someone technical has the relevant executive's ear, there are lots of ways to achieve this and it should be able to catch your one-sided RFC if you're proposing anything important.
At the end of the day though, there's no process-based solution for an organization riddled with incompetence. Business must make judgment calls and if your company is staffed with people who consistently make bad calls process will not save you.
Just to share: I've been an outside advisor to a few companies where some developers may think "this is a politics based monarchy and we're not looking at proposals rationally". Where as an outsider it was clear to me that the higher up roles just had more context to make their decisions.
Especially decisions coming from the CTO and VP Engineering kind of roles tended to be driven by input from the wider company strategy, due to daily interaction with the rest of the management team.
In those cases there was a clear need for coaching the higher ups on how to explain and communicate their decisions, but what they were deciding were the right things. And everybody leaving before they improved communications would be on here posting about how bad their higher ups were and what a political mess it was. While that wasn't really the issue.
I'm sure that's not always the case and big political dumpster fires exist (I've seen a few), but there are also many cases where it's only an issue with communicating more clearly from the (technical) management side of things.
And advisor to the CEO in a scale-up where they used to have a quite technical founder as the CEO who left and got replaced by a not so technical outside hire.
I'm also trying to get a few more board advisor kind of roles, but so far everything has been with an emphasis on the tech side of the business.
but there are also many cases where it's only an
issue with communicating more clearly from the
(technical) management side of things.
They communicated well, but the actual thinking was bad.
I'm not sure you are making the case for it here. You are just making some other unrelated point.
Often - not always, but often - the people seeking to sit on these committees are the least likely to be able to actually build software. It is a function of these kinds of boards that will invariably draw those who aren’t so great at coding or architecture.
A bit like being a film critic or book reviewer.
What I read from Squares description in the article is a lot of power uncertainty, which is sadly all too common in the software industry. What comes through repeatedly in the problem description is any uncertainty defaulting to no communication. This problem won't be local to the RFC process, and ideally should be tackled more broadly while also looking to improve it in processes. Communication needs to be rewarded even if the technical content is inaccurate - more people need to speak up more, and with less concern, and receivers need to approach the commentary with a positive attitude to facilitate both not feeling pressure and to encourage ongoing communication. The approvers list is for example doing this, though for a privileged few. What will likely come to light later is that the approvers portion of the process is causing an asymmetry of inclusion, further feeding a lack of communication from other parts of the teams.
An immediate concrete thing I would suggest that builds atop the existing described structure is to invite rotations for folks to join the "council" and review processes regularly but temporarily. This will increase inclusion and dilute exclusivity and prestige. Explicitly open space for participation and solicit input from visiting members, and again it is important to reward communication even if it did not have the desired content.
Oxymoron.
Every society works that way. When a group has to accept a decision, the most powerful person in the group decides.
I don't understand why any educated person today would think meritocracy is a good thing. A child looking at Wikipedia for 10 minutes could see all the flaws in it, yet it keeps being touted as some idealized model of social contract. In reality it keeps white straight rich men in charge.
Like we all agreed this was worth doing all along, right up to this very moment, and now we’re not sure all of a sudden. All the approvals, all the documents written and decisions we went through were apparently not compelling to this person. We could have had this discussion and perhaps killed the idea in the crib, now we’re having this discussion about something we’re about to launch.
In other words the processes are a fig leaf, which as you say is fine, but then it doesn’t even do the job of the leaf.
It is possible for the system to work, but it’s a fragile equilibrium and easily destroyed.
If you already have that kind of empathic culture, then I think it's just about starting a "good enough" process and refining it as you go. If you don't, then you have to build empathy into the team. I found the book Practical Empathy to be helpful. But basically, if you treat someone with empathy, they can't help but start to feel empathy for you. I think we're wired that way. If your team doesn't have a high level of empathy, be the first.
Many people would agree the latter is needed in an organization beyond a certain scale, but I think the former is extremely important and often over looked. It provides a channel for people to be aware of what's happening across teams and build organizational context. It's also a forum for people to express themselves on anything they might have an opinion on and help establish a sort of the zeitgeist for the group. I've seen the alternative where you work within tight silos disconnected from the rest of the org, and it gets pretty dreary.
1. This introduces a lot of process, bureaucracy, and chances of gatekeeping politics. The whole “yes if” mantra just centralizes the political power to the hands of fewer senior leaders by cutting off legitimate blocking criticism from the rank and file.
2. Judge processes by outcomes. What has Squarespace done / shipped / accomplished in the last four or five years? I’m not alone in feeling like it’s a head-scratcher why the company would pay the most senior engineers to erect slow moving bureaucratic decision-by-committee disempowerment structures like this, and then turn around and act like that’s a good thing or constitutes what you’d want to see as productivity from staff or principal engineers.
While I’m a proponent of writing things down, not surprising peer teams, pulling blockers or conflicts forward quickly … I don’t think templates and councils help more than they hurt, and often domain experts in different teams need empowerment to just totally bypass that time-wasting crap and do what needs to be done. If they don’t uphold the responsibility to do due diligence on review or finding conflicts, fire them or take them off the project, but don’t make everyone wear design-by-committee training wheels and act like that’s non-dysfunctional good engineering.
YMMV of course, but I reckon it’s really worth it to ask what is the real engineering output, speed, productivity following from this kind of large process overhead.
I'm not trying to be facetious, but what if the idea really is a No? Not every change is for the better. What if someone proposes "we should rewrite X in Y language," or "we should use Z database instead." Sometimes you need to say "No."
Otherwise you end up with feature creep, endless rewrites and refactoring, code churn, and a bunch of other issues.
So saying "no" isn't forbidden, but it isn't the default either. At least that's my interpretation.
Yes, if you use our current database rather than a new one.
Yes, if 50% of our development team is proficient in Y, X is a bottleneck or EOL, rewrite will solve significant problems.
> we should use Z database instead.
Yes, if Z is widely used in our stack, Z will realize advantages that cannot be achieved with the current database, Z is required to solve upcoming problems.
That's how I take it, at least: You shouldn't say "No," you should say "this is why not."
It’s just about acknowledging the conditions that would make an idea a good one. All ideas are good in a specific context. Instead of assuming everyone’s aware of the current context, state the ideal context for an idea.
More realistically, you're more likely to hit "yes if there is compelling evidence that $choice is better than $status_quo with respect to $criteria". Sometimes that sort of evidence is expensive-to-impossible to demonstrate such that it's basically a "no", but again, it's a much more specific "no".
One of the best questions you can ask someone (including yourself) when in disagreement is “what new data/fact/different condition would convince you of my position?” The “not yet” framing kind of sneaks that question in via just two words! “Not yet” implies you must define the criteria by which it turns to a yes.
Anybody could reject any RFC with the reasons you gave (especially with the vague “bunch of other issues” catch all).
So I’d say, yes, you can say no, if there’s specific, written down, largely agreed upon principles or goals. The thing is, when a no is in reference to specific principles and goals, it’s no longer a no. It’s a “yes if it fits with XYZ. Can you explain how this works because I can’t see it.”
If it’s a no in reference to vague reasons, it’s just gate-keeping. I suppose I can say yes to gate keeping if you can find a way to be encouraging to the RFC author and value the efforts and thinking that went into it. Otherwise gate keeping breeds frustrated complacency.
What if it is yes with vague reasons? Why "yes" should be easier than "no" ? It's potentially extra work and extra code to manage. If something is vague in it's usefulness enough that it doesn't get yes with good reasons it should probably go back to drawing board
However, this situation can also easily arise where the goals and principles are vague, and in that case it's ultimately means that its more of a vague reasons to say no than a vague proposal.
The "it's potentially extra work and extra code to manage" is, imo, another wildcard vague reason to say no. It can be used to deny essentially any proposal anybody makes. It's Machiavellian. In a cooperative teamwork oriented environment, those things are given budgets so there can be some wiggle room for each proposal to progress.
"This would take resources away from delivering X and limit devs ability to collaborate, so we need a way to motivate the budget precedence / growth and a way to afford 50% + 6mo per dev productivity hit around retraining"
"Database x is a VC co toy that risks dying, acquisition by Oracle, and losing customer data when we do not have time for self-inflicted nonsense. We need ways to mitigate the risk such as $10M in NIH DB dev budget in case the likely case happens. We already spent our innovation budget for X, so need more to motivate doing it again on Y."
Less than gate keeping, this is a great way to help mature early/mid career devs / job hoppers who haven't had the chance to live with many self-inflicted senior-level mistakes yet
At work I use a pretty similar template for my design docs. Lately I have started to include a “strategy” section earlier in the design doc, so the first part of the design doc is giving all the background and context then the next section sets out the requirements and goals. Then you give the future vision that everyone can agree to. Then the next section gives a strategy of how to accomplish the vision e.g “we will take incremental steps first” vs “we will re-architect the service.” Then finally the next section is where the design starts, now that we agree on the future vision, requirements and our overall approach to the project we can talk about specific engineering choices and the plan to accomplish those.
I wanted to ask a question to other frontend folks, have you been able to get a good design doc process adopted by your team?
I have had difficulty because a lot of times FEEs think that their projects are too “simple” but then when I look at their code they have made design decisions that we could have discussed much earlier on if they had actually made a design doc. I also see FEEs going down wrong paths and re-learning things themselves that we could have addressed in a design review. What does your process look like for design reviews for Frontend?
The reason I ask is that this process seems to be couched in terms of politics and psychology, and not about organizational value.
What's that old army adage? "An ounce of sweat prevents a gallon of blood."
A major problem however is if there's not a collective understanding that "yes, if" is not a suggestion or consent for that condition to be actualized.
So it really should be "yes, if ..., however, that situation is very rare/cost prohibitive/high risk, so should be avoided"
There needs to be some way to unambiguously signal that something is a no-go, even if there are situations where it would be a good idea.
I do not love the “Yes, if” attitude though. Putting a condition on something really hinders the creative process. You end up designing by committee or rallying behind the “approvers” which makes for bad design in my opinion.
In comedy improv, there is a concept of “yes, and”. It is much more empowering. You accept the reality as is and build on top of it. If the changes get picked up, then it was meant to be. It helps foster trust in each other leading to great organic discovery on the way to being accepted.
"Yesifs" are a handy way to avoid back/forth and shoe trust. They are also good in code reviews, or many gated places. If people use the yes if to do the yes and ignore the if, then there is of course a problem. Some places I have worked I feel like the team would have done that because of culture.