How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020
codeascraft.com
codeascraft.com
It was overall just lackluster and the TL;DR can be entirely summed up as, more than doubled traffic without issue (and no technical insights really).
A code semi-freeze for weeks is incredibly alarming. How much unremediated tech debt must be causing that?
That is a massive red flag that Etsy engineering culture must be something like the SRE tail wagging the product engineering dog. I would bet that you only get rewarded at Etsy for being a visible firefighter, stopping failures on nights or weekends, jumping in front of outage bullets, and that if you take the time to design systems robust enough and safe enough that they don’t need code freezes in the first place, you are never going to get recognized or rewarded.
It’s baffling that not only did they have to resort to a long code freeze - but they even approved a public blog post bragging about it.
Boy, that tells you a lot about the culture. Not going to be seeking jobs there any time soon.
To put in context, I work for an ecommerce company that routinely sees 10-15x traffic around Black Friday -> Cyber Monday, and around Christmas to New Year. We don’t do any code freezes for the load, almost all of which is handled through auto-scaling in our on-prem data centers.
If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on at least checkout-path deployments during holiday prep and through holiday proper. It isn't a tech debt thing, it's just good business sense: the time of year when people give you by far the most money is the time of year when you least want to risk causing problems for people who want to do so. The problem isn't untrustworthy systems or bad engineering culture; the problem is that anything short of perfection has a measurable impact on revenue, and humans achieve perfection with less than perfect reliability.
Even where I work, that revenue impact can easily amount to a major problem if we let a significant bug slip through. People buy our product for its own sake, and will do so from Amazon or Walmart or wherever if they can't get it direct - but our margin on sales through second-party vendors is nothing like as good as on direct sales, so we still take a hit. For Etsy, it's much worse; on sales through other storefronts, they get nothing. Too, a bad enough break during holiday will also risk some fraction of their subscriber revenue from sellers who, fed up at losing out on what would otherwise also be their highest revenue of the year, might go elsewhere.
It's easy to talk shit about tech debt and bad engineering culture, I get that. It's also poorly founded in fact, and evinces an apparently questionable grasp on some fundamentals of the business, besides. I don't know whether you have represented yourself well or poorly by creating this impression, but it is the impression you've created, and if that bothers you then I might suggest trying to do otherwise next time.
That is a whole mess of giant neon red flags that something is wrong here regarding engineering for resilience and trusting your deployment system to catch errors.
Additionally, for a freeze lasting weeks, unfreezing and resuming merges from some backlog of frozen changes would likely introduce far greater risk to revenue or to customers than managing a trustworthy deployment workflow during high traffic events. I would even question the fundamental premise that this is safer or beneficial to customers at all. It seems much, much more likely to be a “cover your ass” method to prevent any possible blowback from outages due to unremediated tech debt that the weak leadership is not capable of protecting as a worthwhile priority for quarterly goals.
We don’t do code freezes except in very special, isolated cases of known failure risk. When we do need an isolated code freeze for one system, that is a “all hands on deck, this had better get fixed” kind of moment. If the larger system of most of our core services needed an extended code freeze to be safe, that would result in probably (justifiably) firing senior engineering leaders.
If you have a CI/CD system and you can’t trust normal automated deploys at any time, that is a huge problem. If this happened one year and you needed the code freeze, so be it - that’s just responsible risk assessment. If you intentionally plan the system to work this way every year that is egregiously bad engineering leadership.