Hacker News Reader

Comment by woodruffw | Hacker News Reader

Only partially related, but since they mention it at the bottom of the post: I really like GH's "brownout" approach to discontinuing services. They've done it a couple of times before (like this one[1]), and it's always struck me as a very sensible way to handle deprecations on such a massive service.

[1]: https://github.blog/changelog/2021-08-10-brownout-notice-api...

View on HN

Make sure to get the rate right. Like 1/1,000 or 1/100,000. Poetry did a brownout of its install script at 5% failure. With a matrix of builds user's CI had closer to a 100% chance of failing.

These are also known as scream tests. We use them internally at $dayjob as part of EOL'ing APIs. Are they not common?

I have no clue! I just appreciated them as a lowly user.

Lol brown outs is such a nicer name, but I've always known them as a scream test.

They're ever so slightly different.

Scream test is to discontinue service entirely and see who screams, akin to pulling the plug on a server in a rack when you've got no idea what's running on it, to then plug in back in when you find out it's the finance team running payroll software.

Brown outs are typically short-lived purposeful errors or aborts to either shed load to prevent a failure due to capacity exhaustion or cause issues that are investigated by a human to find the service has been sunsetted which raise awareness that something needs to be fixed before it's totally broken.

We started using feature flags to disable, rather than enable capabilities. Turn them off and see if anyone notices. Ramp up the rate. If you identify some cohort that needs the feature, keep it on for them, but isolate them until you figure out a migration plan to close whatever gap you may have (or at least some way to massage unhappy customers if you aren't going to support some legacy thing).

Reply on news.ycombinator.com