Hacker News Reader

Comment by redman25 | Hacker News Reader

That's one of the main reasons I'm leary about them. Such a big f-up is difficult to forget. It shows that they have a move fast and break things culture which for a company that is responsible for critical infrastructure feels wrong.

View on HN

In response to this incident Cloudflare has made big engineering changes, including huge work to move away from C as much as possible.

The offending parsers were rewritten in Rust (https://github.com/cloudflare/lol-html), as well as WAF, image optimization, and a few others. Nginx is being replaced with a custom cache server.

New implementations are using either the Workers platform, or are written in Rust or Golang.

Memory safety doesn't fix fundamental design flaws.

This is an empty tautology. You have no insight into the actual design, so I presume your fundamental design flaw is the CDN existing.

I interviewed there once and they asked me what I would do if a service broke after a deployment. I said the first step was to revert to the last known good version and then investigate. Color me surprised when that was not the answer they expected.

That's strange. What was the "correct" answer?

If I ever interview at Cloudflare and get this question I might answer with "call the sales team and have them fix it by selling someone an enterprise subscription paid upfront by the decade" just to see if the interviewers read Hacker News :P

They wanted me to roll out a fix first. Apparently rolling back first was not moving fast enough for the interviewer’s liking.

Depending on the service’s criticality, the cost of rolling back versus pushing a fix, service dependencies in their environment… having to push a fix might have been the better approach.

Without more details about the environment, it is a 50/50 call.

Was the correct answer related to cache invalidation?

Cloudflare's internal release tool suggests revert when monitoring detects failures during deployment, so this question doesn't describe Cloudflare's practices. There must have been something more to it, or it was a misunderstanding.

Reply on news.ycombinator.com