Hacker News Reader

Comment by dfabulich | Hacker News Reader

That's entirely right. Products have to transition from fast-moving exploratory products to boring infrastructure. We have different goals and expectations for an ecommerce web app vs. a database, or a database vs. the software controlling an insulin pump.

Having said that, at this point, Cloudflare's core DDOS-protection proxy should now be built more like an insulin pump than like a web app. This thing needs to never go down worldwide, much more than it needs to ship a new feature fast.

View on HN

Precisely. This is key infrastructure we're talking about not some kind of webshop.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever.

An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

an insulin pump is a good metaphor; insulin as a hormone has a lot of interactions and the pump itself, if not wanting to unalive its user, should (most do not) account for external variables, such as: exercise, heart rate, sickness, etc. these variables are left for the user to deal with, and in this case, is a subpar experience in managing a condition.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contingency planning that they embody, failure modes of each and every part in the pump were taken into consideration, and there are more such parts that you are most likely aware of. This includes material defects, defects as a result from abuse, wear & tear, parts being simply out of spec and so on.

To see this as the typical firmware that ships with say a calculator or a watch is to diminish the accomplishment considerably.

I had a former coworker who moved from the medical device industry to similar-to-cloudflare-web software. While he had some appreciation for the validation and intense QA they did (they didn't use formal methods, just heavy QA and deep specs), it became very clear to him very clearly that those approaches don't work with speed-of-release as a concern (his development cycles were annual, not weekly or daily). And they absolutely don't work in contexts where user-abuse or reactivity are necessary. The contexts are just totally different.

It is perfectly possible to engineer for faster cycles without losing control over what your code can and can not do. It is harder, for sure. But I do not think it is a matter of this absolutely not working, that's black-and-white and it never is black and white, it is always some shade of gray.

For instance: validating a configuration before loading it is fairly standard practice, as are smoke tests and gradual roll-outs. Configuration fuck-ups are fairly common so you engineer with that in mind.

Thanks for spelling that out. It's so often tempting to be reductionist about things, but there is often a tremendous amount of thankless engineering inside products that we are privileged to consider as being somewhat boring. It takes a lot of work to make something so dynamic and life-critical and make it reliable enough to be considered simple, when it's anything but.

If humans beings had a small chance to transform into say quadrupeds or suddenly grow tenatacles, extra hearts, organs, etc., in any given year… then wouldn’t designing a safe insulin pump literally be impossible?

Compared to what they are already doing it would be marginally more difficult.

The point still stands. The human body still isn't going change. That's why insulin pump can afford to have all kinds of rigorous engineering, while web-facing infrastructure on the other hand needs to be able to quickly adapt to changes.

> That's why insulin pump can afford to have all kinds of rigorous engineering, while web-facing infrastructure on the other hand needs to be able to quickly adapt to changes.

The only reason we have a web in the first place is because of rigorous engineering. The whole thing was meant to be decentralized, if you're going to purposefully centralize a critical feature you are not going to get away with 'oh we need to quickly adapt to changes so let's abandon rigor'.

That's just irresponsible. In that case we'd be better off without CF. And I don't see CF arguing this, in fact I'm pretty sure that CF would be more than happy to expend the extra cycles so maybe stop attempting to make them look bad?

All the more reason to be careful about relying on humans to avoid making mistakes when changing it rather than moving quickly and letting things fail in production.

Indeed. I was trying to make that point on my concluding paragraph.

> This thing needs to never go down worldwide

Quantity introduce a quality all of its own in terms of maintenance.

Reply on news.ycombinator.com