Hacker News Reader

Comment by avidiax | Hacker News Reader

There's a corollary:

   Even if you explicitly deny a guarantee of a certain behavior in your contract,
   if you usually deliver that behavior,
   most of your customers will depend on it.

Some examples:

If you make a queueing system, it's impossible to guarantee anything other than delivery "at most once" (some loss occurs), or "at least once" (some duplication occurs), but if you usually provide "exactly once" in practice, most of your customers will depend on this.

If you provide a data bucket service, and guarantee availability, but not performance, and you usually provide 100MB/s throughput, your customers will have major problems if you only provide 10MB/s throughput in some cases.

If you make a self-driving car, and it requires human monitoring, but it's really good, say one intervention per year of driving . . . your customers will die because they aren't paying attention.

View on HN

“Die normative Kraft des Faktischen” or “the normative force of the factual” is a thing and usually not seen as necessarily bad.

It recognizes that legitimacy often emerges organically from social acceptance rather than top-down imposition. In technology we often see that evolving reference implementations work better than elaborate specifications.

In its form as ‘the normalisation of deviance’ it’s generally recognised as bad.

If I'm not mistaken, CPython's dict preserved insertion order as an implementation detail at first, but because too many users came to rely on it, it was made part of the language specification starting in Python 3.7.

I don't think that's quite right. CPython dicts only started preserving insertion order in 3.6. There were concerns that other Python implementations might not be able to do this or it would be a performance bottleneck, but by Python 3.7 these were satisfied enough to make it part of the spec.

> If you make a queueing system, it's impossible to guarantee anything other than delivery "at most once" (some loss occurs), or "at least once" (some duplication occurs), but if you usually provide "exactly once" in practice, most of your customers will depend on this.

That's only a condition at termination. For ongoing communication, you can guarantee exactly once delivery. When communication ceases, the final state between the ends is indeterminate. If you can keep talking, or resume after breaks, it's a solvable problem.

That's true, but it doesn't help the customers.

In a large system, terminations (both planned and unplanned) happen all the time. For the unplanned ones, it is very difficult to ensure exactly once, at least from the perspective of the queueing service, who can't check that the processing of the message did or did not occur outside of the original connection and it's "acks".

> That's true, but it doesn't help the customers.

Sure it does. It's essential to making banking work. See ISO 20022 reconcilation.[1] This is the process by which errors in money transactions between banks are resolved. ISO 20022 is used by SWIFT, FedNow, FedWire, and systems in about 70 countries.

Unique IDs make transactions idempotent, but that's just the start. Banking is an eventually consistent system. Messages go back and forth until both ends agree. If there's a loss of communication, some transactions will be resubmitted, but they won't be applied twice.

Banks have been doing reconciliation for centuries. It's been sped up recently, but the concepts have not changed much.

[1] https://www.finextra.com/blogposting/19808/how-exactly-will-...

You're not contradicting the post you're replying to; you're describing common workarounds to approximate exactly-once semantics using at-least-once semantics on the message queue combined with external storage on the reader side.

This is explicitly recognized in contract law: course of performance / dealing is a factor courts will consider in evaluating the nature of a deal. (Most contracts will try and carve it out).

Can't disagree with anything you said, though I think there are steps to address at least some of them: for queueing systems, testing with a chaos monkey isn't a bad idea... you'd want a test environment representative of production workloads, which is hard to do, but anything should be better than nothing.

In the self-driving car scenario, you'd probably go with cold statistics: is it killing fewer people than ones that need more interventions? Just like queueing though, experiments in production could be problematic.

> In the self-driving car scenario, you'd probably go with cold statistics

No. There is a big difference in an accident caused by human error and an accident caused by machine failure.

We tolerate much more of the former than the latter.

This feels like a cognitive failure, but I do not think it is

You can look at airport security as an example. Bombs and guns are quite rare in carry-on luggage. It would be far too boring for most operators, which would mean that they tend to tune out of their screening job.

So what the x-ray interface does, is randomly insert guns and bombs into the scan at a relatively frequent rate. The operator must click on these problem areas. If it is a synthetic object, it then disappears and the operator can continue screening the bag. If it isn't synthetic, the bag gets shunted for manual inspection.

So for a self-driving car, if it must be monitored (it's not L5 within the driving mission), then you would perhaps need the car to randomly ask the driver to take over, even though it's unnecessary. Or randomly appear to be making a mistake, to see if the user reacts.

If the user doesn't react appropriately or in time, then self-driving is disabled for a time the next time the car starts.

For the queuing system, it perhaps makes sense to inject a certain number of duplicates by default. Say 0.1%. Enough that it simply can't be ignored during development of the clients. Then, when duplicates arise as a consequence of system failures, the code is already expecting this and there's no harm to the workload.

For software, you can often inject the "bad" data only during test time or in dev mode. E.g. a lot of Maps don't guarantee interaction order but in practice it's insertion order or alphabetical. I always feel a bit of sheudenfrued when someone gets bit by this and I get to tell them told you! It's in the spec! But anyway you can randomize the order at a performance loss during testing to prevent people depending on it. We do this, and for i18n strings too (which can of course change if something is retranslated or the format is modified)

Amusingly, Golang randomizes map iteration to prevent this.

Recency bias is how nuclear reactors go bad and people slip past security. There just hasn't been an incident in so long that we become complacent.

I thought Netflix was nuts for Chaos Monkey, but having read several more treatises on human cognition and particularly cognitive biases, I now see they are crazy like a fox. Guaranteeing something breaks every week keeps things at the mental forefront.

Scarily true:

Users assume a system is always correct if it has been mostly correct before.

This isn't a new phenomenon, although AI certainly brings it to the fore.

We ran into this at SNS (AWS) all the time.

Reply on news.ycombinator.com