This is the reliability engineering equivalent of building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it.
This is the reliability engineering equivalent of building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it.
Not handling failures correctly is a time honored tradition in programming. It is so easy to miss.
For example, how often have you seen a malloc check for `ENOMEM`?
Even though that is something that could be semi common. Even though that's definitely something you might be able to handle. Instead, most code will simply blow chunks when that sort of condition happens. Is the person that wrote it "wrong"? That's debatable.
Some languages like Go make it even trickier to detect that someone forgot to handle an error condition. Nothing obvious in the code review (other than knowledge of the API in question) would get someone senior to catch those sorts of issues.
So the question is, HOW do you catch those problems?
The answer seems obvious to me, you simulate problems in integration tests. What happens when Service X simply disappears. What happens when a server restarts mid communication? Is everything handled or does this cause the apps to go into an non-recoverable mode?
This are all great infrastructure tests that can catch a lot of edge case problems that may have been missed in code reviews. Even better, that sort of infrastructure testing can be generalized and apply to many applications. Making rare events common in an environment makes it a lot easier to catch hard to notice bugs that everyone writes.
It's basically just Fuzz testing but for infrastructure. Fuzz testing has been shown to have a ton of value, infrastructure fuzzing seems like a natural valuable extension of that. Especially when high reliability and low maintenance is something everyone should want.
Consider over-commit. It means that the malloc call will succeed even if there isn't available memory. Instead the system just hopes you wont make use of the memory you asked for. And if you do make use of it, well then the OS might kill you. Just like that. No opportunity for error handling.
When you don't handle it, you get something closer to undefined behaviour instead.
I've never concerned myself with that, but it's apparently a lot more complicated than just doing that check.
See:
https://news.ycombinator.com/item?id=20143277
Some of the thread is hilarious, for instance:
"malloc is not allowed to return a non-null pointer to a memory block that cannot be written to. Linux does it anyway, and in doing so blatantly violates the standard.
...
That is my point, it doesn't fail. Either the kernel finds out a way to map the memory and it succeeds, or it kill the program and the instruction never runs. Code that doesn't run can't violate the standard."
You're right that you'll never be able to cover 100% of all cases, but using that logic, your specs will never test 100% of scenarios, so you shouldn't write specs.
I think the problem is you assumed it was "a magical box that people can rely to cover 100% of test cases of".
That's a poor leap. It's clearly not. It's a good system to test a set of network failures at varying degrees. Just like any engineered system, it needs to be documented what it can and cannot do.
I also eagerly await my do it all, 100% magic box.
Hi Bob, I can't tell you what reports I want or what we'll do with the data until you've first collected the data for analysis. Thanks!
When you do finally figure out why you are in business, you may find that all the data you gathered is worthless. Sometimes it's really innocuous stuff like storing datetime values without enough resolution or accuracy. Only when you know how you plan to use the data can you ensure you are gathering it properly.
Teaching developers to collect all the data they can in hopes of being able to write arbitrary reports down the line is reckless behavior. Hyperscalers love pushing this ideology with inane bullshit like "Datalakes", so you can see where a conflict of interest may arise in the industry.
I've worked on a project (in a peripheral role, fortunately) where the entirety of the CIO's mandate was to "use AI", full stop. So the team was tasked with putting together a data warehouse that would collect random logs and app metrics, and then sprinkle AI fairy dust of their choice onto it to generate Insights(tm).
This kind of thing happens all. the time. in large companies where everybody involved wants either money or bullet points for their resume, and the actual outcome is nearly irrelevant.
I don't see how I could do it differently. I can only truly know which metrics are useful from experience with past incidents, but I can only gain this experience when the metrics were already collected during the past incident.