Build Observable Systems

WHY NOT BOTH? (OP here. I figured the prominent company branding was sufficient disclaimer that yes, we do have a product to sell; but we built the product because we see a better way of doing things, rather than vice versa.)

TL;DR: use probabilistic logging (x% chance of logging, with x chosen by dev for each log statement) instead of importance levels (ERROR, WARN, INFO, etc) to keep common events from flooding log files. Is that it?

Basically. But if your volume is high enough to have this problem, you're probably past the point where humans reading logs is a reasonable use of time, so you need machines to help humans consume the logs (e.g. to produce summary statistics and graphs).

So as well as probabilistic logging, annotate the logs you do emit with the probability of emission. That way the machines consuming the logs can continue to account for the logs that were dropped.