However, if you're selling a physical product, and/or a service that integrates deeply with third party products/services, it becomes a lot more difficult to determine what's even worth measuring. A conservative approach to metrics collection will limit the usefulness of the metrics, for obvious reasons. A "kitchen sink" approach will take you right back to the same "data volume" problem you had with logs, but now your developers have to deal with more friction when creating diagnostics. Neither extreme is desirable, and finding the middle ground would require information that you simply don't have.
On a related note, one approach I've found useful (at a certain scale) is to shove metrics inside of the logs themselves. Put a machine-readable suffix on your human-readable log messages. The resulting system requires no more infrastructure than what your logs are already using, and you get a reliable timeline of when certain metrics appear vs. when certain log messages appear.