The tricky part - just as with logging - is to figure out what metric make sense and give you signal and what metrics are mostly noise.
Sometimes you can even extract metrics from logs by looking for specific log lines if you know what to expect.
This looks backwards from the app dev's perspective.
- Metrics are strictly structured, documented data. They are a first-class output.
- "Strictly structured logs" is contradiction in terms. These are just events to be sent to a pub/sub, also a first-class output.
- Logs are unstructured, unknown, bleeding-edge, grey-area data, which can potentially become either metrics, or events, or bug tickets.
If I output data as metrics, I don't output it again as logs - that'd be just noise.
But of course other apps/systems can have a wholly different approach and force people to invest a lot into sorting out that dumpster fire. The leverage is on the producer side.