> the cost of monitoring an application can easily exceed the cost of hosting the application even for simple applications
(especially if you are looking at saas monitoring for a single-box or serverless thing)
I suspect SIEM is the most established use case for ingesting logs to something actionable; wonder if there are concepts there which port generally.
Datomic's approach to change capture seemed very good when it came out -- I wish this were a thing I could cheaply turn on and off on SQL columns. As things are, every ORM seems to resist having a table with no primary key so 'log important things to the DB' is hard in practice.
In postgres, mvcc makes it expensive to edit a scalar, so it is impractical to store counts. (There is an old uber blogpost I think about why they use mysql for this use case).
Specifically for alerting, I wish I could define application logic statistics in my application rather than PromQL / datadog json. Would be much easier to express 'warn me if this ever fails', 'expect 50% failure', 'warn me if too many retries' type stuff.
On the 'looking at logs' side, the hosted monitoring tools I've used are often uncomfortably slow, even when my data is small. I understand it's a hard problem but I miss graphite / whisper.
(not even getting into mixpanel / posthog type product-level observability)