Metics tell you where and when something when wrong. Logs tell you why.
However, a logging framework, which is generally lossy, and has the lowest level of priority in terms of deliverability is not an audit mechanism. especially as nowhere are ACLs or verifiability is mentioned. How do they prove that those logs originates from that machine?
If you're going to have an audit mechanism, some generic logging framework is almost certainly a bad fit.
Even at very immature organizations, log data within a service is usually structured.
Even in my personal projects if I'm doing anything parallel structured logging is the first helper function I write. I don't think I'm unrepresentative here.
> Even at very immature organizations, log data within a service is usually structured.
unless the framework provides it by default, I've never seen this actually happen in real life. Sure I've seen a lot of custom telegraf configs, status end points and the like, but never actual working structured logging.
When I have seen structure logs, each team did it differently, The "ontology" was different (protip: if you're ever discussing ontology in logging then you might as well scream and run away.)
actually, how many of your messages include the time and date, and how many different ways of displaying timestamp exist in those messages?
That is why I say logs are unstructured, because all but a very few places actually have the discipline to enforce a single log structure.
Until you need them, the you'd trade anything to get them. Logs are like backups, you don't need them most of the times, but when you need them, you really need them.
On the flip side, the tendency is to over-log "just in case". A good compromise is to allocate a per-project storage budget for logs with log expiration, and let the ones close to the coal-face figure out how they use their allocation.
In the loosest possible sense. Binance is an organization that pretended it doesn't have any physical location in any jurisdiction. Its founder is currently in jail in the United States.
Use metrics & SLOs to help diagnose the health of your systems. Derive those directly from logs/traces, keep a sample of the raw data, and now you can point any alert to the sampled data to help go about understanding a client-facing issue.
But, for auditing of a particular transaction, you don't need full indexing of the events? You need a transactional journal for every account/user, likely with a well-defined schema to describe successful changes and failed attempts. Perhaps these come from the same stream of data as the observability tooling, but I can only imagine it must be a much smaller subset of the 100PB that you can avoid doing full inverse indexes on this, because your search pattern is simply answering "what happened to this transaction?"
Sounds like a row in a database to me.
Dumb question, but is that how structured log systems are implemented?