Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.
"We have Best-Effort and Guaranteed-Delivery APIs"
"I want Guaranteed Delivery!!!"
"If the GD logging interface is offline or slow, you'll take downtime; is that okay?"
"NO NO Must not take downtime!"
"If you need it logged, and can't log it, what do you do?"
These days I just point to the CAP theorem and suggest that logging is the same as any other distributed system. Because there's a wikipedia article with a triangle and the word "theorem" people seem to accept that.
[edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]
What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.
People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?"
And don't get me started on "log" messages that are 300mb events. Per log. Sigh.
Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway.
It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.
I think you've completely misunderstood me then. I haven't taken a stance at all on what should be done. I'm only trying to agree with the grandparent poster about logging ultimately reflecting CAP Theorem.
For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.
To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage.
Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine.
(The audit logs might also end up in one's normal logs too, for convenience.)
Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant.
I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.
Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".
In a network partition, you are either offline or your data is not consistent.
If you're writing local to your system, you're losing data if there's a single device failure.
So -- a single system collecting a log event -- it is not reliable (guaranteed) if written just to some device on that system. Instances can be de-provisioned (and logs lost), filesystems or databases can be scrambled, badguys can encrypt your data, etc.
In this context, a "network partition" prevents consistency (data not written to reliable media) or prevents availability (won't accept new requests until their activity can be logged reliably).
If you define "reliably" differently, you may have a different interpretation of log consistency.
Availability is achieved if at least one writer acknowledges a write. In a partition, it means when you have multiple parts of the system disagreeing about the write contents due to a partition in the network. But because logs are immutable and write only, this doesn’t happen in any situation. The only situation this might occur is if you’re maintaining a distributed ratchet with in delivery order semantics rather than eventually consistent temporal semantics- in which case you will never have CAP. But that’s an insanely rare edge case.
Note CAP doesn’t ensure perfect durability. I feel like you’re confusing consistency with durability. Consistency means after I’ve durably written something all nodes agree on read it’s been written. Since logs don’t support read on the online data plane this is trivially not an issue. Any write acknowledgment is sufficient.
No it doesn't. Read your own wiki link.
> In this context, a "network partition" prevents consistency (data not written to reliable media) or prevents availability (won't accept new requests until their activity can be logged reliably).
A network partition doesn't matter for a log system because there is no way to have consistency issues with logs. Even a single partitioned-off instance can accept writes without causing any problem.
Of course if you cannot connect to any instance of your log service then you cannot write logs. But that's got nothing to do with the CAP theorem.
Promtail[1] can do this if you're using Loki for logging.
[1] https://grafana.com/docs/loki/latest/send-data/promtail/
a) If your service crashes and it's in-memory, you lose logs
b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block
In other words it's good enough.
> [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]
i read GD as god-damned :-)
You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last transistor. Does anyone actually do that? Verify the program and the hardware to prove that it will never run out of memory for logs and such?
Continuing this thought, logs are probably generated endlessly, so either whoever wants them has to also guarantee that that they are processedand disposed of right after being logged... or there is a finite ammount of log messages that can be stored (arbitrary number like 10 000) but the user (of logs) has to guarantee that they will take the "mail" out of the box sooner than it overfills (at some predictable, deterministic rate). So really that means even if OUR system is mathematically perfect, we're just making the downtime someone elses problem - namely, the consumer of the infinite logs.
That, or we guarantee that the final resources of our self-contained, verified system will last longer than the finite shelf life of the system as a whole (like maybe 5 years for another arbitrary number)
From a software point of view, this system is unlikely to exist as it doesn't matter that the cause of your downtime is "something else that isn't our system". As a result, you're gonna end up requiring infinite reliable storage to upkeep your promises.
E.g., a service I wrote wrote logs to an ELK setup; we logged over TCP. But the logging was async: we didn't wait for logs to make it to ELK, and if the logging services went down, we just queued up logs locally. (To a point; at some point, the buffer fills up, and logs were discarded. The process would make a note of this if it happened, locally.)
I agree, when stuff is properly written. I don't remember the exact details, but at least with UDP the asyncness is built-in: there is no backpressure whatsoever. So poorly written software can just send udp to heart's end.
Reminds me a litte of the oldtimers trick of adding a sleep(1000) somewhere so they could later come back and have some resources later, or if they needed a quick win with the client.
Now cloud companies are using malloc(300000000) it to fake resource usage. /s