> has throughput limited
The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat.
> relatively expensive
Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways.
> slow to query in bulk
Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2.
> slapping grafana
I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.