This sounds so naive for someone with 20+ years in the field...
Linux, for decades, couldn't get logging to the point that it at least doesn't lose messages (the problem with tail / logrotate that is quite obvious once you think about it, but it took many years to give up the approach).
I recently hit a bug where NVidia's driver abuses Linux kernel logging in some tight loop by spamming log messages at insane speed (happens when you have two video adapters, Intel and NVidia and an external monitor). An interesting side-effect here is that Linux logging tries to throttle loggers who output too much, so, from the log you cannot tell what's happening (because even though the system is burning calories trying to print a tonne of messages, nothing really gets printed).
Several iterations ago I worked on a product where logging had to be implemented as writes to shared memory self-styled circular buffer, and because there was too much info printed too quickly you only had few seconds worth of logs before system crash... on a good day.
Needless to mention the fun of stitching together logs coming from different places in your system with separate clocks.
Even simply processing hundreds of Gigabytes of logs on its own isn't a trivial task.
----
Many things are simple, when your task is simple. Logging is just one of those things.