We (the bacalhau.org[0] project) are interested in helping with this - one of our philosophies has been that part of the problem is with that first step. By first moving to a lake of some kind, you end up giving up lots of optionality. Even basic things like aggregation, filtering, windowing, etc now need to be in the "locked-in" tool, which is exactly the wrong first step to take.
SHOW HN: We have a solution that uses DuckDB to do some of this initial work[1] which can save you 70%+ or more on total data throughput. Further, it allows you to do interesting things like eventing, multi-homing observability data, etc.
I'd be very interested to hear any/all thoughts!
[0] https://github.com/bacalhau-project/bacalhau
[1] https://blog.bacalhau.org/p/bacalhau-x-duckdb-deploying-appl...
Disclosure: I co-founded the Bacalhau project.