What does Parquet do for your problems? I am trying to learn for when I might need it?
https://blog.openbridge.com/how-to-be-a-hero-with-powerful-p... https://stackoverflow.com/a/36831549
And with projects like Apache Iceberg you can also have ACID transactions which will make it easy to update or delete just using SQL. This really opens up the separation of compute and data and you can use any engine you want (spark, drill, hive, impala, athena, dremio, redshift spectrum etc) on top of your files.