Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be.
By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves
By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves
Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.