Ironically I just did this using Go and a 32 core machine processing ~50 compressed gigs per day [1]
[1] https://thinkfaster.co/2017/05/big-data-with-golang-instead-...
[1] https://thinkfaster.co/2017/05/big-data-with-golang-instead-...
1.4 billion of our connection logs (24 fields) takes up 89G on my clickhouse VM. 5 billion records would take ~320G.
Based on http://tech.marksblogg.com/benchmarks.html a 6-node ds2.8xlarge redshift cluster is about as fast as clickhouse on a single i5.