Hacker News Reader

Comment by srvmshr | Hacker News Reader

Genuine question: We signed up for the deci/decahose last month and for now making everyday dumps of about 50GB in flat files.

What kind of system architecture would be good to load & search through such dataset. We tried to explore Mongo Atlas but it is coming out very expensive. Other alternative is to throw away most metadata & just keep ID & Tweet.

View on HN

Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be.

By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves

The access comes with a cost and is pretty expensive unless covered by some academic grants. (Its about 10 grand USD). How expensive would Clickhouse be for data increasing about 1-2 TB per month? We are trying to optimize costs as the Twitter purchase already seems steep

If you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns.

Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.

Interesting. Would you mind talking offline (email in my bio). Would love to take some pointers

If you don't mind a proprietary hosted service, try Google BigQuery.

Typical use case for Presto or BigQuery. Putting it in S3 and using Athena wouldn't take very long.

Clickhouse will churn through it.

Reply on news.ycombinator.com