Ask HN: Storage for Raw HTML? ElasticSearch or Something Else?

Hi,

I am working on a project where we need to store the response of a web page and have later on do the data processing such as NLP tasks, topic modeling, sentiment analysis etc.

The average content/documents we are expecting is around 1-2 million per day, and soon that will be increasing to 10-20 million.

For this kind of data storage what do you suggest? Is ElasticSearch better for this kind of stuff or should we use HDFS/Ceph etc. for the storage?

Currently, we are using mongodb for persistent storage and ElasticSearch for indexing the data and serving that to our frontend but if some better option is available we can look into that as we are restructuring our most of the architecture and data pipeline.

Any kind of help/suggestion will be appreciated.

Thanks

P.S If someone can share some insights what kind of possible architecture does the brandwatch.com, mention.com or ahrefs.com got for their data pipeline would be really helpful.