At my last company, we used kafka as our log to handle the incoming messages and then we just forwarded everything to Elasticsearch with logstash (super easy to create and read from new topics).
For your storage, Elasticsearch can store the source of all documents and basically act as a persistent DB (with kafka as backup for replay). Use well-defined mappings for the data you know is structured (logs, forum posts, already schematized data etc) and dump the rest into their own indices and lets ES' implicit mappings do their work. You can use dynamic templates to drill down any specific things you want to transform, such as converting dates, not tokenizing certain strings and so on.
I've done a lot of search and text extraction work and this is my go to for the past three years.