For ES and solr gurus here what is the recommended max size for documents if you want to index lot of office documents?
I run a large ES index covering 80TB of data, and regularly index documents as large as 20MB. At that size, the bottleneck is often transferring that much data over a network to the ES cluster. You need to make sure your HTTP client can handle it, and the network has enough bandwidth. Elasticsearch itself is not really the bottleneck.
If there are separate sections in the office documents that you can pull out and index as separate fields then you should do that. For example, if you were indexing patents, you would want to index abstracts and claims into separate fields.
What is the realistic expectation here. Should we say 50 mb. How everybody else do?
At my last job, we used to index large documents on Solr. The largest chunks of them were indexed in non-stored fields. Which means they were searchable but you couldn't retrieve the actual text. This drastically cut down on the index size and the resources we needed to support it.
Then after scoring you'd return the top hits to the main application as IDs, which could retrieve everything needed (full text included) along with all the other information they'd retrieve for the document and generate the actual output.