There is no secret about the benchmark, its all open source: https://github.com/wolfgarbe/LuceneBench
So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches.
Could you share the reasons, MMapDirectory did not work for you?
The Lucene's own benchmarks are at: https://home.apache.org/~mikemccand/lucenebench/ , though I admit to not know enough about benchmarking to form a strong opinion.
Either way, good luck with the project/service. More competition is always great. The open-source components look interesting too.