>It currently processes more than a thousand sentences per second
For a well known web service, this sounds severely underwhelming. Is that for a single node or overall?
The next part of that sentence says it is horizontally scalable, which I would say is probably per instance or thread. I doubt the problem parallelizes well.