Would it speed up prompt processing if you increased the -ub (and -b) parameters.
-b, --batch-size N logical maximum batch size (default: 2048)
-ub, --ubatch-size N physical maximum batch size (default: 512)
I also tried double the default. that also means that they can be left to default settings, apparently.