In my experience, you need to care about latency. That affects user experience. It's quite hard to pay for better latency.
If you don't fit in the budget for the specific task, your product features don't matter much.
Like I said, it works until it doesn't, and then you do have to optimize for performance to some extent.