OK - so we are not crazy.
The official response seems to be: set reasoning to the lowest and service_tier to priority, and it should work as fast as GPT-4o. But that is not the case.
They are probably still tuning it. Or maybe GPT-5 is not meant to be replacement for GPT-4o / GPT-4.1. Or this is going to be a new normal (slowest).