Controller route prefixing seems suspicious. Do you include every controller at runtime and use php reflection with some regular expressions to parse all of their prefixes only to figure out what single route to use?
In deployment mode that routing tree is serialized and cached. Only a single (matching) controller is then touched. Unserializing the data structure is extremely fast and finding a match is roughly an O(log n) operation.