This way of working really requires a small company, (and is one of the reasons I think small places have a chance of outsized impact). But at bigger places, either a manager is being judged on a team outcome -> where an individual not on topic is budget not going towards what they get paid to achieve. Or you need seniority enough to work directly with a project director (but most structures expect people at that level to have leadership responsibilities as well).

Unfortunately, while people that work like this can be exceptional, big projects run on organised measurable work. I have found few places big enough for a specific "Manager" role to be flexible enough to get a good match to tasks.