I’m not sure why this would be true, unless you’re willing to cut back significantly on features. Google already spends quite a bit of effort on performance improvements, and most manufacturers aren’t exactly known for being able to produce high-quality software.
There have been a couple projects that have tried to make phones based on pure Linux (Librem, Pinephone). Their performance is limited a lot by not having good CPU options, but it doesn’t seem like they’ve been able to achieve much performance improvement on the software side either.