Yeah, one of the most limiting factors imo about phones-as-consoles is actually getting the game onto a larger display. They're more than powerful enough to run lower-spec console games - as evidenced by the fact that, say, remastered Square Enix titles have also been hitting mobile platforms.

But the Switch, being a dedicated system for this, doesn't have to put nearly as much work into the implementation. You can easily just throw it into a dock and leave it there indefinitely, because you don't need it for anything else. While you can cast a phone wirelessly, the setup will depend largely on your ecosystem and/or accessory vendors. And even then, the entire thing is typically mirrored - which not only leaves the rest of your device otherwise unusable, but also raises further cases to handle around notifications (or worse, calls) while you're actually gaming on it.

These issues would make single-player gaming annoying enough, and local multiplayer effectively becomes a nonstarter. (Never mind that one you introduce that as a requirement you also introduce the need to set up additional controllers a nontrivial amount of the time, and fast pair support within the Bluetooth spec is just as messy.)

For phones to really be able to cover these things without introducing tradeoffs that just don't exist with a dedicated device, we'd need OS- and hardware- level support for running a secondary display, wirelessly, without also introducing lag in the process.

Effectively, we're comparing them to the wrong platform. Because while the internals most directly line up with those of the Switch, the interface design they'd need in order to really make the experience work is arguably a lot closer to that of the Wii U.

[0] Note that, for the sake of this discussion, on Android's side I'm referring strictly to what's available in AOSP. I'm aware of, say, DeX, but the app ecosystem as a whole doesn't tend to have great support for vendor-specific implementations, and the stock desktop mode is still a. experimental, b. wired.