Personally, I think they should have gone with AOSP and leveraged Qt: by the end of 2010, they'd already gotten their two operating systems (Symbian and MeeGo) to a point where it wasn't a lot of work to recompile applications for one to run on the other. (N.B.: I don't know how much of this work ever actually made it out of Nokia. I was working there in 2010, when all of this was going down.) Given that Qt already ran on Android, IIRC, it might not have been that much work to do the same thing on top of AOSP, producing a Nokia-specific release that could run Android software and provide a clear path forward for Symbian users and developers. I was a little shocked that they didn't release a Qt compatibility layer for Windows Phone, instead choosing to pretty much leave their existing developer community on that burning platform.
The Lumias did ship too late, but I think the previous commenter who referred to the "Osborne Effect" has it right; a Nokia that had appeared firmly committed to Symbian and forging a bridge from the existing to the forthcoming generation might have fared considerably better. The Nokia N8 and N9 should have been supported as flagships rather than sort of apologetically shoved out the door with "DEAD PHONE WALKING" written on the boxes. On a practical level, this might not have saved them (especially if they'd stuck with Windows Phone, which it turned out the market really hadn't been waiting for after all), but it might have given them a fighting chance.
I don't think Nokia would have tried to downscale itself either way, by the way -- they'd have kept their network equipment division, and might have even kept their mapping division. They'd balanced that with consumer mobile hardware for years before that, after all; I don't see why they couldn't have kept doing that with Android devices.