1. A world where every human's smartphone is an open-field install-anything no-controls-beyond-antivirus device similar to your desktop PC would be a functionally and utilitarian-ly worse world than the one we live in today where these devices for most people exhibit strong, centralized, corporatist control.
2. There are use-cases these devices are now being adopted for that open-field install-anything desktop PCs have never, even to this day, adopted. You cannot install your drivers license and passport into your desktop PC, nor can you tap-to-pay. Its likely many of these use-cases needed the level of hyper-security Apple and Google are pushing toward in order to digitize these use-cases, validly or not.
3. Apple's extreme of restricting the installation of anything outside of the App Store (and, for that matter, even severely restricting the things you can distribute through the app store for no reason, such as until recently alternate payment providers) is a step too far. As you say, the opposite extreme is bad, but that doesn't mean Apple's extreme is good.
4. There's a middleground we need to find, and by the way, I don't think Android strikes the middleground very well today. A couple examples of things that would move in a more positive direction toward this middleground: (a.) I think phones should be able to be purchased from the factory immutably with the quality of requiring binaries to be signed by Apple/Google. Google should sell Pixels that are hyper-locked down similar to the iPhone and that characteristic can never change about them; its etched into the security coprocessor itself. Conversely, maybe if I have an Apple developer account, I should be able to buy an iPhone that allows me to install binaries from any source. (b.) Apple should have an "App Store Extended" backend capability where developers still distribute their apps through the App Store, all the same security scanning happens, but the developer has to handle their own marketing via the web; the app never appears in the App Store App itself. In exchange, their distribution rules are more relaxed (alternate payment processors, applets, sensitive content, etc).