Yes that's true, that's the main limitation for me at the moment, and I hope with more people using it and with time we'll get enough apps, like what happened for Linux on desktop. Meanwhile to compensate on Linux mobile we have access to many programs from desktop Linux - not adapted for mobile screen, and many people use Android emulation - which should work very well on official SailfishOS but has limitations on other distros. If I recall correctly I think it might also be possible to have a dual boot Android/Linux for some devices.
Personally I have them separate, an Android phone and a Linux phone.
Sadly the Android one is still my main driver, mainly because I chose a not-fully-ready distribution like PureOS on a Librem5, and the battery duration is still too limited for me.
Regarding the thriving software systems of Apple and Google, I think is more due to the popularity of mobile phones than to specific limitations they made on their OSes. I think even on a very open OS, as far as there is a good "famous" framework for development, you can gain the same amount of software.