You're right about UX being central to the problem, but you're missing a major part of it here.
UX is hard. Getting great user experience takes IMHO a lot more pain than getting algorithms and systems to work right (in most cases). UX is about endless revisions, a lot of boring and often ugly code, obsessing about pixels and quirky details of your UI's "story," and other things that are very time consuming, not fun, and that the vast majority of programmers hate doing.
As a result, people generally have to be paid to work on UX.
This is doubly true because most nerds don't need great UX. They know how things work and can use "raw" hacker projects from the command line. So there's really no motive in most cases. "Works for me."
Until and unless there is an economic model for a less centralized less winner-take-all Internet economy, vertically integrated closed silos will continue to dominate if for no other reason than the fact that they can pay people to do the boring tedious work that separates a nerd project from an app non-nerds want to use.
Edit: It's really always been about UX to some extent. Personal computing won back in the 70s and 80s because it offered far superior user experience to closed silo mainframes. But the Internet and especially mobile have changed the game pretty dramatically. Now closed silos are winning because they offer superior user experience.