Why has free software failed to be a commercial success?
Your premise is flawed, and this entire question is nonsensical. F/OSS has been able to compete with proprietary software, and - in many cases - has won. Just because it hasn't won every segment doesn't mean that it's been a failure.
Ever bought a smart tv like the Sony Bravia or basically any of the nice Samsung TVs? Linux as well. How about a home router? Many of those are Linux. I'll go out on a limb and say there are a lot more Linux devices than Windows total (primarily due to the plethora of embedded and mobile).
I feel it's because the time and effort isn't put into actually making it user-friendly. Look at any apple product, for instance. It's so easy and intuitive and I don't need to play around with the command line to get something to work.
I use Linux and tons of open source software every day for running my business. I've tried to use pretty much every desktop version of Linux..but I always switch back to windows or mac. Why? It never feels finished to me. Random crashing, error messages that don't make sense, a horrible UI (that looks like a programming just threw it together with no thought).
There needs to be a movement of Open source UI and graphic designers.