That's the conventional wisdom, but I'm not sure I believe it. It takes (much) longer for an app's usability to mature without some central arbiter, but "you need" it is far too strong a statement.
I think the concept of designing for usability has just been slower to enter the FOSS world. Only now are open source projects starting to grapple with how you do interaction design on a open source project. But throwing up our hands and assuming it will never happen just because it doesn't always happen right now is silly. How about we start recruiting non-developers to open source projects to run simple usability tests and report on their findings? It's not that developers will never work on usability, it's just that it's a difficult and foreign field to most of us, so we avoid the issue altogether.
That being said, my knowledge is not exhaustive... maybe there are examples that I am not aware of.