I think you are partly right, but I also think the mechanical advantage of the bicycle, even the velocipede, was significantly underestimated. You say it saves "a bit of energy going flat or downhill" but some quick calculations suggest it is twice as energy efficient just balancing along, and 3 to 4 times more efficient pedalling. These advantages aren't always realised, especially over shorter distances, but I would have thought they would be quite clear over any longer distances, or when carrying a heavy load. Perhaps this is where initial unreliability became an issue though, as the advantage of saving energy on a long journey bt bicycle would be outweighed if it was prone to breaking down, given it would be unlikely to find someone who could repair in those early days.

I think a fundamental inability to intuitively grasp the potential mechanical efficiency of the concept must have been a major barrier though.