The only concern that I think is preventing growth (Other than production of course) - is the concern about long term reliability.
If you buy a camry or prius you know almost for a fact, that your going to drive 200k miles without a problem, and if there is one, its recognized as an engineering failure and the dealership fixes it for free - and fast.
But with Tesla, there are lots of rumors that the queue for getting stuff fixed it really long, and that after warranty expires, really expensive.
What I am really scared of is another "BMW" style reliability experience, which I would consider to be a catastrophic failure on the part of Tesla.
Edit - also accidents "caused" by "autopilot" software, and liability is still a big question.