I guess startups that provide utilities (I can't figure out a better word, but think Github/Gitlab/CircleCI) should charge as early as possible, whereas startups focus on user contents can remain free for a long period of time, and then think about how to get users to pay (think of Tumblr's paid themes, although really unsuccessful from my perspective).
Everyone takes a different path and some just won't last no matter which approach to take.
Free users are basically visitors and focusing on them early on is a distraction (focus on what isn't interesting to your paid users) - and if there is a group of people willing to pay, they will most likely have common expectations for what the product should be.
I'm wondering whether it would be better to find a smaller number of people willing to pay for the product and really tailor the experience to them.
It's kind of like the idea of focusing on fewer people who love you, vs lots of people who just like you, where love = paying customers :)
Maybe. Or they could just be outliers who happened to be willing to pay for your product early on. You want to be open to customer feedback of course but it's also about your vision for the product.