I agree, I'd "go for it" and worry when it becomes a real problem. If you grow fast enough, then everyone will want to be on there (maybe you could even sell access to the #1 spot) much like Google. If you don't get any traction, then no one will notice or care.

I'd chalk this up as a "good problem to have"... then again, Grooveshark probably thought the same thing: https://news.ycombinator.com/item?id=9468476