Harder to say when to quit. I suppose you quit when you can't take it anymore. Which means deciding that depends less on the product than on you.
Fast forward to now, had I have made that move, it would have literally cost me thousands - as the apps are really gaining traction. I blogged about the book sharing website twice and it seemed to generate little interest.
My criteria would be: a) If you aren't getting ANY traction in a predetermined period b) when you talk/write/tweet/blog/speak about your idea or product and very little seem interested, it might be time to quit or pivot. My 2 cents.
A good idea with low diversification costs
A great idea with high diversification costs
We're undergoing a massive internal change that's pretty much akin to starting an entirely new business within a business, simply because what we're doing has less potential growth than what we're looking at doing, which is new (which is rare in our industry).