Ask HN: When to scale out of VPS?

For a text based site(mostly), what is a good load time(universally). I have written a RoR app and its taking somewhere between 100ms to 300ms to load (it fluctuates) and I was wondering if thats good or bad. My guess is its bad.

My setup is Linode VPS ($20 one). If anyone can come up with a number (may be a bit of guess), how many users can I support with this kind of response time until I have think about scaling it further. My guess is 30% of users will visit the site once or twice daily.

I know this question is vague, so if someone can share their stories like when they started out, what was their setup and what was the point when they had to think about scaling up. Few stories will give me some hint to get an answer.