Hacker News Reader

Comment by JohannesKauf | Hacker News Reader

The problem was: Doing rate limiting on the application level was not enough. Once the request hit my backend the incoming bandwidth was already consumed — and I was charged for it.

I contacted Vercel's Support to block that specific IP address but unfortunately they weren't helpful.

View on HN

So you're probably still vulnerable to this even with the key requirement, but they stopped once you removed the incentive? Did you notice what they were scraping?

Sorry, I mixed up a few topics here:

- Moved everything to a VPS - way better value for money. Extra TB of traffic only costs €1-10 with Hetzner/DigitalOcean compared to 400€ with Vercel's old pricing.

- Put Cloudflare in front - gives me an extra layer of control (if I ever need it)

- Built a proper REST API - now there's an official way to use the converter programmatically

- Made email registration mandatory for API keys - lets me reach out before having to block anyone

That other server was probably running a scraper and then converting the html-websites to markdown. After about 2 weeks they noticed that I was just returning garbage and it stopped :)

Ah! Makes sense now, thanks for sharing.

I've had good success with Cloudflare's free-tier features for rate limiting. If you haven't tried it, it only takes a couple minutes to enable and should be pretty set-and-forget for your API.

Reply on news.ycombinator.com