Hacker News Reader

Comment by JohannesKauf | Hacker News Reader

Thanks, Jina actually looks quite nice for use in LLMs.

I also provide a REST API [1] that you can use for free (within limits). However you have get an API Key by registering with Github (see reason below).

---

The demo was previously hosted on Vercel. Someone misused the demo and send ~5 million requests per day. And would not stop — which quickly brought me over the bandwidth limits of Vercel. And bandwidth is really really expensive!

So that is the reason for requiring API Keys and hosting it on a VPS… Lessons learned!

[1] https://html-to-markdown.com/api

View on HN

Seems pretty risky to not implement rate limits either way.

The problem was: Doing rate limiting on the application level was not enough. Once the request hit my backend the incoming bandwidth was already consumed — and I was charged for it.

I contacted Vercel's Support to block that specific IP address but unfortunately they weren't helpful.

So you're probably still vulnerable to this even with the key requirement, but they stopped once you removed the incentive? Did you notice what they were scraping?

Sorry, I mixed up a few topics here:

- Moved everything to a VPS - way better value for money. Extra TB of traffic only costs €1-10 with Hetzner/DigitalOcean compared to 400€ with Vercel's old pricing.

- Put Cloudflare in front - gives me an extra layer of control (if I ever need it)

- Built a proper REST API - now there's an official way to use the converter programmatically

- Made email registration mandatory for API keys - lets me reach out before having to block anyone

That other server was probably running a scraper and then converting the html-websites to markdown. After about 2 weeks they noticed that I was just returning garbage and it stopped :)

Ah! Makes sense now, thanks for sharing.

I've had good success with Cloudflare's free-tier features for rate limiting. If you haven't tried it, it only takes a couple minutes to enable and should be pretty set-and-forget for your API.

Reply on news.ycombinator.com