Hacker News Reader

Comment by CharlieDigital | Hacker News Reader

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

View on HN

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.

Thank you for the insight. It's very easy to prescribe simple solutions when we are oblivious to the actual problems being solved.

But, under the assumption that the problem are indeed AI crawlers, of the things you listed only the search would be under increased load.

I am sure their pages are not entirely static either. The APIs are used by researchers and AI companies too. Also with search you end up having people trying to use it for RAG with AI. I have dealt with this all and there is no one dead simple solution to deal with things. AI crawlers are one part of things, but they also have increasing submissions, have to deal with AI generated spam papers, and all sorts of stuff. There's always this feeling here on HN that oh it is dead simple you just do "X" as if the people that are dealing with it don't know that.

All these services can be throttled to deal with AI. I don't see this as a justification. The idea that a service like arXiv should be run as a startup is, simply put, foolish.

pardon me but cloudflare workers seem better for this approach.

If we can get for the fact that we require javascript to run it, aside from that. Cloudflare workers is literally the best single thing to happen at least to me. With a single domain, I have done so many personal projects for problems I found interesting and I built so many projects for literally free, no Credit card. No worries whatsoever.

I might ditch writing other languages for server based like golang even though I like golang more just because cloudflare workers exists.

I too am impressed by Cloudflare Workers’ potential.

However Workers supports WASM so you don’t necessarily have to switch to JavaScript to use it.

I wrote some Rust code that I run in Cloudflare Functions, which is a layer on top of Cloudflare Workers which also supports WASM. I wrote up the gory details if you’re interested:

https://127.io/2024/11/16/generating-opengraph-image-cards-f...

JavaScript is most definitely the path of least resistance but it’s not the only way.

they already use fastly.

https://blog.arxiv.org/2023/12/18/faster-arxiv-with-fastly/

Reply on news.ycombinator.com