Hacker News Reader

Comment by simonw | Hacker News Reader

Since the work all happens in your browser, the only victim of a long complicated query would be you own browse and the S3 bandwidth bill of the person hosting the database (since you'd end up sucking down a lot of data).

But if you want to jack up someone's S3 bandwidth bill there are already easier ways to do it!

View on HN

It's a static file, CDN it.

Many CDNs cannot cache 43GB files.

Cloudflare's limit is 10GB, 20GB for Cloudfront, 150GB for Akamai and Azure.

You can chunk the file into e.g. 2MB chunks. The CDN can then cache all or the most commonly used ones. That's what I did in the original blog post to be able to host it on GitHub Pages.

I would look into caching range requests. Simpler than pre-chunking or caching the whole database.

It should be trivial to split it up into 1GB chunks or whatever. In fact, if you only request single pages, you could split the database up into pagesize-sized files. This is a lot of files, but at least it avoids the need to do range requests at all. You probably want to increase the pagesize a bit though (e.g. maybe 256 KiB instead of the default 4 KiB?)

Range requests are the star, so I would vote keep them? I mean, technically, the CDN could represent each 4kb block of the file as an individual URL, so that you do range requests by filename instead of by .. range request .. but at some point I definitely think RRs are more sane.

Probably both, compute file name and relative offset from the original offset. (A minor complication is requests crossing file boundaries, in which case you have to do two requests instead of one.)

Isn't Cloudflare's limit 512 MB? (except for enterprise plan which has no price defined where it's 5GB)

https://support.cloudflare.com/hc/en-us/articles/200172516-U...

TIL, for whatever it's worth :) Thanks!

If I get the essence of what you are saying... "the only victims would be both ends of the communication"? ;)

What is said is this site isn’t any more susceptible to DOS than any other site. Of all the ways of architecting a site this is probably the least vulnerable.

Reply on news.ycombinator.com