Hacker News Reader

Comment by tlb | Hacker News Reader

The question I had is answered by this line of code:

  xhr.setRequestHeader("Range", "bytes=" + from + "-" + to);

I am a little surprised you can just do that. In https://github.com/phiresky/sql.js-httpvfs/blob/master/src/l...

View on HN

Range headers are a pretty standard tools to e.g. continue interrupted downloads and similar.

Any well designed system, especially if it has static sources and is server cached should support it.

Surprisingly many web-frameworks don't support it out of the box, or don't support it well.

Either way gh-pages are static content and probably with some server side regional caches, so I'm not surprised it works.

Range headers are also how DownloadThemAll works.

If you're pulling a single TCP stream across a crowded network, you get maybe 1/40th of the available bandwidth. If you do four range requests instead, you might see >3/40th of the bandwidth.

This place we were contracting at, the managers were widely rumored to stream sports games at their desks, and our release cycle happened to fall on a game day. My poor coworker was downloading our installer every week, and the ETA was over 40 minutes. "Are you using DTA?" "No, what's that?" <fiddle fiddle> ETA: 12 minutes.

12 minute pauses in the middle of a manual process are a lot easier to stomach than 40 minutes. Especially if something goes wrong and you have to do it twice.

I've learnt about this by using https://www.biodalliance.org

It's an embedded genome viewer, you can just point it at a multigigabyte reference files and .seg files and it loads super quick

Here is direct link to GitHub with the usage: https://github.com/dasmoth/dalliance/search?q=bytes%3D%27&ty...

One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.

I'm surprised that works, iirc pdf isn't defined in order and can't be parsed streaming

I think that's what the "optimize for Web" checkbox does

Wow so it actually does something! I wish programs would use such vague descriptions. (Or more of them had helpful instant tooltips.)

Linearized (a.k.a. web optimized) PDF files only help for displaying the first page quickly. The rest of the file is still in pretty much random access order.

For what it's worth, there's a format for content like pdfs that's optimized for viewing on screens (as opposed to printing), djvu: https://en.m.wikipedia.org/wiki/DjVu

Oh, wow initial release 1998,now I'm feeling a bit old...

Linearised PDF files are defined in order and can be read from the start.

https://blog.idrsolutions.com/2010/02/linearized-pdf-files/

Also .mp4 files. The format is designed for seekability, and browsers take advantage of this.

Progressive JPEGs work well for this too, so you could have the same file used for a tiny thumbnail and large preview and full sized photo by sending different range requests. However you need to know how many bytes to request.

I'm surprised this isn't used on mobile browsers to lower data usage. I'm sure with a little research you could figure out what a good mapping from pixel size to byte size should be to give good enough results.

A browser doesn't have enough information to use this optimization. At the point where it's about to request an image, it doesn't know how large the resource will be, whether it'll be a progressive JPEG, or even whether it'll be a JPEG at all. Making range requests blindly would probably be a net loss -- for every progressive JPEG that the browser managed to save some time on, it'd have to make follow-up requests for many more non-progressive JPEGs, non-JPEG images, and progressive JPEGs which it didn't get enough data from on the first try.

You are correct.

However, one could use this approach: download as usual, and in a streaming fashion process the data and if it's a progressive JPEG, you can close the connection before you have received everything; and then you can cache the prefix and later download the rest if needed.

Fast clients will just swallow the whole file, while slow clients would be able to benefit from it.

It wouldn't work for pipelined HTTP connections though without cancelling the whole pipeline, so maybe not a very practical solution given the performance benefit that already gives. And HTTP/2 maybe doesn't support cancelling a transfer either, so.. ?

Maybe a direct "Accept" or "Prefer" header to indicate that it's enough to send just something useful for an icon would be a more ideal solution, but it would require server-side support.

You can even point VLC at a .iso file on a web server, and seek around in it.

I remember putting together a sloppy http range implementation that initially only supported single ranges, it had quite the explosive effect on adobe reader when it didn't get the expected response to its multi-range requests :)

Not all webservers support/enable it, so YMMV.

But as long as you're dealing with a known server that does, then gravy!

nginx enables it by default. Another ingenious use of range request is zsync, it allows you to diff compressed binaries on a remote with local ones, so that you only have to download what has changed on an update. AppImage uses this

> Not all webservers support/enable it

Could you provide an example of server that does not?

AFAIK, Range is supported by all major CDNs, so not supporting it in web server would be a death knell for it's real-world adoption.

Depends on what’s being served. Any decent static file server should support it, but if the content is at all dynamically produced then the authors would have to think to implement it and rarely do.

I can't think of a specific one, but a decent proportion (maybe a quarter?) of HTTP downloads I attempt don't support resuming a partial download, therefore don't support Range. (I.e. resuming always starts from the beginning.)

I would assume this is often because the site in question isn't using Apache etc. to serve a file directly, but is either essentially proxying it to some custom-built file serving service, or a script that processes/authenticates the file in some way, and they just never bothered to implement Range.

I guess "all" that's needed for write would be a webdav server with support for PATCH x-update-range? :)

https://tools.ietf.org/html/rfc5789

https://sabre.io/dav/http-patch/

Unfortunately, solid, stand-alone webdav servers are harder to come by than decent http2/1.1 servers.

Reply on news.ycombinator.com