Hacker News Reader

Comment by geofft | Hacker News Reader

From a technical point of view, what is DownThemAll? It seems like it looks at the structure of the current web page, identifies all links, and then downloads them (in some cases using range requests, as with '90s-era download accelerators), with options for pausing and resuming downloads and renaming them in certain ways.

What prevents this from being done using a Chrome extension to look at the page structure and render some UI, plus a bit of native code using the native messaging API to actually store the files on disk?

https://developer.chrome.com/extensions/nativeMessaging

"Developer frustration" is a more-than-valid reason, but I'm trying to understand if this is a claim that no software like DownThemAll can possibly be written without Mozilla introducing purpose-built extension APIs.

View on HN

The main reason people install it I thought is because

1. It allows downloads to start and stop. No more 90% of that iso then having to start again.

2. It opens up 4 thingies on the file and downloads simultaneously. So 4 times quicker if the website is restricting bandwidth per connection.

> 1. It allows downloads to start and stop. No more 90% of that iso then having to start again.

I'm surprised Firefox doesn't do this itself, as in, this sounds like a clear bug. Is there some reason the upstream project by itself doesn't get this right?

> 2. It opens up 4 thingies on the file and downloads simultaneously. So 4 times quicker if the website is restricting bandwidth per connection.

... this works in 2015? I'm pretty sure I remember people trying this and website owners deploying countermeasures in the '90s.

Is this secretly a bufferbloat workaround or something?

> this works in 2015? I'm pretty sure I remember people trying this and website owners deploying countermeasures in the '90s.

Oh yes, it works. It works wonders. It can very often quadruple the d/l rate.

> ... this works in 2015?

Sites like rapidshare used to 'mysteriously' fail for me at 95% when running Downthemall.

So yep some 'sites' have/had counter measures.

But if you're not a file sharing site counter measures likes these cost $ to implement.

Not saying it's ethical, like ad block (which also has counter measures) it's a fine like between a power user and taking advantage.

This works pretty well if you are behind a HTTP Proxy and the proxy is limiting speed per connection. This was a common scenario in my university.

With the new APIs, you should be able to have 4 web workers do the downloads in parallel still. (Based on how chrome handles such things)

Browsers tend to have a fairly low cap on simultaneous connections to the same server. Seems to be 6 these days in Chrome or Firefox; I remember 2 from a few years ago.

I'm sorta curious if DownThemAll is using XPCOM to get around any similar good-netizen limits in Firefox.

web workers don't have anything except message passing ability to the main js thread, they cant open sockets, and they can modify the dom, they are nearly worthless.

My understanding is that you can use WebSockets within a web worker. But I haven't done it myself, so that could be wrong. In any event, the service worker specs are all still evolving, and I expect that as people try to build more sophisticated extensions on top of these APIs, they will mature to allow more sophisticated behavior.

Yup, it's basically a spider. The Anticontainer plug-in enhances it to fetch images that are a few steps away from the original page. Looks like it just needs unrestricted XHR, which greasemonkey and every other chrome extension already require. Maybe off-screen rendering, which could look like this: https://www.chromium.org/developers/design-documents/extensi...

> The Anticontainer plug-in enhances it to fetch images that are a few steps away from the original page

You should look at Imagehost Grabber, which is best described as Anticontainer on steroids,

http://imagehost-grabber.eu/index.php

Reply on news.ycombinator.com