Curl with Partial Files | Hacker News Reader

Curl with Partial Files

daniel.haxx.se

74 pointsingve19 comments

I'm sure this solves someone's problem, but reading https://everything.curl.dev/cmdline/variables.html makes me wonder whether curl really needs this added complexity

Folks using range requests will certainly welcome these as a great improvement. Without them, you need some ugly shell around your curl command to do it right, sending headers manually and slicing files in the shell.

And, if you don’t need them, don’t use them! Have you seen how many options curl has? But never good ones to support range requests until now.

I'm _pretty sure_ you're referring to `-C - -o $filename` and not this behavior. This one allows slurping ranges of local files and sending those in their entirety. I am open to maybe one day it does something crazypants with `-X PATCH --variable "fred@file[867-5309]"` or such but I doubt gravely that it does any such thing today

As shown here:

  $ python3 -m http.server 9090 &
  # that doesn't matter, just return _something_ on :9090
  $ ls -l .DS_Store
  -rw-r--r--@ 1 mdaniel  staff  18436 Dec 30 11:30 .DS_Store
  $ curl -vo .DS_Store -C - http://127.0.0.1:9090/.DS_Store
  ** Resuming transfer from byte position 18436
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:9090...
  * Connected to 127.0.0.1 (127.0.0.1) port 9090
  > GET /.DS_Store HTTP/1.1
  > Host: 127.0.0.1:9090
  > Range: bytes=18436-
  > User-Agent: curl/8.7.1
  > Accept: */*

One can see where curl helpfully loaded the local file's 18436 size, set up the Range header, and included it on the first-contact with the http server.

While digging into this, there's actually one for its modification time, too. Curl knows all:

  $ curl -v --time-cond .DS_Store -o .DS_Store -C - http://127.0.0.1:9090/.DS_Store
  > If-Modified-Since: Mon, 30 Dec 2024 19:30:50 GMT

As for your "well, just don't use it then" stance, it has been my experience in building software that the Law of Unintended Consequences is brutal. The first bug report filed where someone discovers that a `--header` argument cannot contain {{ anymore due to a fat-fingered function change will let me put another quarter in the I Told You So jar

> Without them, you need some ugly shell around your curl command to do it right, sending headers manually and slicing files in the shell.

This seems much more readable than the stuff introduced here, and I would still heavily prefer to do it this way.

This is powerful stuff, but I can't help feeling like it's adding unnecessary bloat to an already-enormously complicated command. What is the advantage of:

    curl --variable "pwd@secret;[0-31]" \
        --expand-user daniel:{{pwd}} \
        https://example.com/

over, e.g.

    curl --user daniel:`command-to-fetch-password` \
        https://example.com/

? That command may be as simple as an existing one, like cut.

The latter one shows your password in the process listing.

First one will show how to get it :D

… which is not accessible to as many users as the process list is.

And can often be passed in bash via a filehandle, without a file.

`mount /proc -o hidepid=1` recommended :)

Not all places you might run curl allow for shell tricks like process substitution or perhaps for subprocesses at all.

The main benefit seems to be context dependent quoting which is tricky to get right and in most cases (like in your example) omitted completely

What is the use-case for this feature? I get specifying byte ranges for parsing headers and whatnot, but I don't know a curl workflow that benefits from this.

The first one that comes to mind is something I haven’t done lately but used to do constantly, enough so that I wrote a custom fuse filesystem to combine isofuse with http handling, and that’s parsing an index from something with a range request then fetching only the part or parts I want. Think fetching only the trailing index from a zip archive and then fetching the specific file you want out of a 10tb zip rather than downloading the whole thing. Now, how often would I do this on the command line? Not sure, but I might not have bothered with fuse if I’d had the option.

Unless I'm gravely misunderstand this feature currently, it won't help that use case as the variable assignment only supports string literals, env cars, or existing files which wouldn't help your range request for the remote file

It would be a natural extension to treat the @ syntax as a uri (because, come on, it's curl!) if it contains a ":" but even that gets hairy because the next layer of indirection would be --variable-with-curlrc to allow setting the enormous number of parameters one would want when chasing the interior uri

Pausing and resuming downloads, and fetching bytes in splits would turn curl into a download manager like aria2.

Nice! This will make my podcast blog to RSS generator a whole lot easier since I just need to grab some headers from the MP3 in addition to info the the page.

It's nice to see, but the extension on top of the cURL preexisting syntax is rough to look at.

Ever since stumbling upon Hurl a while ago, I've really enjoyed how easy to read the scripts are. Anyone that partially understands web/http is able to read and tell exactly what's going on in the low -> medium complexity scripts w/o even reading the docs.

This could be used to send a single file in a tar. Nice!

Reply on news.ycombinator.com