Hacker News Reader

I agree on the "sane defaults".

Just the fact that `wget url` downloads a URL and saves it makes it a winner for me in command-line use.

Well, the point of the article is that they are not cpmpetitors and are used differently. For me, 99% of the time i'm curl-ing some API and I definitely don't want to save the result to disk (but often want to pipe it to grep/jq).

Agree. I also use both: wget to download files, and curl to talk to APIs, and as a "no magic" HTTP client.

This is not about "sane defaults", but about use cases.

> I tried using curl but out of the box it could not resume or retry the download.

Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.

You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases?

Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different interdependent flags for specifying retry behaviour: --retry-all-errors, --retry-connrefused, --retry-delay <seconds>, --retry-max-time <seconds> and --retry <num>

If I "just" want it to retry, surely there's a boolean flag like "--retry" that enables sane defaults? Nope! --retry takes a mandatory integer argument of maximum number of retries. Surely I can set it to zero for a sane default? Nope again: "Setting the number to 0 makes curl do no retries."

curl is a good tool if you know it through & through and want exact control over the transfer behaviour. I don't think it's a good tool if you just want to fetch a file and except your tool of choice to apply some sane behaviours for you to that end, that would probably make sense if you are a human rather than an application using a library.

> You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases?

But... it does, though. From the man page ( https://curl.se/docs/manpage.html#-C )

> Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.

I just tried it, works perfectly. I don't really see a difference between writing `--continue` in wget and `-C -` in curl. And the use case for specifying it is not so exotic, you might want to do range requests for all sorts of reasons.

Look, it's fine if you prefer wget's command line: I don't think retrying a request is a reasonable default for a tool like curl, but reasonable people can disagree on that for sure. But curl is perfectly capable of resuming downloads automatically, you're just (very arrogantly) wrong on that one.

No, `-C -` is not a flag. It is specifying the `-C` argument with obscure special value of `-`, which causes curl to determine the offset to continue from the output file length. This might be obvious to you if you are well-versed in curl command line, but it's by no means expected or obvious like a simple flag.

> But curl is perfectly capable of resuming downloads automatically, you're just (very arrogantly) wrong on that one.

I've never claimed it doesn't. I've only demonstrated that the default options don't do it and enabling the behaviour is more difficult than it maybe should be for a simple tool. I fail to see the arrogance.

> I've never claimed it doesn't.

Yes you did:

> You must specify the offset from where it should continue

No, you mustn't, you can specify - and it does exactly what you want. The docs are very clear and even provide examples. At some point you should stop blaming curl for your inability to read a man page and admit that you were simply mistaken.

You still fail to understand that curl's -C does not behave as a simple flag but as an switch with a mandatory argument. And there's a magic special value for that argument that finally enables the expected behavior. It's unintuitive, hard to remember and not in agreement with usability. While I agree that curl is powerful I will not concede that it's CLI is user friendly.

It's not hard to remember if you're familiar with unixs tools and syntax. But no one is demanding you concede anything. The point of the conversation is explain the difference in expectations between how you expect a command line tool to act and how most people expect a command line tool to act. If I try to `cp src dest` and it fails. I don't want the tool to guess how to fix the issue, that's not it's job. Ditto for `dd` it shouldn't try to guess offsets. curl exists as a knife, you're expecting it to do the job have a food processor and blender combination. You're not wrong to expect a tool to behave that way you're wrong to expect curl to behave that way. And no one is asking you to concede that a knife is easier than a blender when you want a blender. everyone is pointing out it's not a blender it's a knife but you can still do everything you want to do.

> It is specifying the `-C` argument with obscure special value of `-`

It is not an “obscure special value”. Not only is `-C -` (or `--continue-at -` for the long form) well documented in the correct place in the manual, `-` is a common value in command-line tools (e.g. when specifying that a tool’s input will be STDIN instead of a file).

In what sense is "read offset to continue at from STDIN" a meaningful interpretation of `-C -`? That's not what it does.

That’s not what I said it does. An example (marked by the use of “e.g.”) can be something similar.

What is a sane default for retries? is it to loop indefinitely? should it retry against the same TCP connection or establish a new one? To the same IP it picked the first time or a different one, or reresolve the DNS entirely? against the same resolver?

IMO theres too much complexity for 'sane defaults' to not just be 'surprising behavior' for someone else's use case.

It's indeed miraculous how Wget got this right

Not only are you wrong, I had the opposite experience - whether was all well and good for downloading pages, but it fell over for making e.g. authenticated POST requests. I think that might be possible now/have gotten better, but wget has always been too restrained for me.

I am in agreement that wget has "sane" defaults i.e. it acts like a bot or web crawler, or basic browser. Curl has always been easier to get things done with, though. At least in the land of http requests.

I thought that `-C -` checks the output file for the offset

Yes it does. How does the wget --continue work?

I don't know

Strong agree. The only misbehaviour I believe curl displays out of the box is globbing, which has burned me enough times that I’ve come to believe it would’ve been better disabled by default and enabled with -g instead of vice versa.

do you have an example of the globbing that burns you?

Also add -i which lets wget read URLs from a file. In particular wget -i - which makes it read from standard input, and is very useful in pipelines.

curl cannot, AFAIK, do this. People usually suggest using xargs, which is a mediocre substitute because it waits for all the URLs to arrive before invoking curl, giving up any chance at parallelism between the command generating the URLs and the one downloading them.

`--config filename` allows this. `--config -` for stdin. Not only urls, but any config options

  echo '--url https://google.com/' | curl --config -

xargs doesn't have to wait, you can specify the number of items to include in a single sub-command and it'll batch things as they come in. For instance:

    ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs -L5 echo
    1
    2
    3
    4
    5
    1 2 3 4 5
    6
    7
    8
    9
    10
    6 7 8 9 10
    11
    12
    [... and so on ...]

If the xargs call uses -I then --max-lines=1 is implied anyway.

If you replace echo with something that sleeps you'll see that the pipe doesn't stall waiting on xargs so the process producing the list can keep pushing new items to it as they are found:

    ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs --max-lines=5 ./echosleepecho
    1
    2
    3
    4
    5
    starting 1 2 3 4 5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    done 1 2 3 4 5
    starting 6 7 8 9 10
    15
    16
    17
    18
    19
    [... and so on until ...]
    98
    99
    100
    done 46 47 48 49 50
    sleeping for 51 52 53 54 55
    done 51 52 53 54 55
    sleeping for 56 57 58 59 60
    [... and so on until xarg's stdin is exhausted]

And you can stop the calls made by xargs being sequential too for more parallelism with the --max-procs option (or use parallel instead of xargs):

    ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs --max-lines=3 --max-procs=10 ./echosleepecho
    1
    2
    3
    sleeping for 1 2 3
    4
    5
    6
    sleeping for 4 5 6
    7
    8
    9
    sleeping for 7 8 9
    10
    11
    12
    sleeping for 10 11 12
    done 1 2 3
    13
    14
    15
    sleeping for 13 14 15
    done 4 5 6
    16
    [... and so on ...]

(I adjusted max-lines in that last example because my current timings made things line up in a manner that made the effect less obvious, adjusting the timings would have been equally valid, in a less artificial example like calling curl to get many resources timings will of course be less regular, perhaps these examples can be improved by randomising the sleeps)

I'm not sure what you would do about error handling in all this though, more experimentation necessary there before I'd ever do this in production!

Reply to self to add a note of something that coincidentally came up elsewhere¹ and is relevant to the above: of course xargs being able to push existing things forward while the list of actions is still being produced relies on it getting a steam of the list instead of the whole thing in one block. If your earlier stages cause a pipeline stall it can't help you.

For an artificial example, change

    (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs -L5 echo

to

    (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | sort | xargs -L5 echo

The sort command will, by necessity, absorb the list as it is produced and spit it all out at once at the end. xargs can still use multiple processes (if max-procs is used) to make use of concurrency to speed the work, but can't get started until the full list is produced and sorted.

----

[1] An unnecessary sort in an ETL process causing the overall wall-clock time to increase significantly

Right, but then you are invoking curl several times, and so not reusing a single connection, as you would with wget -i, so it still loses.

Valid criticism, ish, but that wasn't in what was previously asked for so well done on being like my day-job clients and failing to specify the problem completely :)

You can specify multiple URLs on the same command in curl so using xargs in this way would do what you ask to an extent (the connection would be dropped and renegotiated at least between each batch) as long as you don't use any options that imply --max-lines=1.

With the --max-procs option you could be requesting multiple files at once which may improve performance over wget -i – though obviously take care doing this against a single site (with wget -i too for that matter) as this can be rather unfriendly (if requesting from multiple hosts this is moot, as is the multiple files-from-one-connection point).

Both tools have their use-cases. I think with the advent of LLMs like ChatGPT it has also become a lot easier to get the proper command line incantation for whatever tool you're using. Even if you've read the manual previously, it's easy to forget the exact flags you want to use, and validating the generated command line is usually less effort than having to build it up yourself from scratch by reading the manual.

With the modern web, sometimes it's easier to use a tool like Puppeteer from a custom script. Especially if the sites you're interacting with are using a lot of JS.

I just look at my curl / wget note for some sane examples. i cant believe one needed chatgpt to be productive

You don't need much of anything, it's just another tool to help people figure stuff out. If you've already invested the time in reading through the manuals for these tools then maybe you don't get much out of something like ChatGPT, but consider that there's thousands of new people entering the industry every year and the number of tools which they're expected to learn to use has increased over time.

It's fine when you've read the man page at least once. If you just cowboy your way through every task, you'll never even be aware of what your tools can do. Which results in comments we've all seen: "I can't believe curl/Firefox/vim/readline/whatever can do something like that!" about something so trivial that every good poweruser/sysadmin has known it for decades (because it's in the man page).

Sure, but that's knowledge you pick up over years. You can't reasonably expect people to frontload all of it on the first pass with every single tool they interact with, especially when starting off.

I've read through most of the man pages of every tool I use at least once, but it has taken me years, and I've done it incrementally.

Personally I use my memory. It's a bit disturbing that some people can't write code without reading the reference for basics.

I just memorise the man page. I can't believe etc etc etc

(Seriously, there's a million nix flags, and only so many brain cells. ChatGPT's better than Google for simple "what's the magic incantation?" searches, and laziness is a virtue. If you don't want to be lazy that's fine, but I think you're missing out).

If chatgpt makes someone more productive, then why not. It's just a tool as any other tool.

I agree with this sentiment. But I also have personal antidotes that make me cautious about it. When I was a child I memorized dozens of phone numbers of people I called a couple times a month. Now my phone memorizes any phone number for me. We used to memorize poems and such. I can't give you any scientific reason why it is good to have memorized the Gettysburg Address. But I feel intuitively that there is some benefit to exercise your brain similar to the intuition I have about exercising muscles.

If a person thinks they benefit from 100 situps a day, I'm not going to disagree. And if they think there is some benefit in reading man pages, well, thanks to those who take the time to write all that ducumentation.

When it backfires you're screwed, I have experienced it one using curl -L

I would love to see your curl / wget note if you don't mind.

I like tldr for quick examples. If it doesn't have what I need, then I fall back to phind or some other llm. Nice thing about tldr is there's an offline version.

https://tldr.inbrowser.app/pages/common/curl

https://tldr.inbrowser.app/pages/common/wget

Another pet peeve of mine is that curl's URL parser is a lot more strict compared to wget.

For example:

  $ curl -sSLOJ 'example.com/file name.txt'
  curl: (3) URL using bad/illegal format or missing URL

  $ curl -sSLOJ 'example.com/file%20name.txt'
  $ ls
  file%20name.txt

On the other hand, wget (without any additional flags) will produce a file called "file name.txt" for both URLs. Well, technically you also need to add a --content-on-error flag to wget because this example URL 404s.

I thought curl had the ability to encode html entities if you asked it to?

HTML entities (&.....;) are distinct from URL encoding (%..).

Retry with `wget` was one of the most incredible Linux distro included features when I started running it at home. Pretty crucial thing on 56K dialup, and it worked better than the Windows tools I was aware of at the time.

Our dialup used to disconnect every 2 hours by design. wget and wvdial were the only alternative to mail-ordering CDs

We were way out in the sticks, so interruptions were very common. Still nothing but dialup, satellite, and LTE (no 5G) out there.

I think our setup was very particular to the UK. We didn't (and still don't?) have free local calls like the US, so we paid per minute for ISPs.

Almost all ISPs went through a scheme setup by British Telecom - you could either have free internet but you paid for your calls, or you could pay for your internet, and have access via a freefone number - so effectively flat-rate.

But the flat-rate option disconnected after two hours, on the dot. Which was hugely frustrating because we had a voicemail variant that was hosted by the telco, and let you know you had messages waiting by pulsing the dialtone. And my modem did not recognise the pulsed dialtone as a valid dialtone, and refused to connect until we called the number and marked them read.

Which lead to one of my most UK-centric retro stories. I tried to connect to the internet, and it refused to dial. I blew away my wvdial config, and it refused to dial. I blew away my ppp config, and it refused to dial. I grepped / for the error message and it didn't exist. I ended up blowing away my OS (and accidentally installing onto the wrong drive, and blowing away everything non-OS too), and it still wouldn't dial.

So I dragged the modem & extension cord to my mother's PC, and shot off a mail to my preferred mailing list (one hosted by John @ linuxemporium, my preferred source of mail-order distros), and swiftly received the response that in order to be certified by BT to operate on their network, one of the rules equipment had to obey was to refuse to redial the same number x many times. And that all I needed to do was power-cycle the modem. Which I'd done by dragging it upstairs to my mother's PC. And my own machine had been wiped twice over needlessly.

Aside, there was a lady named Helen on that mailing list who knew everything about everything, and is everything I aspire to be today. She had opinions on which harddrives best survived salt/sea air, and why they weren't deathstars. Just an incredible amount of lived experience. I miss mailing lists.

Interesting information w.r.t. UK telephone practices! In the USA, it was usual to get free local calls, so ISPs would set up modem banks to try and get maximum coverage for a given NPA-NXX range. There was some arrangement with CLECs and ILECs where it was extremely profitable for IIRC CLECs to pass data-only calls through to ILECs, so one or the other was subsidizing a lot of the early dialin ISPs, to the point of buying them modem banks and whatnot!

That was probably one of the biggest death-bringers for the BBS era, no more long distance calls to get to what you wanted.

> Which was hugely frustrating because we had a voicemail variant that was hosted by the telco, and let you know you had messages waiting by pulsing the dialtone. And my modem did not recognise the pulsed dialtone as a valid dialtone, and refused to connect until we called the number and marked them read.

Yeah, some VM providers in the USA did that too, and it similarly confused modems. It's called "stutter dialtone" here, and the usual fix was to put some delay elements in the dial string, which were commas for Hayes command set modems.

> and why they weren't deathstars

They sure did earn that name! I was so hesitant to switch to HGST for ZFS pools due to my 90s/2000s deathstar experiences. Wouldn't run them in production for a while, of course now that I'm over it and trust them as well as any other enterprise brand, they'll screw it up again!

For downloading large files I would rather just use aria2c TBH.

I only used aria2c once a long time ago, but it was awesome for huge files. As I recollect one thing it does is download different sections of a file in parallel over multiple connections, which speeds up downloads from servers that throttle per connection.

Aria2c currently looks unmaintained https://github.com/aria2/aria2/pulse

Also here

https://github.com/aria2/aria2/issues/2078

Also following redirects by default. "Run curl, be confused run curl -L" is something I do many times a month.

I mean, it all depends on who you are and what you are primarily doing. “Sane defaults” for one person could be everything another doesn’t want.

One point where I would argue wget doesn't have a sensible default is on filenames - it really should make --content-disposition the default, at least for single file downloads. Otherwise it will often use the wrong name if there is a redirect or similar in the chain, which seems increasingly common.

I'll argue not trusting the server to dictate the saved file name is the only correct default behavior, and taking the filename from the user input (URL) is reasonable.

A server configured with a docroot to serve a static site will map requested URLs to hierarchical filesystem paths, but that isn't the only possibility; it's a common but quite loose coupling of ideas.

But the filename directive of the content-disposition response header is entirely coupled to the idea of a filename. Therefore, it ought to take precedence.

I don't want the stranger deciding to save the content as .bash_profile or such.

Fair enough, but do you have that fear when using a browser? I guess a Downloads folder is lower stakes than whatever other working directory you're wgetting from, though.

Also (annoyingly) URL params

curl is an excellently powerful library and utility but I agree that wget has better defaults. I am almost certain to get the behavior that I want by just throwing a URL at wget, including retrying from the point where it had an issue. I actually ran into a case where our corporate firewall was a little too eager to block a download being performed by the Visual Studio installer because of a signature match partway through a specific download. All I had to do to grab the file was have wget download it. No magic incantations, it was just smart enough to not start the download from the beginning after being cut off, and since it started midway it no longer tripped the signature match rule.

Indeed - and curl requires `-L` to follow redirects whereas wget just does that by default too. So for ad-hoc CLI use, I turn to wget rather than remember all the curl options required.

I think of it like this: curl makes HTTP requests, wget downloads files.

(Though both can be made to do the other thing in some capacity)

I would use GNU Wget2 for this. It's supposed to be a wget successor.