Wcurl Is Here | Hacker News Reader

Wcurl Is Here

daniel.haxx.se

78 pointsthunderbong58 comments

> This is one often repeated reason why some users reach for wget instead of curl on the command line.

Is there a reason not to do that? I've always used wget and curl interchangeably, because both meet my moderate needs. Is there a reason I should avoid wget?

sometimes when using containers, curl was already installed and I use that. Other times it is wget. I might want to skip adding "apt-get install wget" etc

the only reason i use wget over curl is that when i'm downloading a file, i don't have to specify the output file name with wget lol

-O (for name based on end of the URL) or -J (for name based on the returned HTTP header)

and wget is -o ; when i have a head full of tasks and code i only remember that they are different and tend to wget things, unless of course i am on a bsd and reach for fetch

> remembering what curl options to use when they just want to download the contents of a URL is hard

Which curl options do they mean? "curl -O URL"?

curl -LO url

`curl -LOf url` is the simplest one I'm aware of that doesn't act too weirdly :)

Without `-f`, it'll happily save the 404 error page if you do `curl -LO https://example.com/index.htmll`

After playing with `wcurl`, I may or may not remember that one.

Also, wcurl puts a lot of effort into making `wcurl url1 url2` work equally predictably.

Finally, wcurl doesn't yet support `wcurl https://example.com` (without a filename at the end of the URL), but it might eventually.

The readme at https://github.com/curl/wcurl explains some other stuff it does by default which makes sense if the goal is to download files:

> By default, wcurl will:

> Encode whitespaces in URLs;

> Download multiple URLs in parallel if the installed curl's version is >= 7.66.0;

> Follow redirects;

> Automatically choose a filename as output;

> Avoid overwriting files if the installed curl's version is >= 7.83.0 (--no-clobber);

> Perform retries;

> Set the downloaded file timestamp to the value provided by the server, if available;

> Disable curl's URL globbing parser so {} and [] characters in URLs are not treated specially.

Is curl able to retry from the middle of a file? Beyond the sane defaults that's where wget has been nice for me; if the connection drops it can pick right back up where it left off.

Yes, curl can resume partial downloads. You need to pass `-C -` to do so, which is a pretty good example of "the command line flags aren't easy to remember"

It's also yet another example of "could you please just do the right thing by default".

curl wants to be a swiss army knife, and so you'll need to configure everything just right so you don't accidentally get the bottle opener instead of the downloader. wget just downloads, and does it well.

So far, I’ve gathered the default command to be

curl -LOCf URL

I don't remember.

"HTTPie CLI — modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more."

https://github.com/httpie/cli

I'll consider HTTPie when they make man pages I can look at in my CLI. This web "documentation" is atrocious.

What is it that makes the web documentation atrocious in your view?

As for man pages, HTTPie has been shipping them since v3.2 [1].

Try `man http` or `http --manual`.

[1] https://httpie.io/blog/httpie-3.2.0#man-pages

I preferred this for years but beware of new management, telemetry and little mention of it. This is from the desktop app but I've seen Little/Open Snitch warn about the CLI as well:

https://github.com/httpie/desktop/issues/176

HTTPie creator here. Just a clarification: HTTPie is still under the same management, and the CLI doesn’t have any telemetry. Little Snitch is probably warning you about requests to packages.httpie.io/latest.json [0]. That is done to let you know when a new version becomes available. It’s a static file hosted through GitHub pages [1] without any tracking. However, there’s currently a bug [2] that prevents the response from getting cached so that the request fires frequently for some users. It’s already fixed in a PR [3] and will be part of the upcoming release.

[0] https://github.com/httpie/cli/blob/master/httpie/internal/up...

[1] https://github.com/httpie/packages.httpie.io/blob/master/lat...

[2] https://github.com/httpie/cli/issues/1527

[3] https://github.com/httpie/cli/pull/1531

Good to know, however updates are a job for my package manager. Imagine dig or awk phoning home for updates? I could see a flag to ask for it specifically.

This is what I've been using for a while now as well. I wasn't aware of the telemetry in the CLI, as mentioned in another reply. That's something I'll need to look into.

Please see my reply here: https://news.ycombinator.com/item?id=41337485

Was fully expecting wcurl to be an alternative name for curl on Windows, because Microsoft made "curl" an alias for their own http downloader command in Powershell.

`curl.exe` is actually the real deal curl, now. Though it's got less built options than a standard Linux distro version. I don't know if they still alias it in PowerShell, but you can still use curl.exe to get it.

I was thinking "web curl", like, an alternative to fetch() or XMLHttpRequest that exposed even more stuff.

Switched to wcurl from my own alias. It's neat. curl doesn't pollute $HOME unlike wget.

How does wget pollute $HOME?

  ~/.wget-hsts

You can disable creating this file or change its location by adding a config option to ~/.wgetrc (or using args to wget every time)

I think it's for disallowing http when an https connection has previously been established?

It's for HSTS: HTTP Strict Transport Security

https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security

So if you're wget-ing a file from a server that uses HTTPS and HSTS, but you only specify http:// (or no protocol at all), then the next time you wget from that server, it will automatically change the URL to https://

It's under .local in Fedora.

They must have patched it, upstream wget doesn't do it.

This is a great start!

Some suggestions:

- Allow insecure URLs by providing -k option automatically for https URLs

- Auto follow HTTP redirects by providing -L option by default

- Rather than use the curl-options setting, just accept —-user and —-pass as options then pass this to curl

I know people will feel strongly about some of these however it would simplify curl usage for a majority of the download only use cases

Other curl options can be used for the more “advanced” use cases like Headers

> Allow insecure URLs by providing -k option automatically for https URLs

Why would you ever do that? This should be a very conscious choice if you decide to ignore the main system keeping the internet traffic trusted.

I use it all the time in house because we have in house, air gapped systems

I know some people feel strongly about this one

But the only time it could lead to a problem is if you pass user/pass and you have a MITM situation.

So maybe only allow it if not passing user and pass

If it’s just a download and we know we aren’t on a TOR node situation then privacy isn’t that great of a concern

My two cents! Open to changing my mind

A MITM situation is relevant even without a credential and isn't at all about privacy: an attacker can swap out a different file for the one you wanted to download.

You are right. Hash (if provided) would still need to be verified upon download

If you have an closed system, then you have two options: use plain http if you really trust the environment, or use your own CA and have a trusted https. Having an untrusted https and disabling it is a double waste of time.

We have our own CA but they don’t originate with any known root. They are self signed certs

That's ok, that's how you normally do it. But then the second step is adding that CA to the trusted store on all relevant clients, so that it can actually get verified. (Otherwise why bother with the CA, just self-sign everything individually)

It’s our lack of a DevOps / Platform Dept. Our traditional IT groups won’t do it sadly

I mean invest in Smallstep SSH - nope

So let me get this straight: your IT won't do something, you're too lazy to add one flag to your scripts, so your solution is to ask that everyone has their security downgraded instead? That's... one way to approach tech issues.

It’s obviously not a suggestion for everyone as I understand. If I could edit my original comment I would strike out my suggestion.

Add the signing authorities to your systems certificate store if it's that big of an annoyance. Or make your own custom alias that includes -k. But this absolutely cannot be default. HTTPS ensures that you are connected to the server you think you are and that no one is messing with your data in transit.

I totally understand this isn’t popular. But even if it doesn’t originate from a certificate chain, it is still encrypted between you and the website. Having the certificate chain lets you know the certificate is part of a chain of trust and prevents MITM

Just don't do that. Some of us (hello) live in countries that perform or tried to perform HTTPS MITM on a massive scale, and only had to roll back because so much well behaving shit broke.

If software suddenly started accepting invalid certificates, they would have no incentive of rolling it back. HTTPS would make zero sense then.

curl does accept if you enable the option to do so. It is optional

This doesn't make it a good idea to break HTTPS by default. Defaults matter, if everything ignored HTTPS errors by default, I would be talking to you over a MITMed connection right now. Because so much software stopped working, they had to roll back that braindead idea in less than a day.

If you're downloading and running a binary or script (pretty common use of curl), anyone on your local network (and beyond) will be able to modify the file and thus take over your machine.

This is like downloading from local Artifactory or Bamboo

> - Auto follow HTTP redirects by providing -L option by default

The script already passes "--location", the non-shortened version of the argument.

For the other things maybe the both safer and more scalable approach is the script should see if it's being run interactively and default to interactively prompting the user for what action to take on detectable things (like missing basic auth or invalid certs for secure domains) or logging an example version of the same command to run as part of the stderr output otherwise. Apart from avoiding debate by taking a stance on what the default should be this will still be allowed by e.g. the Debian repo maintainers.

Good point on —location. I normally use -L. So I missed that

It does default to https as the proto-default if no scheme is provided - in your example it could default to interactively asking the user however this may fail in automated scenarios and/or hang.

Anyways it’s a good thought exercise. It’s hard to satisfy all use cases equally

> - Allow insecure URLs by providing -k option automatically for https URLs

I don't want this by default, but for me it's already better than wget since -k is the same thing as --no-check-certificate. Less typing.

alias wgetnc='wget --no-check-certificate' ;)

I tried it out, it works. Personally I'd write something like this in Rust not shell. It's not hard at all, you basically can just ignore the borrow checker because all the program is doing is spawning processes and a bit of text processing.

The main issue with writing it in shell instead is incorrect error handling. I see a case that's a bit sketchy there. For now maybe don't expose wcurl to user/attacker controlled input.

Why would you bother with such a heavy language, just to wait on a single network event?

Because of the error checking problem shell has. I'll put it on my list to check the script, so I won't only be complaining.

I checked it. Looks okay, but Bash errors are pretty subtle.

[dupe] misleading

Wcurl was here.....2 months ago.

Discussion: https://news.ycombinator.com/item?id=40869458

Reply on news.ycombinator.com