Hacker News Reader

Comment by yrro | Hacker News Reader

FYI, 'curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/ | man -l /dev/stdin' works for me - no need to safe the roff file locally.

View on HN

I'm pretty sure the OP is intentionally not doing this because piping commands/things-from-the-internet into "bash" (or anything else) is generally considered bad practice.

For me, I'm all for this. Anyone that is aware of the security implications is extremely likely aware of how to make the conversion like you've provided. In that case, no reason to tell people, because they already know. But it's not a great thing to tell noobies to do because they will get burned. So don't tell them. As they advance they'll naturally learn about this feature. And hopefully they have learned the implications by the time they learn how to do this.

(not me): https://www.seancassidy.me/dont-pipe-to-your-shell.html

Piping to your shell and piping to `man` are not the same thing.

The really dangerous thing is copy/pasting from a browser to your terminal. Always do Ctrl-X Ctrl-E to open an editor, paste it in there, then inspect it before saving/closing the editor to run the command.

Never knew that shortcut, useful!

I only recently learned of it myself despite being on nix for over a decade and reading many advanced bash shortcut blogs. IDK why it isn't well known.

For zsh users see here: https://nuclearsquid.com/writings/edit-long-commands/

Note: this can be tricky depending on where it goes in your zshrc. If you use a plugin manager (like Sheldon) then this should be above that. I ended up with this and it works well on OSX and linux

  autoload -U edit-command-line
  # Emacs style (<C-x><C-e>)
  zle -N edit-command-line
  # make sure `set -o vi` is above this line
  bindkey '^xe' edit-command-line
  bindkey '^x^e' edit-command-line
  # (VIM) Use visual mode
  bindkey -M vicmd v edit-command-line

Aren't they? Are you sure piping to 'man' can't result in arbitrary code execution?

The two things you need to be able to say you trust are your CA store, and the source of your curl -> shell.

No, they're not the same thing! It's not piping to your shell! The shell's single purpose is to execute code. Man is not supposed to do that and it would be considered a huge security issue if it could. In any case, how would you check the downloaded file? With a text editor? Are you sure that can't result in arbitrary code execution?

Man can run groff which can in turn run arbitrary subprocesses.

Even if it were:

There is no practical difference. "Nobody" will inspect the man page using a different viewer first. So if I download to disk and then view via man or directly via man is no difference.

A shell script one might inspect first using some viewer. While only few probably do.

There's not much difference doing a pipe or using and intermediary file and "excecuting" that straight away. There's no manual step in the documentation there to read the output before it's run.

This is not accurate. If the download gets interrupted bash will execute the partial line. That means `rm -r /tmp/foo.ext` or `rm -r ${HOME}/.tmp_config` can execute as `rm -r /`.

This can be mitigated by wrapping the script, but clearly no one is looking at the code so this isn't really verified anyways. And it's not like we see that all the time.

Edit:

But my main point is about habits. There's the concern mananaysiempre brings up[0], but either way, it is best to be in good habits.

[0] https://news.ycombinator.com/item?id=39554895

From the post

curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/ > post.page && man ./post.page

If curl's process is interupted, it'll generate a non-0 exit code and the man command won't be exectued. That's how double ampersands work in shell.

I think you've missed the context of my comment. We're talking about `curl | man` not `curl && man`. You're right, `&&` doesn't have that risk. But `|` sure does.

thankfully rm -r / doesn't work on modern systems, though that doesn't invalidate your point.

On what system are you talking about? I mean it won't kill everything because sudo but everything you have permissions to are still in danger

Which usually include your most valuable data.

Nobody cares about losing a system. This is the data it is hosting that is valuable and takes the most time to recover from a backup.

On modern, sane systems (meaning those running GNU coreutils, which doesn't include os x iirc), rm -rf / will refuse to run unless --no-preserve-root is also specified.

https://www.gnu.org/software/coreutils/manual/html_node/Trea...

There definitely is a difference. See [1] (or [2] since the original site has misconfigured TLS). Long story short, using some fairly trivial heuristics, a malicious server could change its response when it detects its output being piped to a shell rather than saved to a file. Thus, a security-minded person downloading the file (who could inspect it) would be given a clean copy, but someone less security-minded piping it straight to bash would be given the malicious copy. The security-minded person wouldn't be able to warn people not to pipe the script to a shell, since it appears safe.

[1] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

[2] https://web.archive.org/web/20240228190305/https://www.idont...

This is piping to man, not bash.

Interesting vector if you're worried about people piping into man from curl, but there you go.

This is extremely specifically is about detecting that Bash is pulling on the stream, by using it's line by line interpretation to do a timing attack. It's not clear there's a timing difference between pushing the stream to disk versus pushing the stream to man, since it seems fairly likely that man would read the entire file into memory before operating on it (at least modern man)

> I'm pretty sure the OP is intentionally not doing this because piping commands/things-from-the-internet into "bash" (or anything else) is generally considered bad practice.

It’s generally considered bad practice only for bash and similar commands that execute their input. It’s not a bad practice at all for commands that just display or transform their input, like `man`, `less`, `ffmpeg`, etc.

Man on Linux runs groff, which (like the original nroff/troff) is a fully general macro processor in addition to being a typesetting system. I wouldn’t bet on it not being able to launch subprocesses, or especially on it having no overflows and such on untrusted input. I’m not even sure about OpenBSD’s much more limited mandoc.

(Also, I don’t know about ffmpeg as it is somewhat more rare to have it be public-facing, but there have definitely been exploits against ImageMagick, usually targeted at websites using it to process user input.)

[insert confused trollface]

> ffmpeg There is certainly a few hundered exploitable vectors in that program alone... to say nothing of the rest.

When in doubt, spin up a VM to run the random untrusted thing -- And then go read its mailing list/issue tracker for known VM escaping exploits. I have a machine setup to test malware, so I just hit my "airgap" switch to isolate the system from my network once the questionable code is in place and ready to run (potentially amok). Study-up about ARP-poison attacks, and remember ARP does not transit to upstream routers/switches (Y "combinate" your network for fun and profit).

Before you assume non malicious simple text output, consider "ANSI" escape code complexity as an intrusion vector for whatever terminal you run. I've got "0-days" for this going back to MSDOS: ANSI Bomb => arbitrary CMD entry. You don't have to take my word for it, your terminal of choice is most certainly vulnerable to some ANSI/escape code related exploit, look it up.

This is why I spin up a VM whenever I want to look at an image. The risk is too great. Even text files, after all we never know if there's a zero day in the UTF-8 decoder. Better safe than sorry.

Wait a minute I just realized there could be a zero day in the VM hypervisor too. I guess I'll just have to buy a fresh Raspberry Pi for each file I want to open.

Fine but we’re not talking about piping random stuff from the Internet here; we’re just using curl as a convenience not to use an intermediary file.

Has there been a single recorded case of malware getting around this way?

I'm sorry, your argument is... what exactly? That we shouldn't take simple or even trivial preventative measures that also reduce other potential problems because we... haven't seen anybody abuse such a thing before? Really?

What a terrible argument. Not it's trivial to resolve. Why not just fix things that we know are problems or can lead to serious problems instead of waiting for it to become a problem where it'll then be FAR more work to clean it up?

Seriously, you're a human, not a bug. You have the ability to solve things before they become problems. Use it.

While I'm not sure of a specific example, I feel quite confident in saying that this has been done before.

There was no argument here, only a question. And it's a fair question to ask for the purposes of discussion.

You're right, piping to sh is definitely a risk, and we should do better as a community, and not make users normalized to piping to sh.

It appeared as a rhetorical question used to rebut the claim. Maybe I misinterpreted it. Fair. But that is a possible interpretation.

But as you can see elsewhere, we shouldn't pipe streams into anything. It's just not hard to avoid this. A few extra characters and you're good to go. Let's take rust for example. Users are copy pasting anyways

They give

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

But why not

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > /tmp/rust_install.sh && sh /tmp/rust_install.sh

For user experience, it is till a one liner, but it is infinitely better. Still has some risks, but a lot less. (at least rust does wrap the install so you don't run the risk of partial line execution) And it gives the user a way to verify that the file was correct if they provide a checksum.

There's just no good reason to not do this. We especially shouldn't be teaching noobies to pipe streams into any command. Let's be real, most people don't know linux well, even if they use linux.

> But as you can see elsewhere, we shouldn't pipe streams into anything.

Firstly, I'll assume you mean untrusted streams since if you didn't mean that, we might as well throw out UNIX entirely.

Even given that caveat though, I disagree. Downloading an image and then converting that image into another format is not a case where there's a material improvement in security when making temporary files instead of using pipes. I'd argue in some cases it may have the opposite effect.

It's indicative of the lack of rationale on this more general case that you selected an example where the piping is into sh here.

I agree with all of your points about piping into sh, just not that we can turn that into a general principal. It's OK if we have different best practices between these two cases.

This kind of advice taken by a laymen or junior dev can cause problems, because the second you put the file on disk there's a risk you won't clean up that content, which for an automated system will ultimately bring it down when the disk fills up. In addition, if the information downloaded is sensitive, you are creating a security problem if you are intentionally writing it on disk even if you do clean it up later, as filesystem data remains persistent after unlink.

If you're willing to run an executable installer from a website then piping into sh is no better.

Read elsewhere. Piping can be dangerous if there's a disconnect. You can also detect piping server side. Piping is strictly worse. Convenient, yes, but 99% of the commands can non-streams and no difference for the end user who is copy pasting it.

Yes. A particularly well known one was codecov.

https://news.ycombinator.com/item?id=26819983

Can't edit my comment so I'll write as a reply

  This is to all the people saying no difference between downloading and running right away

If the download gets interrupted bash will execute the partial line. That means `rm -r /tmp/foo.ext` or `rm -r ${HOME}/.tmp_config` can execute as `rm -r /`. This can be mitigated by wrapping the script. Best way to do this is wrap the whole script into a function and then execute at the last line[0]. Oh, and you can detect `curl|bash` server side[1]

[0] https://archive.is/20160603044800/https://sandstorm.io/news/...

[1] https://archive.is/20230325190353/https://www.idontplaydarts...

Edit: as an example, rust does the warpping. But they still place this stupid shit on in their install instructions. Bad Rust! Bad!

https://www.rust-lang.org/tools/install

I agree that it's a risk to pipe to sh but that's not what we're doing here, and I'd hope that most of the people aren't saying there's no difference between piping to man and piping to sh.

>Best way to do this is wrap the whole script into a function and then execute at the last line[0].

It's not the best way. If your function is named `lsp_init` and your last line is `lsp_init`, a partial line can result in the execution of `ls`.

AFAIK the best way is to just wrap your script in `()`.

You're right! That's a good thing to point out. Elsewhere I suggested Rust change their install to this though. Solves pretty much all the probems

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > /tmp/rust_install.sh && sh /tmp/rust_install.sh

(obviously doesn't solve the problem of inspection, but I think we all know that's not going to happen anyways)

How will it execute the partial line? Interrupted downloads don't just randomly insert <CR>s into the line?

Unfortunately, that command doesn't work on macOS (`/usr/bin/man: illegal option -- l`). I tried to get a one-liner with piping working on my Mac but I always ran into errors. The -l flag doesn't exist in the macOS man implementation (I consulted the man page; meta!).

The `-l` can be skipped (probably because there are `/`s in man's argument):

  curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/ | man /dev/stdin

(which still doesn't work on mac)

This still doesn’t work for me on macOS, perhaps a difference in how /dev/stdin is implemented?

Fortunately, macOS has ZSH as the default shell so the following does work:

  man =(curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/)

Maybe 'man <(curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/)' works then?

Nope

This doesn’t work because it’s equivalent to piping stdout from curl into stdin for man.

It's not, that would be `man <<< "$(curl ...)" `. `man <(curl ...)` is a slightly different version of the process substitution[1] from your comment - `man =(curl ...)`.

[1] https://zsh.sourceforge.io/Doc/Release/Expansion.html#Proces...

You’re right, it isn’t quite the same as piping.

man <(curl ...) ends up working on Linux but for whatever reason still yields an error on macOS:

  No manual entry for /dev/fd/11

  curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/ | mandoc -a

works on macOS (and should also work on (Free|Net|Open)BSD, which ship similar man toolchains).

It's actually process substitution: https://www.gnu.org/software/bash/manual/html_node/Process-S...

It's essentially running the process in another shell and sending its output to a tempfile. The name of that file is then substituted in the original command. So, in theory, equivalent to `inner > /tmp/file; outer /tmp/file`. This works with programs that don't read from stdin, or need two or more inputs, for example.

You can save a few chars with a process sostitution instead of a pipe if you're using bash:

    man -l <(curl -sL -H "Accept: text/roff" https://jamesg.blog/2024/02/28/programming-projects/)

And the world rejoiced as the bytes were saved.

If you’re into that you can save four more chars by using `-sLH Accept:text/roff`

Reply on news.ycombinator.com