Hacker News Reader

Comment by naikrovek | Hacker News Reader

Oh thank you for submitting this to HN. I’ve been telling people not to use dd for years and everyone looks at me like I just gave birth to a full grown dinosaur or something.

“Well why does the entire internet say to use dd then?”

Because they copy from each other just like you copied from them. Just use cat.

View on HN

> Just use cat.

We’ve gone full circle.

https://en.m.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_ca...

The reason to use cat is so you don't risk accidentally overwriting the source file by messing up a command.

Anything after the "cat file |" can't overwrite the file.

That seems like a bad reason. You can always overwrite the file by messing up the command.

  cat file | cat > file

(This won't, as you might hope, leave the file unchanged.)

Re-read what you typed before hitting enter, and keep backups of important things.

That's an unlikely mistake.

More likely is you are messing around with some new/infrequently used utility and you pass in the arguments incorrectly and specify "file" as the output instead of the input as intended.

The one I typed is unlikely (I wrote it in response to your "Anything after the `cat file |` can't overwrite the file", which is factually incorrect as demonstrated), but there are plenty of other simple mistakes. Understand what you're running, and keep backups, otherwise you'll screw up eventually. Selecting your tools this way doesn't move the needle much.

Removing cat and specifying the file directly to the utility doesn't move the needle in any tangible way either.

At the end of the day do whatever pattern works for you - for me it's doing "cat file |" at the start of a pipeline and "> outfile" at the end.

I also avoid globbing inside a pipeline as it can be dangerous too.

You can replace "cat file |" with "<file".

Well, yes, but then it wouldn't have been a very good demonstration of why the statement "anything after the `cat file |` can't overwrite the file" is incorrect.

>“Well why does the entire internet say to use dd then?” Because they copy from each other just like you copied from them.

Great statement. This brings me some much needed internal clarity on my own thoughts and actions.

This is basically the parable of "Grandma's Ham"

https://www.executiveforum.com/cutting-off-the-ends-of-the-h...

tl;dr nobody in 2 generations knows why they cut the ends off the ham before cooking it, until they talked to grandma, who said her pan was too small

A Unix thing that's been posted to HN for a decade, that's almost literally the same story:

Understanding the bin, sbin, usr/bin , usr/sbin split

https://news.ycombinator.com/item?id=3519952

tl;dr /usr/bin is separate from /bin because someone had a small hard disk once

So Grandma's Ham is a variant of Chesterton's Fence... When you do or don't do something for reasons of tradition, find the real reasons for doing / not doing it.

You’d be amazed how many people I’ve found that do ‘tar xzf filename’ without knowing what xzf is doing.

Passing "z" is pretty pointless for extractions.

I think it was required until about 2010 or so for GNU tar. (I still use it from habit.)

There was a problem about integrating support for different compression applications, with each one needing to get a new letter in the tar command!

Won't this fail when the file is not gzipped but, for example, zstd compressed?

It doesn't assume gzip, it detects the compression format.

Yup, I remember when you had to pass `z` if it was gzip, and I remember my surprise when I missed it once and it still worked (apparently about 5 years after it was no longer needed!)

I'll admit to being one of those people. My brain still spells it out as eXtract Ze Files every time :-P

You're 2/3 correct:

x - extract

z - gzip format

f - file (must be last so it parses the filename arg correctly

I also add "vv" in the middle so it lists every file as it goes, so I can see it work instead of just waiting with no output.

It's not often I actually unzip stuff on the command line these days, but saying "extract ze files" in a terrible French accent in my head when I do is a highlight.

I don't tend to amaze myself.

What initially mystified me into thinking "I guess I have to use `dd` for block devices" was learning that `rsync` doesn't work with block devices.

I just tried to copy an ISO to a USB drive with `rsync`, and it didn't work.

Looking it up, I read these comments[0]:

"rsync operates on files which are on a filesystem. It doesn’t do comparisons between blocks. If you want to back up a block device to a file, use dd"

"I completely understood the use case. rsync (still) does not operate on block devices, so it’s not the solution here."

"Agreed, rsync has never operated on block devices."

So I thought, "Ah. There's File Data, which is what `rsync` deals with, and RAW BLOCK data, which is what scary hardcore tools like `dd` deal with."

Then the notion that `cp` and `cat` can deal with both species of data is confusing. :p

0: https://old.reddit.com/r/linuxadmin/comments/eappzm/rsync_bl...

"Well why does the entire internet say to use dd then?"

Because you usually need root to write to drives, and you can't `sudo` redirect. `sudo dd if=whatever.img of=/dev/whatever` is nicer and easier than `sudo sh -c "cat whatever.img >/dev/whatever"`.

In the cases where you need root it's often even easier to use the 'tee' command instead of spawning another shell; esp. if the command has symbols which need to be escaped.

Your example would then become something like...

cat whatever.img | sudo tee /dev/whatever > /dev/null # https://stackoverflow.com/questions/82256

Just like in household plumbing, the 'tee' command basically takes the input and sends it to more than one place. Naturally, running 'sudo tee' will let you send things all over but as another user.

All that said, I won't speak to the comment "why does the entire internet say to use dd". I've never actually found the "whole internet" to agree on much of anything (-:

Well, I’d like to thank the author for writing this. I always enjoy having the veil of “magic” lifted from computers.

UNIX’s simplicity of “everything is a file” continues to surprise me, even though it shouldn’t any more. I think a lack of confidence in my understanding of the basics, leaves me tempted to copy from others as you mention.

Author here. You're welcome ^^

What if you need to test direct IO, or test various partial-page, partial-block, or full block/page access to pagecache? Sometimes you might actually want to try using direct IO too, if buffered IO is not giving you sufficient control or reporting of errors from the device, or continuation from part-way through the copy, and in some cases possibly because you don't want to consume memory for page cache (although dd isn't good with direct IO so that's probably unlikely).

That said if you don't know whether or not you need it, you don't. And those who do would know not to listen to your advice (for what they need to do) anyway, so it's not bad general advice to give.

Direct I/O is notoriously finicky. Depending on what filesystem you're using there are apparently alignment restrictions and other corner cases to think about. I don't think it's that `dd` isn't good with Direct I/O, I just think its hard to use.

Direct IO is not finicky (on Linux, since 2.4 era kernels). It's just that at least with dd, if you don't know what it does then it probably does not do what you want (synchronous, single queue depth). If you know what you want and know what it does, then it's fine to use, and can be useful (I used it today to measure block device interrupt latency in a VM).

I've been taught at my uni to use dd...

Reply on news.ycombinator.com