Hacker News Reader

Comment by linsomniac | Hacker News Reader

Beware that rust coreutils on 26.04 currently has really bad performance with larger block sizes. Ganeti uses "dd bs=1M" as part of a disc image transfer pipeline, and with 26.04 performance dropped from ~350MB/sec to 30MB/sec, with the dd command CPU-bounded.

The issue seems to be that it is operating on partial buffers and then having to copy remainder of that 1M buffer around as part is consumed. Coreutils 0.11 solves that but 26.04 is at v0.8.

I've always used "bs=32k" because decades ago I did some testing and found that seemed to be large enough that it reduced the overhead while not being so large that it caused other problems, one of the most noticeable being scheduler issues where performance went into the toilet, especially on SD or USB thumb drives. Using this also seems to work better with rust coreutils dd.

View on HN

How can a command which was explicitly designed to do nothing but read and write disk become CPU bound?

That's is famously not what 'dd' was designed to do. Its description in the original Unix 6th Edition manual was 'convert and copy a file'. It's infamous for people mis-remembering it as a disc copier when it was actually a file transcoding utility that understood EBCDIC and could re-block things.

I think we are saying the same thing. By disk, i meant disk I/O which is the main practical purpose of the tool. Of course, Linux being Unix, a raw disk is a file, so is /dev/null.

Of course a file can be everything, which is a kind of the point of Linux. But that doesn't change the fact that 'dd' is for disk (or file) I/O. Also please don't argue semantics, it leads nowhere.

But that's not the main point. The main point is the Rust coreutils version of dd is slow. I highly doubt it's impossible to write an implementation in Rust that's as fast as the C one. But the Rust community needs to invest time and energy into finding why that's the case and fixing it. Until that happens the broader Linux community is right to push back on adopting it.

>But the Rust community needs to invest time and energy into finding why that's the case and fixing it.

As I said in my post, the rust coreutils team HAS fixed this in 0.11. It's just that Ubuntu 26.04 is stuck on the older one.

Shipping a coreutils that is version 0.8 in an LTS release seems like an interesting choice.

Sorry I didn't see your post.

But I guess that shows another weakness of Ubuntu's model of versioning. You can't fix a bug if they don't merge. They have their own bugtracker (and so does every distro?) so you, Mr Dev have to deal with angry users not only on your Github Issues page, but on every distro's own bugtracker as well, and will have to perpetually support whatever old version they decided to shit (times X distro).

Does that mean BTW that they'll keep shipping the same broken dd for the next 2(4) years?

Why are we still using dd?

Seriously, it's a weird obtuse command. Weird syntax that doesn't match anything else, and the block size shouldn't be needed 99% of the time. A modern tool should figure out a good value on its own, either by interrogating the devices at both ends, or by benchmarking, or both.

I don’t know of a better tool that operates on raw blocks, maybe there is one.

`dd` is a real power user command, I’d prefer that it isn’t doing “intelligent” things, especially not by default.

    cp file.iso /dev/sda

this works great but in my experience sometimes you should add

  && sync

not sure why it's necessary only sometimes!

The kernel has a cache used for block devices.

If you want to be more targeted:

    blockdev --flushbufs /dev/sda

This isn't portable though, it's a GNU-specific "smart" things. Elsewhere, this overwrites the target device file.

Redirection doesn't overwrite device files, so

  cat foo.img > /dev/sda

works and should be portable. No control over the buffer size, though.

And no printing of progress either.

How exactly is cp not portable?

Whatever behaviour you’re describing is not how POSIX mandates cp to work. In fact you can do this with busybox too.

cp will open("/dev/sda", O_WRONLY | O_TRUNC) and simply start writing to that file descriptor.

How about bmaptool? It can work on sparse images.

In general, I don't see the point in manually specifying block sizes most of the time. What I want nearly always the maximum performance, and that should be possible to derive from the hardware, and/or benchmarking.

What about uses like say, skipping data? seek= and skip= are in block sizes, that may not be ideal for performance.

A smart tool should be able to write in 1 MB blocks and yet still skip 512 bytes.

The 'wierd syntax' has finally been getopt-ified. On the C GNU coreutils 'dd', and in a couple of the BSDs 'dd's.

* https://jdebp.uk/Softwares/dd-with-getopt.html

It only took 52 years. My prediction stands, by the way. (-:

Reply on news.ycombinator.com