Hacker News Reader

Comment by jmclnx | Hacker News Reader

This is one thing I really miss on Linux when compared with the BSDs, especially with dd(1). On BSD you can press ^t to see the status of a command. All you need to do is issue this command to activate it:

% stty status '^t'

btw, for Linux dd(1) I know about "status=progress", but for me it is a bit hard to remember and specific to dd(1). But, nice little utility :)

View on HN

    dd if=<input> | pv | dd of=<output>

to get a count of bytes passing through, or

    pv <input> | dd of=<output>

to get actual completion progress. For tarchives,

    pv <tarchive> | tar x

Compression progress

    pv <file> | bzip2 > <file>.bz2

This is also handy when using socat to pipe across a network. You can use pv on both ends, one looking at compressed and the other looking at uncompressed data, in order to observe the real-time compression ratio.

tar c foo | pv | gzip | socat - tcp-listen:9999

socat tcp:bar:9999 - | pv > foo.tar.gz

If pv shows that you aren’t saturating your network and are cpu limited, replace gzip with lzop. If vice versa, replace gzip with something more aggressive.

pv has a multi line mode so you can do

tar c foo | pv -cN raw | gzip | pv -cN compressed | whatever else

It’s handy to see multiple progress bars at once.

zstd --adapt, used to have a few rough edges though.

pv also has little known ability to watch already running process in similar fashion to progress tool

Try

cp $BIGFILE ${BIGFILE}_1 & pv --watchfd $(pidof cp)

'pv' is great, but you need to have the foresight to use it in your command before you run it. 'progress' seems great for those cases where you didn't realize your job was going to take so long, and you don't want to start it all over again.

Indeed. I created some aliases a few years ago to use pv in these cases. At least until I developed the habit of thinking about pv while composing the command.

You can get status on-the-fly from dd on Linux as well by sending the USR1 signal.

  [user@machine ~]$ dd if=/dev/zero of=/dev/null &
  [1] 3254428
  [user@machine ~]$ kill -USR1 %1
  19061394+0 records in
  19061393+0 records out
  9759433216 bytes (9.8 GB, 9.1 GiB) copied, 6.06968 s, 1.5 GB/s
  [user@machine ~]$ kill -USR1 %1
  25868762+0 records in
  25868762+0 records out
  13244806144 bytes (13 GB, 12 GiB) copied, 8.97352 s, 1.5 GB/s
  [user@machine ~]$ kill %1
  [1]+  Terminated             dd if=/dev/zero of=/dev/null

Ah, exactly the comment I was hoping to see on here as I forgot how you could query this. Thanks!

Glad to help. Luckily, it's noted in the manpage for dd, so I can know it even after I forget!

ahhh, nice! Maybe I would have eventually found it there, if I thought to look.. probably not though! haha :)

This only works because you're allowing backgrounded processes to write to the terminal (stty -tostop).

Try stty tostop

  kaz@sun-go:~/txr$ stty -tostop
  kaz@sun-go:~/txr$ dd if=/dev/zero of=/dev/null &
  [1] 14604
  kaz@sun-go:~/txr$ kill -USR1 %1
  kaz@sun-go:~/txr$ 1142200+0 records in
  1142200+0 records out
  584806400 bytes (585 MB, 558 MiB) copied, 2.62053 s, 223 MB/s

  kaz@sun-go:~/txr$ kill -USR1 %1
  2054884+0 records in
  2054883+0 records out
  1052100096 bytes (1.1 GB, 1003 MiB) copied, 4.70066 s, 224 MB/s
  kaz@sun-go:~/txr$
  kaz@sun-go:~/txr$ stty tostop
  kaz@sun-go:~/txr$ kill -USR1 %1
  kaz@sun-go:~/txr$ kill -USR1 %1

  [1]+  Stopped                 dd if=/dev/zero of=/dev/null
  kaz@sun-go:~/txr$ kill -USR1 %1

  [1]+  Stopped                 dd if=/dev/zero of=/dev/null
  kaz@sun-go:~/txr$

I just backgrounded the process for pedagogical purposes. Usually a process I'm concerned about is running in the foreground anyway, so I signal the process from another terminal.

How often do you use dd that this really matters? Just curious. I’ve run dd maybe 20 times in the 5-6 years I’ve worked with Linux professionally.

I occasionally have a new stupid idea for which I want to use a fresh install on a Raspberry Pi, so I flash a new SD card with dd maybe 4-5 times per year.

Inevitably it turns out my idea was not as useful as it seemed when it first popped in my head, so after a few weeks/months that Pi is turned off and returned to the pi drawer.... ready for the next brilliant idea.

(ps. I typically use the `pv` command to see progress with stuff other than dd)

> returned to the pi drawer

Love that philosophy of keeping an inventory (recycled in this case) of Raspberry Pi’s !

I thought it was the recommended best practice, like with resistor series, 100nF ceramic capacitors, and the more common OP-amps.

Do you buy a Pi when first starting a project?

It sure is best practice, with them still costing like 4x msrp to get your hands on thanks to the neverending shortage.

Regardless of frequency of use, I always want progress and to verify we haven't stalled. With no output, I'm at the mercy of $RANDOM_BS.

At least 10 times this year so far to copy an image to sd cards. Its reliable and easy

Probably not used as much by programmers, but it's a great swiss army knife type tool for copying stuff around with more finegrained control. Sysadmins use it a ton. Probably more often through scripts than directly.

The nice thing with ^T versus status=progress is that ^T will work even when dd was invoked from some script that you don't necessarily want to edit, etc.

It's also a quick and dirty tool for wiping out partition headers on a disk.

Yeah, and backing up the MBR, creating files of a given size like keys, yadda yadda. Anyone getting nitty gritty with storage media, legacy hardware etc, should know their dd.

I first came to love it learning how to fix my bootloader and undo all sorts of horrible things I'd done to my system when I was a teenager exploring Linux well over a decade ago. I also worked in repair for a while and there it was indispensable for data recovery purposes along with its cousin ddrescue.

Now I just use it for my constant tinkering needs.

wipefs -af

recovered a few usb drives this way they wouldn't be recovered any other way (or even recognized beyond a /dev/* entry)

The infrequency just does the issue worse, because every time I want to see a progress, I'd remember that I needed to add a special option to dd.

If you get into anything Raspberry Pi-based, you'll do it a lot. They are shockingly stable once booted, but an M.2 port is my dearest with for the Pi5.

I use it constantly. My company's products are flashed using an sd card and to create it I dd into and from the sd card several times a week

I feel that dd is never the correct tool to write to SD cards. At least cp can figure out the block size itself. Or the more elaborated bmaptool can even skip empty blocks which are often found in disk images.

> Or the more elaborated bmaptool can even skip empty blocks which are often found in disk images.

  dd conv=sparse

> At least cp can figure out the block size itself.

Why does it matter? Use `bs=$((4 * 1024 * 1024))`. It'll work perfectly for any imaginable block size.

My issue with dd is that it's possible to write corrupted data with some weird flags which I did once. Something with conv=sync I believe which does unexpected things. But if you're not trying to be too smart, dd works fine.

Reminds me of this: https://www.vidarholen.net/contents/blog/?p=479

^t (SIGINFO) works with a lot more than just dd. It also triggers kernel behavior (printing the current waitchannel, e.g., named lock, if one is relevant) which can be useful for diagnosing why a command (any command) is hanging. (Is it busy in userspace, or in the kernel? And if in the kernel, where?)

I used it a lot (didn't miss the progress bar though). With it's options to control caching, it can be used as bare-bones single-threaded performance test for sequential access. Won't replace more elaborate test suites, but has its use as sanity-check.

dd is the command to use when transferring bootable images onto USB dongles for installations; when working with embedded boards, especially for testing, one can use it like 20 times or more in a single day. Having a progress feedback isn't vital per se, but becomes useful when using by mistake a slow USB dongle, or plugging it in the wrong, slower, port.

I very rarely use dd. And thank $DEITY for that, because my typical use case is creating a USB boot drive and then backing up or recovering what I can from a failing disk.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-init...

I'm astonished that the BSD projects are merging un-Unix-like fluff like this.

Meanwhile, the Linux kernel has removed Shift-PgUp scrollback from the console.

Why did they do that? I'm sure there's a good reason but it sounds interesting!

The good reason is apparently that scrollback in vgacon was broken and there was no-one to fix it. No-one has stepped up to volunteer to keep it fixed in the future in the two and a half years since then.

* https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...

Long-term we'll probably end up with the console in userspace, which should naturally normalize scrollback behavior across all the kms drivers.

There's already kmscon but I haven't looked closely into what prevented that transition from moving forward as the default. Presumably there's a bunch of edge cases a kernel based console doesn't suffer from, like having to avoid oom killer or getting paged out/becoming unresponsive on systems falling over etc.

The console process will have to be something exceptionally configured, pinned memory, realtime priority, oom-immune, all that jazz.

Not really. Most of such a system can just be ordinary processes, and there's no requirement for pinned memory anywhere. Pretty much only the terminal emulator processes are problems if they die, as that effectively becomes a hangup signal, so only they need the oom immunity, if that.

* https://jdebp.uk/Proposals/linux-console-daemon.html

What happened to kmscon is that the systemd people, having adopted it with much hullabaloo, dropped it a while later with far less fanfare.

That said, there are a range of options for people who want to build Linux without CONFIG_VT.

* https://jdebp.uk/Softwares/nosh/user-vt-screenshots.html

* https://jdebp.uk/Softwares/nosh/guide/user-virtual-terminals...

(-:

What does this mean ? IIRC ^t has been in the BSD for over 20 years, maybe even in early days. ^t just maps to a signal.

I can find SIGINFO all the way back to 4.3BSD-Reno, released in 1990.

WAT, how is this "un-Unix-like"?

Unix-like is that the program is dead silent while nothing is going wrong and then indicates its progress by a successful termination status.

Unix-like sounds unreasonable if there isn’t leeway for optionally reporting progress on stderr (--quiet/--non-quiet or similar).

No news is good news

It is silent until you press ^t, and then it prints the current progress. It's not constant output.

UNIX-like is having your session or app time out because it gets no output. going all the way back to HP Omniback 2 running BCV split pre-exec scripts on an EMC array to back up a database to tape off a split mirror. Then aborting the backup because the array-side mirror split took over 15min, with no output.

UNIX-like is not a convention of UNIX - it's a convention of "we didn't get to making that useful thing yet, there's other stuff to do."

Sure, but the BSD convention meets that until you hit ^T, and then it does the thing the user wants (give a report) and then it goes back to being silent. There's nothing wrong with that, and the BSDs are Unixen, so... It's not like ^Z (SIGTSTOP) is un-Unix-like just because Unix VII (and System V) didn't have it!

Perhaps an alias in your .bashrc for dd would solve it? I usually just use ddrescue most of the time (primarily because I prefer it's usage syntax, but it also reports status).

Similarly you could alias rsync instead of copy and move:

   alias pcp='rsync -au --info=progress2'
   alias pmv='rsync -aP --info=progress2 --remove-source-files'

There's also dd status

       status=LEVEL
              The LEVEL of information to print to stderr; 'none'
              suppresses everything but error messages, 'noxfer'
              suppresses the final transfer statistics, 'progress' shows
              periodic transfer statistics

and pv

    pv - monitor the progress of data through a pipe

So with pv, you can do something like

    dd if=/dev/zero count=2 bs=512 | pv | dd of=/dev/null

to visualize your dd progress.

I am aware of both of those, but in my case I prefer ddrescue for the syntax. For backup scripts I use dd and pv (and pigz).

I recall I started using ddrescue because one of my distros would not autocomplete with the if= prefix, but I can't seem to replicate it currently - so it's probably been fixed (or my memory is failing me).

Ctrl-t works for me on DD on Linux.

Not here. I guess you've somehow got ^T to send SIGUSR1 to the foreground process?

I think I'm mistaken. Tested on a few Ubuntu machines and it didn't work. I must be conflating my experiences between local on MacOS vs remoting to Linux machines.

Since MacOs is based upon a BSD, ^t should work fine on it.

Can't speak to the others, but on FreeBSD you don't need to activate ^t (SIGINFO); it just works that way out of the box.

kill -USR1 {dd pid} ;)

The problem with usr1 is it defaults to killing processes, so you may only use it with processes you know handle it that way.

Info defaults to dumping generic stuff, it’s completely safe (unless a dev decided to handle it by dying but I would not want to use their software).

This is with dd though, which will print the progress with a USR1 signal. It's not for other binaries.

FWIW, dd will give a status if signalled. SIGUSR1 on Linux, SIGINFO on macos.

Linux did also spits out status if you kill -USR1 it, which is useful when you forgot to progress=status it.

Don't try to remember adding status=progress. Add a shell alias for dd.

And then remember to copy that to every machine you use, I guess. Including one-offs.

Or automate it.

You personalize your shell, don’t you? At least I do. My dot files such as .zshrc are in a fit repo, which I check out on a new machine.

I don't personalize shell environments on every machine I ssh to, no.

I tend to setup my servers, vms and machines with Nix. It takes care of having the same setup everywhere I interact with.

You can send a USR1 signal to dd and it will print its progress.

Reply on news.ycombinator.com