Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
> I wonder what could I have done better.
Used an Ethernet cable? That’s not an impressive throughput amount over local. WiFi has like a million more sources of perf bottlenecks. Btw, just using a cable on ONE of the device => router ~> device can help a lot.
Rsync cannot transfer more than one file at a time so if you were transferring a lot of small files that was probably the bottleneck. You could either use xargs/parallel to split the file list and run multiple instances of rsync or use something like rclone, which supports parallel transfers on its own.
6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.
One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.
Elsewhere I recommended the Thunderbolt4 cable—it screamed at ~10GBs and didn't need any optimization like tar/compression.
> No, I didn't use compression. Would it be useful over a high-bandwidth connection?
Compression will be useful if you're IO bound as opposed to CPU bound, which at 10MB/s you definitely are.
> --adapt[=min=#,max=#]: zstd will dynamically adapt compression level to perceived I/O conditions.
"Carrier-sense multiple access with collision avoidance (CSMA/CA) in computer networking, is a network multiple access method in which carrier sensing is used, but nodes attempt to avoid collisions by beginning transmission only after the channel is sensed to be "idle".[1][2] When they do transmit, nodes transmit their packet data in its entirety.
It is particularly important for wireless networks, where the alternative with collision detection CSMA/CD, is not possible due to wireless transmitters desensing (turning off) their receivers during packet transmission.
CSMA/CA is unreliable due to the hidden node problem.[3][4]
CSMA/CA is a protocol that operates in the data link layer."
https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_...
dd doesn't skip empty blocks, like clonezilla would do.