I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.
As to slow operations... that is more likely because of synchronous implementation.
The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination.
A better implementation (what I would do) would be to pipeline operations. Basic pipeline would have three components, each streaming data to and/or from buffer
1: Read from source to pipeline buffer
2: Read from pipeline buffer to write to destination, write information about files to delete to another pipeline buffer
3: Read files to delete from pipeline buffer and execute deletions.
Using *nix shell you could do something like that in single line
1: tar -c files to output
2: pipe output to tar -xv, write files to destination producing list of written files, pipe written files to output
3: read piped list of written files and remove them from input dir
Now, this is not perfect because we are wasting performance on creating tar file when we immediately discard it, but you get the picture.
Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened.
> As to slow operations... that is more likely because of synchronous implementation.
Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.
There are obviously improvements you could do. For some filesystem you can just remove entire folders rather than remove the files individually just to remove parent folder.
Reminds me of the kind of patterns functional programming languages introduce, where you process data by describing operations on individual items and assembling them into "a stream". I'm always wary of those - without a good implementation and some heavy magic at the language level, they tend to become the kind of context-switching performance disaster you describe.
I am personally of the opinion that, to be a good developer, you have to have mental model of what happens beneath. If you are programming in a high level language it is easy to try forget about the fact that your program runs on real hardware.
I know, because I work mostly on Java projects and trying to talk to Java developers about real hardware is useless.
I have an interview question where I ask "what prevents one process from dereferencing a pointer written out by another process on the same machine" and I get all sorts of funny answers and only 5-10% candidates even have beginning of understanding what is going on. Most don't know what virtual memory is or are surprised that two processes can resolve different values under same pointer.
Yikes. How many of them have a degree in computer science?
What happens is they know what virtual memory is but can't connect the concepts. It is knowledge without understanding.
But that was incredibly hard, and error prone (and comes with tonnes of limitations). The fact that today, it's very easily possible to write working programs without knowing any of the underlying details, is a marvel.
If i were to hire a truck driver, i wouldn't expect to have to ask him to understand how the truck itself worked (e.g., fuel injection when he presses the accelerator). He only needs to know how to operate the truck from the interface (steering wheel!). Why isn't this the same for a java programmer?
No, it wasn't. If it was incredible anything it was tedious. But nobody really expected anything super complex from you. Just look at the kinds of programs that were produced in 80s or 90s.
It would be incredibly hard today. Machines got very complex, operating systems got very complex.
But it wasn't so complex in 90s. I think I stopped writing assembly when I started using WINAPI because that was the point when assembly stopped being practical.
Sometimes I wonder how I would fare if I was 18yo again today and had to start in development and learn everything from scratch. I feel being able to learn everything as the technologies were evolving is a huge advantage I enjoy.
I pick Rust every time.
I have this concept of easy problems and hard problems. Every decision to choose technology is a compromise and comes with its own problems. It is your job to know whether these are easy or hard problems.
Compilation time is easy problem. Just put more hardware to it or modularize your application or schedule your coffee breaks correctly.
Building reliable abstractions to prevent hard to debug problems is hard problem. Building a large, complex, reliable application in ANSI C is hard problem.
I try imagine, what would you rather spend your time on: coffee breaks or debugging complex bugs?
This is not that easy as it seems.
Rust compilation is not embarrassingly parallel as a lot of what's going on seems to be deferred at the link stage.
Modules inside a crate can't be compiled in parallel because of some reason I don't truly understand, please educate me.
Crates can be truly compiled in parallel, crating projects with hundreds of crates is quite a pain for other reasons.
The result is that a change single line of code in the project I'm working in can take more than 2 minutes on my last gen MacBook pro.
I have a M1 Mac mini where the compilation is twice as fast but I don't have enough RAM there and I'll have a hard time to convince my manager to make an exemption to my companies laptop refresher policies "because of rust".
If I have to take a coffee break every time I need to wait for an incremental compilation my heart would explode.
My current solution is:
1. Multitask. Do somethings else while I'm waiting.
2. Split the code that I'm iterating on in a new project. With minimal dependencies and copy the code back in the main project when I'm done (or just import it as a crate if it makes sense)
Both options suck and I wish they weren't necessary. Please let me know what else could I do. What hardware should I buy etc
(I already use zld on Mac and use minimal debug symbols, iirc debug=1)
MTP is terrible[1].
[1]: https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Perfor...
I've had multiple experiences of issues on the device side causing the entire explorer.exe process to crash!
Explorers handling of the MTP protocol is not resilient to badly behaved devices, would not at all be surprised if there are security implications where a badly behaved MTP device can get an RCE in explorer.
It's sad that with cloud being the solution for everything those days, this will probably never be improved within next decade.
The same could be true here where you're moving from a zip file to probably the same filesystem the zip files is in; if removing a file from the zip file is actually an in-place move data then truncate. The problem, of course, is that removing a file from the zip file is tremendously expensive. Reading the file with one syscall per byte doesn't help (especially post-Spectre workarounds that make syscalls more expensive).
But ideally I’d want the system to delete at the end if possible, and to otherwise delete as needed, instead of either doing only all at end or only after every single file.
Wait, does that mean that a file which is larger than half of the partition in fat32 cannot be moved? Or not even be renamed?
These determinations can be made with few tests each. Start with some reasonable amount of data, keep doubling it until the test fails. Continue binary-searching between last success and last failure.
The results may come out surprising. Performance does not scale linearly - just because your program takes 1 second for 1 unit of data, and 2 seconds for 2 units of data, doesn't mean it'll take 20 seconds for 20 units of data. It may as well take an hour. Picking a threshold far above what's acceptable, and continuously monitoring how much load is required to reach it, will quickly identify when something is working much slower than it should.
In practice we only batch when it starts hurting. It doesn't hurt to delete files one by one on a normal file system. It's made for that. So the API wasn't "many by default" and that's how it works for zip files as well.