Byenow – A command-line Windows utility for faster folder removal | Hacker News Reader

Byenow – A command-line Windows utility for faster folder removal

iobureau.com

77 pointssoheilpro33 comments

This is mine. I was meaning to do a Show HN post for it actually, just was waiting for the weekend :)

I've been using this internally for several months. For local deletes YMMV. There seem to be some sort of central lock in the NTFS driver, so the speed up comes from being faster with scanning and generally idling less. For freshly mounted fast drives you may see a speed up of 1.5-2.5 times, but in other cases it will be just 10-20%.

However for remote deletes this tool is really great. Due to the network latency, single-threaded deletes will spend most of their time idle - a request needs to get there and then back before the next one can be fired. But using threads allows to have multiple requests "in flight", forming a queue if needed, so the remote end is always doing something. A typical speed up for a LAN is around 2x, and for VPN links this should be even higher.

In any case, AMA.

will it delete files even if open in another process? this is my main bugbear with windows file deletion: it won't delete it and you need special tools to track down the process that has the file open

Depends on how they are opened.

If they aren't opened in FILE_SHARE_DELETE mode, then they will in fact be locked against deletion... because that's exactly what file's current user wanted. Force-deleting a file in this case is not the wisest thing to do.

[1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/n..., Ctrl-F for FILE_SHARE_DELETE

Which is the right thing to do and the approach taken in all non-UNIX clones.

In any case, since ages one does not need special tools, resource monitor listing of open handles is enough.

It's the right thing to give an error that some process has the file open but never tell you which process that is?

It might not be the most user-friendly way, but sure tops removing the file altogether.

That’s an UI issue with the tool.

The underlying behavior is correct though - people with open handles have the priority unless they have explicitly stated that deleting a file while they use it is OK.

What tool? That's windows' behavior. I would agree that locking a file and preventing deletion is good or possibly queueing a file for deletion when it is free, but windows does not handle this scenario well.

Telling you what process has a folder or file open should be added to windows explorer. It’s hard to explain why it isn’t.

A workaround to this is to use Process Explorer[1] to search for the filename, then kill the process that has a handle with that file name.

[1] https://docs.microsoft.com/en-us/sysinternals/downloads/proc...

Thank you for sharing. You write:

> The source is incomplete, because it depends on several libraries that are not a part of this distribution. However, with an exception of simple_work_queue, these libraries are simple Win32 API wrappers and their functionality should be fairly obvious from their usage.

Just curious - if they're trivial wrappers - why not share them?

A couple of reasons.

It takes time to prep a public release. Review the sources, comment them, check formatting, add docs, legalese, build instructions, etc. I really don't like publishing stuff that's not in a proper order.

Second is that a public release needs to be maintained and this complicates our own workflow. We can, in theory, do a release and then abandon it, but it doesn't seem like a right thing to do.

First of all, great piece of software thanks for sharing it. Secondly re: releasing then abandoning some subset of code being “not the right thing”, I would submit that as a user I’d much prefer abandoned code than unreleased code. Please consider just slapping a disclaimer on it and releasing it.

One of these days I hope to get a peek at how you build UIs - you've got some of the best looking software for Windows.

Thanks, appreciate the compliment, but there's not much to it really. Do version A, use it, change what's off or annoying, rinse and repeat. At some point it always converges to a stable point.

Pretty much the only criteria is to remove friction from fast paths and add some where safeguards are needed.

I always assumed that a delete command would execute at max disk capacity on a single thread and that multi-threading it would even slow the command (more random accesses on the disk).

This utility would have been a godsend a couple months ago when I had millions of XML files to delete on one of our server (12 years of accumulation). It took several days (it's an old Windows 2003 server with SCSI disks I think).

It always amazed me that deleting a remote directory with thousands of files can be so much slower than remoting into the server, and deleting the directory. I don't want to recursively list the remote files and then delete them one by one. I just want to remove the directory from the remote filesystem!

Can a tool solve that problem? (I guess powershell remote scripts could?)

There's no way to delete multiple files with a single syscall. There's also no way to purge the folder of all its contents. At least not on Windows, using an API.

The syscalls are quick enough when done locally. What I think is going on is that for a directory with N files on a share, I get N (or likely multiples of N) network roundtrips.

If the network round trip time is 1second and the time to delete the directory with 1000 files is 1 second when done on the local drive, then I want the delete on the network share to be 2 seconds not 2000 or 4000 seconds.

For deleting - yes, you will get a round-trip per delete request. So if they are serialized (which is the case with standard tools), it will be excruciatingly slow indeed.

For scanning - you'll get the list of content in chunks. The size of a chunk depends on the size of the buffer you provide. For smaller folders it'll normally fit in a single request.

(I'm just a hobbyist)

Is that how .+n[i|u]x's unlink command works? IIRC you can unlink a directory.

If that's true, how does MS's WSL handle unlink?

Why doesn't Windows del work like that, surely at the physical layer the drive is only removing allocation entries?

unlink(2) returns EISDIR (on Linux at least) if you try to unlink a directory. I'm not sure if you can still do it, but long ago you could unlink(2) a directory as root, but it was not a good idea since it did not recursively free up the contents back to the file system's raw storage.

To remove a directory on Linux you have to empty it out (recursively, using unlink(2) on files and rmdir(2) on empty directories) and then rmdir(2) the top level directory.

So not a single system call.

Thanks for humouring me, everyday a learning day - especially when there are kind people happy to share their knowledge ;o)

FYI builtin command "RMDIR /Q/S foldername" is fast, at least much faster then using windows file explorer.

It's mentioned on the homepage.

There's also "del /f /s /q" to delete just files, followed by "rmdir /q /s" to delete just folders; and there's also "robocopy /mir" with an empty folder as a source. From what I've seen straight "rmdir" is the fastest of these three.

just "RD Foo /S" if you know you want it gone why all the extra flags and steps

Now I don't know is this makes things slower, but Windows File Explorer by default never deletes files. It just moves them to the recycle bin.

If you want to bypass that step you need to use the Shift + Del keys and for that scenario Windows even warns you the action you are request is a permanent delete.

Came to say the same. `rmdir /s` is an order of magnitude faster on windows than other methods I have tried.

byenow node_modules - let's see if it can keep up

This was my test actually. Because that is the exact reason that I would use this for.

And it was spectacular. I'm amazed at the speed at which it clears out an entire folder.

Another nice touch is if I ask it to delete a folder and that folder is open in a command prompt somewhere, it just goes ahead and deletes it's contents cleanly.

Excellent piece of software.

Is there any difference to npm rimraf? This is removing the node modules folder in lightning speed

Btw is there a cli trash utility for linux that handles multiple partitions correctly?

Funny that you need a third-party tool to do that, isn't it.

Reply on news.ycombinator.com