Can anyone explain why Windows is so bad at copying files?
andrewducker.livejournal.com
andrewducker.livejournal.com
Show me the same effect with actual timed runs. If it still shows that effect then it'll be interesting.
Now, consider that calling read() on a socket that's really an SMB share is a complicated, multi-round trip action. The read must involve some locking in case someone else tries to write to the file in between, and probably a ton of other stuff. In my experience, even doing something like opening a 5k php file in vim from a samba mount can take a few seconds. That overhead is getting paid far more often in the network -> USB case than it is in the network to HD case.
That seems like a very reasonable hypothesis. I'm not sure how MS could solve it without using arbitrary amounts of memory or caching on the HDD, which would be a really bad solution, IMO.
* Variability in flash drive speeds.
* Variability of USB implementations. For instance, a 2008 MacBook is less than half as fast as a 2010 MacBook Pro, and both were slower than Linux running on a Dell.
* Most USB drives use awful filesystems like FAT16 and FAT32 that are inefficient when it comes to any complexity, as far as both space needed and speed.
There are so many little Windows gotchas. However after he explains them, it makes total sense (or at least you understand why it is that way).
There is something seriously wonky in the file access, network and (!) audio stack within Windows.
A good example is the automated wireless network search that runs every 30 seconds or so. If you have somewhat flaky (like, standard Atheros) wi-fi drivers, there's no way you will be able to listen to a song without interruptions and distortions. There are tools that can disable the wi-fi search, but good luck finding this solution if you are a newbie.
The same goes with copying files over the network while playing an mp3 file at the same time.
I wonder how xkcd would draw the same comic about the new advanced file copy of Windows 8, where you can see performance graphs and stuff.
To sum up, flash disks can only access data by big chunk of say 32KB. So if you have 32 files of 1KB each, the (dumb and not optimized) flash disk driver will overwrite 32 times the 32kb block where the 32 files will be stored. In this example, 1024KB (32 x 32KB) of data is written instead of just 32KB. So it slows you down 32 times in this case.
As you can see, the bigger the block and the smaller the files, the slower the transfer.
If you just want to make the blinking light on the drive accurately show when it's safe to unplug you can easily delay writes for at least a quarter of a second.
Another folder on the same network share is 5.96 GB worth of data in 2795 files across 190 directories. According to the detailed dialog, it is estimating "1 day" to finish copying despite the fact I'm getting sustained transfer rates greater than 5MB/s and it looks to be about 20% complete after a several minutes.
I believe that by default, windows disables write-caching for USB devices. Somewhere in the device manager, there should be a choice between "optimize for performance" (enable write caching) or "optimize for safe removal" (disable write caching).
NT is a lot like linux (no, really), external hard drives and flash drives are represented as SCSI drives exactly like local SATA disks (think /dev/sda for your root and /dev/sdd for your thumb drive).
the big difference is in the cache manager. windows recognizes that "surprise removals" of thumb drives is a lot more likely than "surprise removals" of fixed disks and it adjusts the aggressiveness of the cache manager accordingly (this is controlled via "device properties->policies" from the device manager).
in the client case (copying something onto your flash drive from a CIFS server) there shouldn't be any difference to the FS/IO stack than copying it from your desktop via explorer. the server case is a little different, srv.sys does writes a little differently than usermode clients, but...
Also, some utilities can help somewhat, depending upon specific circumstances:
RichCopy
http://blogs.technet.com/b/keithcombs/archive/2009/03/22/ric...
There's another popular utility whose name escapes me at the moment. However, its author has insisted on continuing to use a package format that some anti-virus programs flag due to high use of the format by mal-ware authors. I've also found it to behave incorrectly in some circumstances, in the past.
This is why windows always feels laggier than other OSes. Windows 7 also increased the processing latency of the OS.
What does "the entire OS has to do a lot of system calls" even mean? I don't think you're talking about the kernel, because the point of avoiding system calls is avoiding a user/kernel transition (something Windows' I/O completion ports is good at, IMO the reason its async I/O implementation is fairly solid). But it leaves open the question of what it is you are talking about; is the problem Explorer? The shell? The Win32 layer?
Windows, for me, does not feel laggier than other OSes. In particular, Linux has always felt substantially more laggy for mouse input. Linux has long given me the impression of having an egalitarian scheduler, giving equal weight to user interactive apps and background processes when they have the same priority; but Windows bakes interactiveness into the scheduler algorithm as another input alongside priority. I haven't used OS X long enough to give a solid opinion on it, but it seems about as responsive as Windows and almost certainly better than Linux.
laggy video - I couldn't get Windows 7 to play any kind of video smoothly on a GT 450 and 9800 GT at 720/1080p on an i7 with w7 64bit; you think you can play it smoothly, but then I play the same file on linux box and it's so smooth you'd think there's no such thing as frames
if you kill ALL your apps (including services and background apps which are part of your apps) and you check how many requests per second to the registry there are, you will have a shock; I did this back in the XP era and it had like 50-100 registry requests / second.
if you leave a W7 machine on for more than 24 hours and you have more than 6-8GB of RAM, you will have degraded performance even if you close the apps (memory leaks and/or memory fragmentation, maybe other issues)
As for Linux, it's been quite laggy and buggy before in the aspect of responsiveness, but it seems to have improved in those areas recently.
Mac OS X is doing fine too so far.
edit: Let's not forget game input lag. Also, some CPU intensive games perform better when you set the priority of the processes which are 100% idle in 99.99% of the time to idle. I am talking about things like print spooler, app notifiers & tray area apps, updater apps and others like these.
If linux seems slower in the graphics department, keep in mind it's got an X server, apps which render in that X server and that brings quite some overhead. Even though this is how things are, Linux is pretty close to "being there", pretty much like OS X is.
Windows also lets processes starve the entire OS completely. It's not ok to have resources be eaten up completely so that the user might have to wait even hours until a task is done so that CPU usage drops below 100% across 4 or 6 cores of a system. This isn't normal. Also, Windows can give you high latency and bad performance even when there are more than enough resources: CPU & memory.
Therefore, I'd say that the Windows scheduler is closer to the treat all processes equally and / or sink more time into idle processes with more idle threads than into processes with fewer threads and more active threads model.
Linux hasn't left me with this impression recently and OS X hasn't left me with this impression since Leopard / Snow Loepard.
Video is fine and clear for me across a much larger range of codecs than I've ever seen on Linux (or Mac, for that matter). I normally get no more than 2% CPU utilization for video playback; any HD content I have is hardware accelerated, I expect.
50/100 requests per second is meaningless without knowing the cost of those requests. My machine, if you ignore superscalar architecture and hypethreading, is capable of over 12 billion operations per second. Unless these registry reads have fairly substantial costs, they're completely unnoticeable.
My Windows 7 machine here currently has an uptime of 48 days. I seldom reboot until the Windows Update nag gets the better of me, and I frequently turn off the update service to get rid of the nag. I have 12GB of RAM, and I've never seen this degraded performance you speak of.
I wrote my own print spooler (it actually runs on a different machine) and have a bunch of tray applications: CrashPlan, DropBox, Process Explorer, corporate VPN, occasionally Skype and Steam. Again, I've not noticed any performance drain from these things. Not even games these days are CPU constrained; with 4 cores (or 8 if you include hyperthreading) the only thing that taxes my CPU is video transcoding. There's plenty of spare CPU capacity in games; they seldom get above 30 or 40% utilization.
If you really think Windows is odd in starving the system for non-low-priority runaway processes, try running a forkbomb without ulimits in place on your Linux or Mac machines. But I have no difficulty running Handbrake or SuperPI or similar utilizing 100% CPU but still having responsiveness enough to shut them down etc.
It's extraordinarily rare for me to find myself in front of a Windows machine where the UI is completely unresponsive except in cases of driver conflict or flaky hardware (e.g. a paging error from a bad hard drive will do horrible things). I don't know why you've soured so much on Windows, or what bad experience it was that pushed you over the edge, but when you bring up things like registry traffic and this "event 500/501", I get the impression that you are actively looking for things you think are worth complaining about, rather than pointing out things that are actually hampering your interaction. It hurts your credibility in my eyes, frankly.
(FWIW, I was a computer technician before I became a software engineer, and I've been working with Windows of all varieties since about 1993, including the early (3.1 and 3.51) versions of NT (almost 20 years, scary!). I've dealt with thousands of machines running Windows. And for MS OSes based on NT, problems have overwhelmingly been because of hardware or driver issues, almost never because of OS issues.)
really? can you provide some analysis here? I've done a bit of hacking on I/O layer stuff in both linux and NT and I haven't seen anything that would lead to that conclusion ...
Take a machine which boots the OS and check how much stuff gets on the screen and if you can interact with it while the HDD led is at full brightness or blinking heavily.
Take an i7 with 8GB of RAM and one 64MB WD RE4 drive and see how much that stalls compared to OS X on the same hardware...
How come the cost of accessing the disk for reads / writes is greater for performance when compared to OS X? Why is Linux able to get stuff done and displayed on the screen while doing disk IO? I am guessing it's trying not to block as much due to starving other threads which want to read / write
Notes: I've been using Windows since 96, Linux since 2001 and Mac OS X since 2004. I don't claim to be an expert or anything, nor a fan of any OS, I just made some observations over the time and wish that all the OSes improve in one or many areas.
>Windows 7 also increased the processing latency of the OS.
Huh?
The most likely answer is that the USB stick itself is pretty slow at writing incoming data into its internal flash modules. He also ought to test writing to a USB hard drive (with the physical spinning spindle inside) and I'd be surprised if his throughput would be less than 100Mbps even with a crappy drive.
You can change the defaults in device properties, if you want caching on USB stick.
I can show you an USB stick, that is slower than 2 MB/s. :(
That's why there used to be performance problems in apps with fsync().
I've been bitten a couple times before by pulling drives on linux that weren't being written to, but still had a cache waiting to be written. I learned my lesson about not dismounting on an OS configuration where it isn't safe, but disabling caches is not the only way to be safe.
It might take 10 seconds to flush the cache. That's okay. The key is that you never go more than a quarter second without writing to the disk until the cache is empty.
If he is getting radically different times for Network->USB vs. Harddrive->USB, then it's probably not a USB issue.
As for the different OS... I mean, he isn't posting this to HN, so he may never have even used a different OS in person (most "normal" people I know haven't. Yes, including OSX).
His two tests look like this:
1. Network -> USB
> I'm copying files from a my networked hard drive to a USB stick.
> It tells me it's going to take about 7 hours. For 3 Gig of data.
2. Network -> Local HDD -> USB
> I copy it from the networked drive to my C drive. This takes about 4 minutes.
> I then start copying it to the USB stick. This is varying between 8 minutes and 45 minutes, depending on whether it's copying a big file (fast) or lots of little files (slow).
Test #1, network directly to USB, is showing a 7-hour time estimate.
Test #2, network to HDD to USB, is showing an 8-45 minute time estimate (truth will be somewhere in between).
This is not a generic complaint that "Windows I/O is slow", it's a complaint that the obvious method (straight from network to desired USB device) is many times slower than the less-obvious method of hopping through the local HDD for no good reason.
In my tests, the time of "a lot of small files" is dominated by the USB stick slowness (800 files totaling 5 M took 80 sec from the HDD and 90 sec from the network share), and the time of "copying a big file" gets two times slower when the sizes of write blocks to USB stick are not "convenient" and that happens in the "network to USB" case (130 MB for 7 sec from HDD, 14 sec from the network share).
Note that Windows by default doesn't cache writes to the USB stick; specifically, the scenario "plug out the stick without unmounting" is supported, whereas the last time I've checked I had to unmount USB sticks on Linux to be sure that everything is properly written.
So once you don't cache writes you're at mercy of the USB stick, which is reasonably fast only for one specific block size (specific multiple, actually). When you don't get the best block size over the network, you'll get slower writes to the USB stick.
What I see in my tests (on Windows XP) is that the copying is done by CopyFileEx API, which uses blocks of 65536 bytes (64KB) from the local HDD but uses blocks of 61440 bytes (that's exactly one 4K block less than 64KB) over the network, which is then slower on the USB stick.
Conclusions:
a) Don't copy small files to the USB stick on Windows if you need speed. It's hundreds of times faster (when the bytes are counted) to copy an archive of the files vs. a file by file.
b) If you want efficient copying of bigger files from the network share to the USB stick, avoid CopyFileEx API function, at least on Windows XP, use the software which writes in 32K multiples.