Behind the Windows 7 memory usage scaremongering | Hacker News Reader

Behind the Windows 7 memory usage scaremongering

arstechnica.com

55 pointschmike20 comments

Kind of reminds me the first time I started freaking out over the "Memory Bug" on my Solaris 2.6 install in 1998. So many people got worried by that data, that they modified several of the system binaries to report "filesystem cache" as "free" in later versions of Solaris.

What's old is always new, again.

Linux makes it (reasonably) clear:

   shephard@u804:~$ free
                total       used       free     shared    buffers     cached
   Mem:        515440      39976     475464          0       2684      16452
   -/+ buffers/cache:      20840     494600
   Swap:       192740          0     192740

   shephard@u804:~$ dd if=/dev/zero of=/tmp/475MB bs=1048576 count=475

   475+0 records in
   475+0 records out
   498073600 bytes (498 MB) copied, 73.7244 s, 6.8 MB/s

   shephard@u804:~$ free

                total       used       free     shared       buffers     cached
   Mem:        515440     508312       7128          0       1760     475756
   -/+ buffers/cache:      30796     484644
   Swap:       192740          0     192740
   shephard@u804:~$

Many people don't know how kernel caches and buffers work. Even many developers get confused by this and think that a system is almost out of memory. Luckily someone created http://www.linuxatemyram.com/ - that saves me time to explain it over and over to people who panic about the issue.

I'm by no means fan of windows, but the article was an obvious troll from the beginning, as it was quite evident that they were mixing up cache with real memory usage.

What really surprised me was that such a pointless 'story' generated so much fuss. Waste of time.

> mixing up cache with real memory usage.

And how real is the use of memory for cache?

All this reminds me of a discussion I had with a mainframe admin in the mid-to-late 80's:

Me: Look at this numbers. We are using more than 90% of the processor time and about 100% of the memory. Is it normal?

Sysadmin: Good. We paid them in full. It's better if we use the whole computer, not just part of it.

In some cases, the sysadmin is right. Say you are concerned with performance, not energy-efficiency. If you have an I/O-bound workload that is only using 50% of your CPU cycles, wouldn't it be nice to automatically use the other 50% (otherwise wasted, mind you) to figure out some other things you might need in the future and prefetch them? On the memory side, what good is that 1GB or unused RAM doing you? If you need more for your application, the OS can drop its cache on the floor so quickly you'll hardly notice the latency.

Granted, this prefetching is not without downsides. DRAM banks have low-power states you can put them into if they're not being used, so if you can shove all of your used memory into a contiguous region of your physical address space and free up a DIMM's worth, you might be able to save power. This is unlikely, however, because the virtual-to-physical address mapping function is usually designed to spread the load very evenly across all available DRAM banks. One more likely source of power savings is from avoiding the CPU and disk activity of prefetching, especially if prefetch accuracy is low. Most people tend not to care about this though, and want the OS to make their app run faster by using otherwise-idle resources to do prefetching.

In addition to local use, this is also every distributed computing project.

It was an 80's mainframe. I doubt power saving was one concern. We had folks typing forms in data-entry terminals 24x7 generating the tapes that were then fed to the bigger machine.

But you have an interesting idea. I wonder if there is any OS that changes memory distribution and allocation strategy according to desired power consumption on, say, portable computers.

The Linux kernel supports, IIRC, plug-and-play memory and processors and it could, conceivably (it it doesn't already) power up and down unused parts of the machine in order to reduce power consumption and heat dissipation. Even reads and writes to and from memory could be grouped in bursts if that could save power.

For instance, right now my running applications are within half of the memory, both cores are running slowly and the ethernet interface is disconnected. One memory module, one core and a network interface could be powered off without perceivable performance degradation. Also, as much data could be cached in the remaining memory so the disk could be powered down too or, and that would be interesting, spun at a lower speed.

Lots of interesting ideas in one lazy saturday afternoon. That's what I like in HN. :-)

In the mainframe world you're often paying by the processing unit consumed -- if you're not using the cycles you're not paying for them, and if you're using cycles you don't actually need to use you're just burning money.

It's surprising that the CTO of a company specialized on system performance doesn't know how OS cache memory work.

It's not like you can read the Windows source code to see what it's doing, so I am going to give him a pass.

He could read the available documentation and literature. This is not some grand secret. This feature has been around in Vista for years, there's no excuse for this level of ignorance. It's just laziness.

http://en.wikipedia.org/wiki/Technical_features_new_to_Windo...

Better yet, the same article circa Feb. 2007: http://en.wikipedia.org/w/index.php?title=Technical_features...

Documentation is never accurate. No source code, no knowing-for-sure.

Any operating system developer who knows anything about paging would know exactly what was going on. Especially since you can immediately tell using the Windows task manager that it was doing caching.

He's CTO of a technology company, and should have this stuff down cold. No pass.

I think several people nailed this when the story broke, it really was a pretty superficial observation for such an apparently broad analysis.

Slightly off topic, but it just struck me that I'd like to see app icons in the task manager and windowed programs highlighted in some way. The unabridged list of processes is a little too lengthy if you're looking to quickly nip something in the bud. I know there are task manager replacements, but to me they're not as intuitive.

One for the Win8 suggestion box maybe.

I wonder if Windows 7 behaves like Linux in that by default, when faced with memory pressure, it will much prefer to swap out application pages rather than abandon buffer cache.

This is really annoying when this memory pressure is due to trying to fit more (and potentially useless) things into the buffer cache. Let's say you're performing a backup of a 100+ GB directory to another machine using rsync. Linux will want to jam that entire directory into buffer cache, which of course won't fit. Instead of immediately pitching out the existing disk cache, it will aggressively (depending on the value of swappiness) swap out application pages, (potentially owned by Oracle or MySQL) rather than lose existing buffer cache.

This is a real problem, and one that's a pain to fix.

No, Windows 7 and Vista always abandon the Superfetch cache before paging out anything else -- this is verified in the comments section. It's the sensible thing to do, since

a. Superfetch data is heuristic and application pages are mostly not.

b. Superfetch data always has a backing store (the actual file on disk), so it's really cheap to forget about it. This may or may not be true for application pages.

Could you clarify how the problem is hard to fix under Linux? If you set swappiness to 0, does it still swap out application pages?

Yes. Even with swappiness at zero, Linux will still swap when you don't want it to.

Why do people even care about what computer uses its memory for. What I care is performance, as measured by wall clock, and energy consumption.

Reply on news.ycombinator.com