Exo Research CTO still does not understand how OS file caching works in Windows
exo-blog.blogspot.com
exo-blog.blogspot.com
To the people who said the Windows source code is needed to understand how it works, hogwash. The file caching in Linux has been doing it this way for years. Windows does it for years as well. It's fairly standard OS caching technique.
Estimating real memory pressure is tough. There's the working set; but the working set may expand when you have sufficient physical memory, even though the marginal performance gains may be small. Similarly, the file system cache may expand when you have sufficient physical memory, but the marginal gains become small there too.
Probably the best unified metric would be the amount of time applications spend blocked on I/O, where that I/O is either paging committed bytes in from disk, or data that would be cacheable - and would have been cached - on a system with more memory. But that's extremely hard to predict.
Committed bytes is completely the wrong metric to appeal to, since it includes pages that have been committed but have never been touched. Windows reserves space for such pages in the page file (it never overcommits), but saying that it leads to "memory pressure" is ridiculous.
Please explain, for the (majority?) of us who don't spend time at the Ars Forums
Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns.
As to the Ars forums, with a brief perusal it seems similar to the programming Reddit - i.e. a lot of opinionated and frequently misinformed noise, interleaved with a small amount of signal. It looks like a time sink, to be frank.
But to give Peter his due, I registered with the forums and scanned through his past posts (DrPizza), and I can't find much that I disagree with; I particularly agree with him on the consolification of PC games, and the lack of MW2's appeal to a mature gamer, etc.
edit: and do you think he doesn't know about procexp/perfmon? :)
By bringing the person into this, you turn it into an irrational, emotional thing, where facts don't matter. That's not where you want to be.
And at the risk of extending this thing far beyond its worthy lifetime, that comment you link to proves very little. As I already said:
> Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns.
Allocating a huge bunch of memory and not touching it is one statistical pattern, but it's not particularly common, which is key. It's an outlier, so it isn't strong evidence. On average, committed bytes can be interpreted as a proxy for memory pressure, but like I said, there's likely to be a fair amount of variance there, so it's a poor proxy. There certainly wouldn't be a 1:1 relationship between committed memory and physical memory pressure; if I had to guess, I would guess somewhere like 60% of committed memory should ideally be in physical ram.
For example, my machine has 12GB of RAM, 3.5GB committed bytes total and 2.5GB working set total. But that working set includes both private and shared working set pages - calculating the active working set properly would require dividing shared page sizes by the number of copies in different processes.
But just for fun, I plotted the log of working set values in MB along the X axis, and the log of private bytes in MB along the Y axis, for each process, and this is what I got:
http://imgurl.filetac.com/img/68343980.png
Looks like a pretty close correlation to me - yes, there are some spikes in private bytes, but on average, private bytes is a proxy for committed bytes (I excluded shared pages for confounding reasons, as they are usually things like EXEs and DLLs - not counted against commit charge), committed bytes is a proxy for physical memory usage, which in turn is a proxy for physical memory pressure.
http://blogs.zdnet.com/BTL/?p=31024&page=1
FWIW, I never downvoted you - I actually upvoted you when I saw you dropped to 0.
An operating system will typically not swap out (or in) cache memory to disk, so, as long as you are exhibiting no swapping behavior, (And you have swap enabled) then you have enough memory. I think it was Adrian Cockroft, in fact, in "Sun Performance and Tuning: Sparc & Solaris" who in fact said that was _the_ way to determine if you have memory pressure on your operating system - run vmstat and watch the si and so column for activity.
Also, predictive prefetching may increase paging in, but in an async fashion, so the application is never actually blocked. This may cause spurious increases in the stats. You need to examine the amount of time threads are actually blocked by the paging behaviour.
Does it _feel_ fast?