Hacker News Reader

Comment by jacobgorm | Hacker News Reader

But why would Chrome's setup.exe need a 32MiB working set? If all it does is download and unpack some files in the background, it would seem like something that is doable in 32MiB or less.

The Windows BACKGROUND mode is useful for stuff like virus scanning or database log compaction, where you want to make some progress, but only if you can be certain not to hurt any foreground workloads. Back when we had hard drives, this wasn't actually trivial to achieve, because it is very easy to interfere with the foreground workload just by doing a few disk seeks every now and then.

I agree the 32MiB working setlimit is somewhat arbitrary and should be documented, but Windows is full of these arbitrary constants, like the 32kiB paging chunk for instance.

My recommendation for Chrome would be to stick with the background mode, and fix whatever problem is causing the working set to exceed 32MiB.

View on HN

> The Windows BACKGROUND mode is useful for stuff like virus scanning or database log compaction, where you want to make some progress, but only if you can be certain not to hurt any foreground workloads.

But you are hurting the overall system performance, or battery life if this is a laptop and you keep thrashing the working set for no good reason. I don't see how this proactive approach has any advantages over just swapping out the process first if you run into actual memory pressure, or maybe start paging out pages of that process if they've been idle for a few minutes if you insist on some kind if proactive measure.

The background mode is not hurting anyone, it is the thrashing process that is hurting itself by using too much memory. This shows up as high CPU but, because it is in the background, the rest of the system performance should be largely unaffected. Anything with foreground priority will still be scheduled to run whenever it needs to.

Had the background process been allowed to keep growing its working set then foreground applications would be forced to page out, defeating the goal of having a background mode in the first place.

But the limit of 32 MB is arbitrary and undocumented which is a huge problem.

Doing useless work nobody asked for is always a problem. People use systems with batteries and they would appreciate their system not doing that.

The foreground applications will still be paged out, only now they will be paged out for longer.

Without background mode setup.exe would have a 48 MiB working set, for a short period of time.

With background mode setup.exe has a 32 MiB working set, plus 16 MiB of its memory is bouncing back and forth between the working set and the standby list.

Thus, none of its memory will get paged out and foreground apps will get paged out, and this will continue for 250 times as long.

The foreground processes will always get scheduled ahead of the background process, and be able steal its evicted pages from the Standby list, thus reducing their own need for paging to external media.

If all you care about is getting your updater to complete quicker, you should not be using the background mode, but ideally the Chrome updater would run in the background without affecting the performance of the foreground applications. Allowing for working set peaks, no matter how short, will fail to achieve that on systems experiencing memory pressure.

The foreground process will only get scheduled ahead of the background process if there is a shortage of CPU power. On today's many-core consumer machines it is much more likely that the background process will continue to make progress, with the only thing slowing it down being the repeated page faults.

Counting on CPU starvation to save memory seems like a very fragile solution.

I agree that commit peaks can affect foreground performance. I've filed bugs for an ephemeral 480 MiB bug in Chrome (caused by an errant product image), and I blogged about an errant 4 GiB allocation that wiped out the disk cache on my 8 GiB machine (https://randomascii.wordpress.com/2012/09/04/windows-slowdow...).

But treating working-set peaks as synonymous with commit peaks is problematic. I still believe that trimming of working sets will rarely save memory, and a working-set cap will virtually never be better than occasional trimming. The cap fails to address the higher-than-the-cap memory consumption in many cases, it just makes it more expensive.

> This shows up as high CPU but, because it is in the background, the rest of the system performance should be largely unaffected.

And, as said, draining your battery for no good reason.

> Had the background process been allowed to keep growing its working set then foreground applications would be forced to page out

Why would they? As said, you could still page out the processes running in background mode first if you actually start running out of memory. And if after trimming all the background processes to 32mb you still don't have enough memory, you would be in the same situation either way.

The current implementation has absolutely no advantage.

What's wrong with using 32 MiB of memory? That's a relatively small amount, especially if you're trying to patch a 225 MB DLL.

And I would argue that the 32 MiB is worse than arbitrary - it is pointless. I have been thinking about this for a while and I cannot think of a situation where it makes things better. It wastes (a lot of) CPU, and I claim that it doesn't actually save memory compared to letting the balance-set manager do the trimming.

Low CPU priority and low IO priority are important for being a polite background process. Not using too much memory is important. But a working-set cap fails to achieve that last goal.

Replying to myself, because this is important:

In what scenario would be 32 MiB working-set limit, enforced when pages are swapped in, be more effective at saving memory than a similar limit enforced by the once-per-second balance-set-manager?

Obviously trimming the working set at swap time does a better job of reducing the working set, but that's not what matters. What matters is saving memory. If you are trying to save memory then trimming once a second is just as good, and far more efficient, than trimming on every page fault.

But, trimming at all on a 64 GiB machine with 47 GiB free is just silly. It doesn't make sense to spend lots of an expensive resource (CPU time == electricity == battery life) in order to save a resource which you aren't even fully using.

I think what mattered to the people who designed this is not create memory pressure for foreground applications. Anyone who ever tried using a WinXP machine right after boot knows it would need a couple of minutes just to sort itself out from the many auto-starting processes waking up to big pile of new "background" work and fighting over the disk and memory, and I think that was what they were trying to repair with background mode. Windows has a long history of getting a bad reputation due to buggy 3rd party pre-installed software, so they built a rather strict performance isolation sandbox to address that.

Something that only runs once per second is not going to be able to keep up with with a process that is dirtying memory at full tilt, which is probably why they chose to enforce a hard limit by evicting the least-used pages to the standby list, from where one can usually get them back quickly without encountering a hard fault to disk (a soft fault was probably like 10,000x faster than a hard fault back when they did this work). They must have viewed thrashing as a pathological case that programmers would diagnose and repair, just like you have. After all they do provide some pretty good tools for diagnosing memory use.

You may argue that the 32MiB limits needs adjusting to follow Moore's law, but Moore's law stopped working for laptop DRAM sizes many years ago.

That all makes sense (although I think this feature appeared in Windows 7 rather than XP), and I certainly understand the value of a background mode.

But does the working-set cap _work_? In that scenario where you've got heavy memory pressure I still think it doesn't.

If the background process is typically touching less than 32 MiB in a second then a per-second trimming could reduce the working-set effectively.

If the background process is typically touching more than 32 MiB in a second then the fault-time trimming doesn't work because while it trims the memory from the working set, there is no time for the memory to be paged out, and if the memory did get paged out it would make it even worse because it would need to be paged in. So, CPU (and perhaps disk) overhead is increased, but memory pressure remains the same.

The problem (a process touching too much memory) is real. However the solution does not work. A working-set cap doesn't actually reduce memory pressure on foreground applications any more than a per-second trim.

Yes I meant that XP had the problem, and that Win7 tried to remedy it.

You are overlooking that the read-only part of the working set can be released immediately once another process needs those pages -- just zero them and add them to the free list. Only the dirty pages need to get written out to disk.

Anyway, it would actually be simple to test the effectiveness of the working set cap by writing a program that allocs and dirties memory aggressively, and then running it either in normal low priority or background modes, to see how it affects overall system behavior when running in each mode.

My test program already proves that if you alloc-and-dirty memory aggressively then in background mode you will consume much more CPU time. If your metric is "interfering with foreground processes" then this is fine, but I think the slowdown is severe enough to matter.

As for saving memory, I see your point, but I still don't see how the cap would be better than per-second trimming. If the clean-then-zeroed page is not touched again by the background process then either method would make it available. If the clean-then-zeroed page is touched again by the background process then the whole process of removing it from the working set, zeroing it, then reading it back in and faulting back in is a waste of time. So, again, I'm struggling to find a scenario where the cap is more effective than once-per-second trimming.

It actually does a lot more than you think.

For example it does pretty interesting diffing - Chrome downloads a patch and uses the previously left-behind copy of the installer to re-build new binaries from the small diff it pulled. See more:

- https://chromium.googlesource.com/chromium/src/+/HEAD/compon...

- https://www.chromium.org/developers/design-documents/softwar...

I realize that they do binary diffing, but it was not clear to me if setup.exe was only the thing that runs on first install, or if it keeps running. If it keeps running they should definitely try to keep it in check.

> Windows is full of these arbitrary constants, like the 32kiB paging chunk for instance.

I assume you’re referring to the 64KiB allocation granularity? That’s actually not at all arbitrary: https://devblogs.microsoft.com/oldnewthing/20031008-00/?p=42...

Huh, I never knew that particular detail, and I’ve played with Windows quite a bit. But, with my modern OS developer hat on, I’d say it’s a bit lazy, not arbitrary.

> So if allocation granularity were finer than 64KB, a DLL that got relocated in memory would require two fixups per relocatable address: one to the upper 16 bits and one to the lower 16 bits.

…

> Forcing memory allocations at 64KB granularity solves all these problems.

It does, at the cost of silly limitations. A much nicer solution IMO would have been a way to ask for a 64kB aligned allocation for DLLs.

What advantages would you get by being able to reserve memory with a finer than 64-KiB granularity? Especially with 64-bit Windows it seems quite unnecessary to have finer granularity. Am I missing something?

64-bit is recent and 64kB granularity is ooooollllld.

With 64 bits, my real objection is that it’s unnecessarily complicated. There’s some data structure that maps addresses to whatever is mapped at those addresses, and on Windows it also needs to track reserved memory, and it needs to do so at a different granularity. (For huge pages, additional granularities are needed.)

By paging chunk I mean the amount of data that gets read in by a hard fault to disk.

Your recommendation is to stick to a pathological implementation where every page fault results in the equivalent of doing a Java GC run? All independent of the actual memory pressure?

This isn't even about the exact limit, this implementation just makes no sense. Add a hysteresis at least because running this on every page fault is guaranteed to cause a lot of CPU load while not returning very much to the system.

All it takes a third-party dll (antivirus, antispyware, or just some other big brother hook) to be part of your executable, and the memory is gone. Or call to some Win32 library (like shell32.dll), and some caching going on there. 32MiB are nothing nowadays.

Just a callstack can be 1MB easily, granted it's RESERVED most of it, just some parts commited, but it can grow there (like directory recursion). Then you have 3 or 4 ThreadPool threads created just like that, etc. etc.

The offline installer reports as 103 mb, depending on the compression settings, it might need a significant amount of that mapped in at once.

Probably because the files to be binary patched are now quite large, and are loaded into memory verbatim. Perhaps they could memory-map them instead, and ensure that their binary diffing tool works from left to right to avoid thrashing?

How do you know it's not memory mapped?

Working left to right means you have to force the compiler to always align the code in the same order, which sounds difficult and like a waste of time.

No.

If you want both the input and the output to go left to right, Yes.

If you only make the output go left to right, you don't actually save much memory. You're still randomly accessing a huge buffer.

In general using a 32 MB or larger compression dictionary really improves compression size.

Not sure if that's why the Chrome installer uses 32MB+.

Reply on news.ycombinator.com