brucedawson

My test program already proves that if you alloc-and-dirty memory aggressively then in background mode you will consume much more CPU time. If your metric is "interfering with foreground processes" then this is fine, but I think the slowdown is severe enough to matter.

As for saving memory, I see your point, but I still don't see how the cap would be better than per-second trimming. If the clean-then-zeroed page is not touched again by the background process then either method would make it available. If the clean-then-zeroed page is touched again by the background process then the whole process of removing it from the working set, zeroing it, then reading it back in and faulting back in is a waste of time. So, again, I'm struggling to find a scenario where the cap is more effective than once-per-second trimming.

That all makes sense (although I think this feature appeared in Windows 7 rather than XP), and I certainly understand the value of a background mode.

But does the working-set cap _work_? In that scenario where you've got heavy memory pressure I still think it doesn't.

If the background process is typically touching less than 32 MiB in a second then a per-second trimming could reduce the working-set effectively.

If the background process is typically touching more than 32 MiB in a second then the fault-time trimming doesn't work because while it trims the memory from the working set, there is no time for the memory to be paged out, and if the memory did get paged out it would make it even worse because it would need to be paged in. So, CPU (and perhaps disk) overhead is increased, but memory pressure remains the same.

The problem (a process touching too much memory) is real. However the solution does not work. A working-set cap doesn't actually reduce memory pressure on foreground applications any more than a per-second trim.

I'm running some monitoring that will tell me how setup.exe behaves on my machines. I tested it by forcing an update from Chrome. The private commit and private working set were 66.76 and 61.70 MiB, so no capping, but that makes sense because a user-initiated update isn't running at low priority, so we don't trigger background mode.

So, the only time the bug happens is for background updates that may be happening when the user isn't even present. And, since the CPU priority is low they won't harm responsiveness of foreground applications. But they will waste CPU time and electricity and battery life (if on battery). So, a silent waste.

What advantages would you get by being able to reserve memory with a finer than 64-KiB granularity? Especially with 64-bit Windows it seems quite unnecessary to have finer granularity. Am I missing something?

Replying to myself, because this is important:

In what scenario would be 32 MiB working-set limit, enforced when pages are swapped in, be more effective at saving memory than a similar limit enforced by the once-per-second balance-set-manager?

Obviously trimming the working set at swap time does a better job of reducing the working set, but that's not what matters. What matters is saving memory. If you are trying to save memory then trimming once a second is just as good, and far more efficient, than trimming on every page fault.

But, trimming at all on a 64 GiB machine with 47 GiB free is just silly. It doesn't make sense to spend lots of an expensive resource (CPU time == electricity == battery life) in order to save a resource which you aren't even fully using.

I'll just reiterate what is said later on. The API is documented. However this extremely important effect of calling the API is not. We called the API expecting lower I/O priority and lower CPU priority or something like that. We perhaps expected lower memory priority. Really the documentation should have spelled all of this out.

But in addition to this we got a problematic working-set cap which can absolutely destroy performance. That detail needs to be documented (and the rest of the behavior should be as well).

It's a good question. It could be that most people don't notice this things, those who don't don't report it, etc. Or maybe it only affects some people, but I can't think how that could be true. More mysteries

Yep, that is generally what Windows does, using available memory for the disk cache. In the user trace the system had lots of disk cache (available memory) and lots of _free_ memory, so there really was zero value in trimming the working set.

But, (undocumented) rules gotta be followed, so trim the working set it is.

Sorry, I don't own them so there's nothing I can do, unless I can buy them back

The foreground applications will still be paged out, only now they will be paged out for longer.

Without background mode setup.exe would have a 48 MiB working set, for a short period of time.

With background mode setup.exe has a 32 MiB working set, plus 16 MiB of its memory is bouncing back and forth between the working set and the standby list.

Thus, none of its memory will get paged out and foreground apps will get paged out, and this will continue for 250 times as long.

What's wrong with using 32 MiB of memory? That's a relatively small amount, especially if you're trying to patch a 225 MB DLL.

And I would argue that the 32 MiB is worse than arbitrary - it is pointless. I have been thinking about this for a while and I cannot think of a situation where it makes things better. It wastes (a lot of) CPU, and I claim that it doesn't actually save memory compared to letting the balance-set manager do the trimming.

Low CPU priority and low IO priority are important for being a polite background process. Not using too much memory is important. But a working-set cap fails to achieve that last goal.

There probably is a way to find what processes have this mode enabled, but it's probably more productive to search github for references to the problematic constant.

I reported the documentation bug. I don't use Feedback Hub because it never works for this type of issue. I know that some Microsoft developers will see this blog post and I hope that internal bugs get filed.

But ultimately as a non-paid tester of Windows I'm under no obligation to report non-security bugs in any particular way. I like reporting them through twitter and blog posts.

And local variables, including tracking where they are stored at various points in a function

Even modern CPUs can explore the 32-bit space extremely quickly. If you get a 128-core CPU running at 4 GHz then can do 2^32 ops in 1/128 seconds. Pretty amazing. The improvement in the nine years since I wrote the article is fun to see.

That is because:

> We repeatedly make a political decision in the US not to prioritize them

Public transit can be fast and efficient if it is prioritized. That may require raising the federal tax on gasoline (last raised thirty years ago!!!) and directing that overdue increase to funding alternative transportation options.

I bicycle commuted for years in an American city. It is as safe as your local council decides it should be, and my commute always felt safe. Don't give in to the belief that unsafe cycling is inevitable.

Just remember that this is a societal choice. Biking can be safe and efficient. In Vancouver BC they have prioritized modes of transportation other than driving, including creating a dense network of cycling paths and slow-streets.

There is a big difference between what is, and what must be. Improvements are possible, which is what the linked article is all about. These are baby steps, but if you want to bike more (and get healthier and live longer and live happier and pollute less) then demand more from your local politicians.

The built environment can make a huge difference to how dangerous drivers are. Slip lanes that allow right-turns without slowing down could be eliminated, thus reducing or slowing many vehicle/pedestrian crashes. Right-turn-on-red could be eliminated. More speed bumps or other traffic calming measures, separated bike lanes, bike-priority streets - all these things can help to make pedestrians and cyclists less likely to be killed by drivers making bad decisions.

Education and testing alone won't stop drivers from making mistakes. I would love to see more enforcement however.

This sounds like exactly my symptoms. Check for crashes.

Switching is a huge hassle, with uncertainty about whether things will actually be better or not, or if I will just hit different problems. Windows is actually excellent in many ways (see the tools I used to analyze this) so I would definitely be missing some things.

So, momentum plus uncertainty about whether other environments are actually better or just different. Plus my job.

I don't think that is true. If they had that many developers burning down performance issues then bugs like this would (IMHO) be getting fixed earlier. Anecdotally it sounds like a lot of customers are hitting this bug and I'd love to know whether it is known, being investigated, or just noticed now.

I'm sure it has been showing up in their start-menu telemetry.

I also wonder why they haven't fixed this issue: https://twitter.com/BruceDawson0xB/status/160716343915961139... And why this was ever allowed to ship: https://randomascii.wordpress.com/2022/09/29/why-modern-soft... And why this was ever allowed to ship: https://randomascii.wordpress.com/2022/07/11/slower-memory-z...

They only send crash reports if you opt-in to the customer experience improvement program (or whatever it's called), I believe. In general I think that sending crash dumps is _great_. Sending crash dumps is the number one reason why software is more reliable now than it was twenty or thirty years ago.

But, synchronously sending a crash dump together with not telling the user that you are synchronously sending a crash dump - that is problematic.

But, I can see how they got to this state.

But, if they are going to upload crash dumps they need to promptly address the crashes and I'd love to know why that did not happen.

An underappreciated consequence of this is that if doubling the number of cores doubles performance (meaning the multi-threading is good) but going on to the hyperthreads only gives a 15% speedup then that means that in the hyperthreaded case each hyperthread must be running 57.5% of normal speed - barely half as fast. That's just math.

This slowdown of each thread may not matter, or it may lead to increased latency.

TL;DR - when both hyperthreads in a core are in use it is highly likely that each one is running significantly slower than if only one was in use.

McAfee/Trellix says that they have fixed the bug in the latest (7.4.0) version of their disk encryption product. So apparently my guesses about the root cause were correct.

Interesting conclusion given that I found two functions and a (presumed) third-party driver/what-not that were violating the ABI. One of these was causing crashes, and the other one was going to. The crashes went on for over a year and a half, so, ...

My experience is that most non-trivial assembly language is doing processing of large chunks of data (high-precision math, encrypting blocks of data, FFTs, etc.) and therefore the startup cost of the function is not significant, so a tiny bit of inefficiency in that area is not something people care about.

Or, put another way, writing tiny little assembly language functions is probably not worth it because the mere fact that you are using assembly language instead of (say) C/C++ means that you have missed many opportunities (code reordering, inlining, etc.) so assembly language functions _should_ be doing enough work to justify their calling cost.

But, I'm not working on an assembler or even using one so I don't think I'll even file a feature request.

Those tactics could work. They would be a bit expensive however, and it would be a shame to have all users paying the performance penalty because of a few bad pieces of software. And, this would not have caught the two errors in the assembly language code within Chrome - that would require testing at every function call, not just Windows API calls. It would be more practical (I think) to do this checking on a special build of Chrome that is shipped to a small percentage of users, so that not everybody pays the price. But, this is an ecosystem problem and I'm not sure Chrome wants to shoulder the entire burden of finding bad software :-)

I think most assemblers know what functions are. There are usually directives to indicate this. They help with emitting symbols (you need the function name to be emitted or no-one can call it), stack unwind information, and other information (some of which may also be required by the ABI). Here's some relevant documentation for MASM https://learn.microsoft.com/en-us/cpp/assembler/masm/proc?vi...

The short version of it is that assemblers these days are rarely - if ever - just a zero-context stream of machine instructions. There is far more, some of it actually required.

It appears that the question of what is valid on a moved-from object is tricky. Here is one discussion: https://stackoverflow.com/questions/7027523/what-can-i-do-wi... FWIW, here is the move operator for the type of the object in question: https://source.chromium.org/chromium/chromium/src/+/main:bas...