Hacker News Reader

Comment by jeffbee | Hacker News Reader

That's one way to look at it, but a very biased take. An equally valid take is that Firefox was calling an expensive platform feature too often, and even though it has been killing performance for years (possibly, for the entire history of the project) nobody noticed or bothered to fix it on the application side.

View on HN

The platform feature in question was normally cheap and just made artificially expensive by Defender intercepting calls to it and blocking until analysis was performed. I don't think it's the FireFox' team's responsibility to be aware of and take into account arbitrary software intercepting system calls.

Windows Defender real-time protection is enabled by default.

> I don't think it's the FireFox' team's responsibility to be aware of and take into account arbitrary software intercepting system calls.

Per the bug report, Firefox was generating up to ~14,000 calls where Chrome was generating ~300, though.

Surely it is Firefox' team's responsibility to use system calls in a sane way, say not almost 50x more than the competition?

Did you read the bug report? This is literally about writing to files in a temp folder. Surely you can optimize that but you should also be able to assume that this does not use excessive amounts of CPU on a modern operating system.

Yes, I have read the bug report. It mentions that Firefox writes wayyyyy too much in the temp folder. It also mentions that the team should fix this behaviour independently of the fact that some of those calls are more costly than they should be because of the bug in Defender:

> With a standard Firefox configuration, the amount of calls to VirtualProtect is currently very high, and that is what explains the high CPU usage with Firefox. The information that the most impactful event originates from calls to VirtualProtect was forwarded to us by Microsoft, and I confirm it. In Firefox, disabling JIT makes MsMpEng.exe behave much more reasonably, as JIT engines are the source of the vast majority of calls to VirtualProtect.

> On Firefox's side, independently from the issue mentioned above, we should not consider that calls to VirtualProtect are cheap. We should look for opportunities to group multiple calls to VirtualProtect together, if possible. Even after the performance issue will be mitigated, each call to VirtualProtect will still trigger some amount of computation in MsMpEng.exe (or third-party AV software); the computation will just be more reasonably expensive.

> It mentions that Firefox writes wayyyyy too much in the temp folder.

> > the amount of calls to VirtualProtect is currently very high

Calling VirtualProtect is not writing to the temp folder. The VirtualProtect call is to change the permissions of the in-memory pages. It should be an inexpensive system call (other than the cost of TLB flushes and/or shootdowns).

You really shouldn’t assume anything in software or any complex system. I know this wouldn’t fly at my job, and I don’t work at Mozilla.

This is basic testing.

Normally this is the mark of a bad software engineer, but attempting to blame the platform you’re on for your lack of testing takes it a to a new low.

Mistakes happen, admitting full incompetence that basic testing isn’t done is damning. This is not a good defense of Firefox nor Mozilla.

Not sure what your job is, but in my job:

- we implement a feature, test it thoroughly for functional and non-functional requirements

- when we are happy, we release it

I don't see myself being responsible for a third party software company coming along years later and introducing a bug in code that injects itself between my software and the operating system that users of the software I wrote happens to install at some point.

Maybe you're not responsible, but if someone says "something changed in the OS and your previous method is now adding substantial overhead", you could either a) report the change to the OS and mitigate or b) report the change to the OS and ignore the problem for years. It sounds like Mozilla chose b, for whatever reason.

As a software developer, I've had to workaround many many bugs in OSs, especially when dealing with updates to Android. It's just part of the job.

The OS isn't some random third party software, it's one of your dependencies. Your software doesn't work without the OS and if it also doesn't work with the OS, it just plain doesn't work.

That's really not a tenable mindset to be taking these days. With how much Windows has become a constantly-moving target rather than a stable platform, you need to regard it first and foremost as your adversary, whether you are developing against it or are simply an end user. And the days of being able to thoroughly test against every relevant version of the OS are long gone; Microsoft has ensured your QA will be Sisyphean.

At the end of the day, it's about your users.

If your users are on Windows, you have to be where they are. Moving target, wonky API, warts, and all.

Yes, it's Sisyphean. That's why my shop had a whole room stuffed with parallel Windows installs. We couldn't afford to have our users be the first ones to notice Microsoft pulled the rug out from under us again.

You basically just said you stop supporting things once they ship. Doesn’t work properly on Windows? Shrug.

Which is my original point about quality software engineering.. Apparently many don’t test, and if it’s broken, don’t care!

I usually assume that even vaguely considering looking in the same direction as a file on windows will melt my CPU.

Windows Search Indexer automates that for me. CPU keeps burning even when monitor is off and I'm working on another computer.

Why is Search Indexer constantly rescanning the same files? Can they not cache the results from the previous scan? That and OneDrive are constantly making my work laptop scream.

Come on, anyone that has even unzipped Linux-centric stuff on Windows knows how slow individual file operations are compared to Mac or Linux.

It's very common knowledge that on Windows you will get terrible performance if you have many many small files.

I don't know why Microsoft doesn't fix that. Maybe they can't for compatibility reasons or something. But that's the way it is, and any software that wants to run well on Windows needs to deal with it by using fewer bigger files.

> Surely it is Firefox' team's responsibility to use system calls in a sane way, say not almost 50x more than the competition?

The docs for that function don't say anything about performance: https://learn.microsoft.com/en-us/windows/win32/api/memoryap...

They also don't say anything about "sane" usage, and while I don't have an MBA, I'm pretty sure they don't teach anything about `VirtualProtect` ratios when doing competitor analysis.

One possibility is that the Chrome team's implementation was more efficient due to luck, or they invested the resources to identify the performance characteristics of this function call, whereas the Firefox team missed it. I don't think "Chrome has more development resources than Firefox" is news to anybody.

There are three facets to any protocol, API, or standard in software:

The spec, the intent of the spec, and the implementation of the spec.

Doesn't matter what the docs say; what matters is what performance testing shows. Docs lie.

And even if Chrome lucked into a cheaper implementation: that luck has given them a market edge.

It's the application owner's responsibility to make it the app run as best as it can on a given platform. Platforms are messy, but you have to deal with it. You should escalate to the platform owner, sure, but you can't rely on them fixing it in any reasonable time-frame.

I worked on a desktop<->cloud file sync app. On Windows, only one badge can show up on a file's icon in Explorer. If there's multiple apps trying to set the badge, who wins? Well, it depends on the lexicographical order of the registrants names. So what did we do? We added some spaces to our registration name to make them show up first. Good for the user, as best as we can know - since the user or their admin had to install the app to get these badges in the first place. And they were useful ones too - whether a file was synced or not. We tried our best, and escalated.

> I don't think it's the FireFox' team's responsibility to be aware of and take into account arbitrary software intercepting system calls.

One of the first, hard lessons I had to learn about web development (like, stare-at-a-wall-and-consider-my-career-hard) is that web development is way more about network effects than application architecture.

Real people run systems with real configurations, and when you're targeting "the public" as your userbase you must account for that. And Mozilla knows this: if you go into the source code (circa 2009, YMMV) and look through the initialization and boot-up logic, you would find places where the system used heuristics to figure out whether some extensions had been installed in odd places instead of the "Extensions" directory (because the tool had been installed before Firefox) and hot-patch paths to pull in that component. Because if a user installs Flash and then installs Firefox and Flash doesn't work in Firefox, it's not Flash that's broken... It's Firefox.

It doesn't matter if the bug is in "Microsoft's code" or "Mozilla's code." That's unimportant. If you're a Mozilla engineer, all that matters is whether this bug would cause a user to get pissed off and uninstall Firefox.

Thats. All. That. Matters.

Yes, but it’s incredibly difficult to work out what is causing the problem. That’s what happened here.

I completely agree with you and have been on the other side of this too, having worked on a native enterprise app running on various MacOS, Windows, iOS and Android versions. Customers don't care if you have a great explanation why stuff with your app doesn't work. That being said, it's completely unreasonable to have the proactive expectation of something working well today (writing many files) breaking tomorrow (due to defender heuristics changing) and proactively trying to prevent this by optimizing. Mozilla reacting to this by both reporting the bug to Microsoft and optimizing to work around the problem is really the best you can do.

"They shouldn't have written so many files in the first place" is not a valid preventative strategy, but a one way road to premature optimization hell.

> arbitrary software

Windows Defender isn't "arbitrary software" - it's built into the OS and enabled by default. To anyone building an application for Windows, it should be considered part of the platform.

I'm not sure how you can possibly qualify VirtualProtect as "an expensive platform feature". Looking at the operation that VirtualProtect actually has to perform, from first principals, it should be one of the cheapest syscalls in the entire kernel.

The bug was that ETW (in the antivirus process) was doing something braindead; zeroing a megabyte of memory unnecessarily every time someone called it just to get the size of a buffer.

> it should be one of the cheapest syscalls in the entire kernel.

That's an educated guess... that is unfortunately very easy to disprove :(

Exactly. If you're going to assume some call is free, write that down in a test that can be periodically verified and, preferably, is.

I'd love if you'd elaborate on this. I know very little about what VirtualProtect actually does under the hood but, in theory, it should just have to flip a couple bits in the address space mapping which says what the protection level is.

It also needs to flush TLB entries. Changing permissions on page table entries is unfortunately a bit more complicated than just twiddling some bits.

> I know very little about [...] but, in theory,

You are assuming things you are unsure about :) Even if your assumption was correct things could change from one Windows update to another.

When I worked on a time sensitive java project, our test suite had benchmarks for JDK functions as simple as Arrays.copy() to make sure we are the first to notice if something changed under the hood.

Branch prediction should be a super-dumb algorithm, but then Spectre comes along and, oh dear.

Malware protection algorithms make fools of us all.

There's nothing about the bug that had anything to do with malware protection, or branch prediction, so I'm not sure how that statement applies to the conversation.

The bug was in ETW, which just happened to surface in a windows utility that ostensibly protects you from malware.

I would not consider a fast AND accurate branch predictor a trivial matter at all

Also worth noting that the "expensive platform feature" you refer to in this specific case means "writing to a file". Something as basic as this should be assumed to be fast on modern operating systems.

It is not a bug that there are overlooked optimizations in some platform features. Windows has a ton of slow features. Starting a process, for example, takes forever. It is the responsibility of application authors to write their performance-sensitive critical path in such a way as to avoid bogus platform behaviors. This goes for Linux, which has more than its fair share of brain damage, as well as Windows.

I generally agree with you. Having worked on lots of cross platform software, a big part of that job is to work around quirks of the underlying platforms, which can be significant. However in this case, it's not that Firefox was introducing the usage of these APIs and was then starting to have performance problems. They used the APIs without problems when suddenly Defender came along and slowed them down by orders of magnitude when they had been working fine for years.

Yeah, your program definitely should not do as many useless writes on the system it runs on, it's just bad behaviour. If every program did the same the disk would grind to a halt, SSD or not.

No it had nothing to do with Firefox writing files. Firefox was making a bunch of calls to VirtualProtect. Windows Defender (MsMpEng.exe) was then writing to file (an sqlite database) every time one of these calls was made, which was slowing down the system.

This comment is a good summary of what the issue was once they understood the problem: https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c82

Where did you get that idea? Sqlite? Windows Defender isn't using sqlite at all.

It detects the use of SQLite, then copies it, etc etc. Read the bug for more details.

I can assure you that I perfectly understand the bug and corresponding patch/fix. The patch fixes Event Tracing for Windows (ETW) and how it allocates temp buffers.

The speculation about SQLite at the top of that Mozilla bug report is mostly irrelevant.

Recent discussion of this here also cited a problem (not sure if it was the same problem) with Defender causing 100x performance drop with some PowerShell operations.

Reply on news.ycombinator.com