Hacker News Reader

Comment by rightbyte | Hacker News Reader

Maybe he should have split his 120Mb executable into more executables ... or using a sane number of processes.

It would be bad to have the OS cover this use case by default.

You don't think an OS/kernel should be expected to perform the most basic of kernel tasks well?

Also creating a lot of processes to run your tests is a perfectly normal use case. In this case they're running 10 per process, but creating 1 process/test (so as to not leave any corrupt memory/state behind after a failed one) would also be perfectly reasonable.

Am I misunderstanding this; there are 1000 instances of a process stemming from a 120Mb executable, and initializing it 1000 times is kind off slow unless you chose to turn off some branching integrity check.

I don't think other OSs have this much trouble, but I haven't performed my own testing, so take this statement with a grain of salt.

Dunno. But reading the test readme it seems like gLinux (GNU/Linux? RS' new angle?) has other problems that need dehacking.

I mean running test like this is fine who cares. But it's probably bad architecture for the unit test that makes it take time rather than the OSes. And that's fine because it's not production code. But not ground for a condescending snarky blog post.

https://www.chromium.org/developers/testing/running-tests

'Run tests 5x faster on gLinux Browser tests on gLinux start up extremely slowly due to idiosyncratic NSS configurations. If you need to regularly run browser tests on gLinux, consider using the run_with_dummy_home.py helper script:

  testing/run_with_dummy_home.py testing/xvfb.py out/Default/browser_tests

This can speed tests up by 5x or more.'

That is something specific to gLinux, not Linux.

Also it's not even related to the kernel.

They're likely using some NSS configuration that does network lookups (maybe they're using NIS, LDAP or something else).

gLinux is Google's internal Linux distribution, based on Debian.

You do understand that windows creates about 1000 processes on boot alone? A clean start?

(Of those about 100 survive as services.)

In short this fail adds as much as 10 seconds to boot time? (Sort of hidden on most hardware by parallelism, but if you have something not super recent, well...)

This OS performance bug does not add 10 seconds to boot time. That would only be the case if those startup processes were enormous .exe files, which they generally are not. The O(n^2) chart in the blog post shows that the cost of this issue is modest for most executables.

Yes, but he creates 1000 processes of the same executable with different arguments to run different tests as I understand it. With a really big executable.

That's not the same thing as it's slow to add normal sized processes.

That's not out of the ordinary though?

Any time you compile C or C++ code, you're creating a process from the same GCC/Clang/whatever executable but with different arguments for each source file. For a big project, it's not uncommon for there to be thousands or tens of thousands of small source files. Creating and maintaining processes is the core responsibility of an operating system.

gcc is like 1Mb. It would be interesting if he did the same test on Debian or something with default options.

I would personally link in the tests as a shared lib or something. chrome.exe on my laptop is around 1Mb. That would probably speed up his tests more than his hacks to the validation routine.

I have been told that moving the meat of unit_tests.exe to a DLL would avoid the repeated CFG initialization costs (they would be paid on the first CreateProcess call and then the CFG data would be shared). This wouldn't speed up the tests "more" than my change did (it would speed up the tests slightly less, actually), but it would be a possible fix.

But, such a change would be more work. And, honestly, it shouldn't be necessary. Windows could avoid this problem by using an O(n) algorithm for the initialization, or by sharing the CFG data between .exe files as well as .dll files. I am content with my current workaround and I'll consider reverting it when the underlying OS issue is fixed.

Note that the chrome executable on Linux contains all of the code so it is 50-100 MB. It is only an accident of history that chrome.exe on Windows puts all the code in chrome.dll/chrome_child.dll

I have a sneaky suspicion you're right, windows generally assumes any 'shared' code will be in DLLs and is optimized around ensuring they are shared as much as possible in memory. It's likely that if the tests were in a DLL vs. the EXE then windows would just share the CFG page for that DLL read only between all the processes and call it done.

The size of the executable doesn't really matter after the first load as subsequent loads share the mapped section so it doesn't need to be loaded from disk again. Loading an executable multiple times after the first time really shouldn't take all that long from an OS perspective...just some mapping of shared pages.

> The size of the executable doesn't really matter > after the first load

shouldn't really matter. If you have CFG enabled then it does. If they fix CFG's initialization then it will go back to not mattering, and even this huge process will be created in just a few ms.

Reply on news.ycombinator.com