24-core CPU and I can’t type an email – part two
randomascii.wordpress.com
randomascii.wordpress.com
For those of us here with > 15 years in the business, when's the last time you really felt a BIG uptick in performance or responsiveness from a new computer upgrade?
I buy a new laptop every 3-4 years, and I buy a nice one, but generally speaking I feel like we've been kind of flat in terms of usable power for a while. 6 or 8 years ago I switched to an SSD, and THAT was a big deal -- easily the most dramatic uptick in performance since I moved from an AT clone to a 386 in 1991. But since then? Not so much.
My laptop is smaller. It runs longer on the battery, and is generally cooler. The screen is better. But in terms of how long it takes to boot, or open large data sets, or whatever? Not so much different than 5 years ago.
Buuuut, my next laptop (if I stick with Apple) will probably be a MB 12" with medium to low specs, with most of my demanding dev stuff moved to a Linux server.
Admittedly, the most recent cycle felt somewhat the same, but generally I see a boost in quality of life. Not things like how fast a script runs on a large dataset, but like how responsive my machine is _while_ that script is running, or _while_ I'm doing some intensive task.
Also, I/O is often the bottleneck, which is why most people perceive a speed increase moving from spinning platters to SSDs. If you move stuff to RAM, I bet you'd see another perceptible speed increase. (I've recently started using the RAM-backed /dev/shm on Linux for transient, throwaway data generated by my programs, and boy it is fast.)
I have a Linux machine running on a Core 2 Duo machine (2005) at home and an Core i7 (2017) at work. An "ls -la" feels about the same on both. But when I try to train an ML model, the difference is stark and perceptible.
- gigabit ethernet LAN and a speedy Internet connection
- doing "desktop" work -- browser, SSH, office docs, media consumption
All my machines feel snappy, ranging from the $250 NUC through the MBP2011 (with 16GB RAM and an SSD) through the midrange Intel desktop (also 16GB RAM and an SSD).
I don't play any significant games and major computation happens on servers, not on anything I'm typing directly on.
It's been snappy for a decade, and I don't expect a noticeable improvement the next time I get something new. I might be pushing more pixels to a screen, but it will respond at about the same speed.
On the one hand, for software that's not complete garbage in terms of optimization (scientific software, video/audio software, games, my own code) the difference is staggering, I hadn't felt a change this drastic since I moved from Pentium 1 to Pentium 3. On the other hand, I now know that there is no amount of resources that will make a modern web browser or Electron app run well.
The good news is that now that I feel I deserve good performance I've made an effort to get rid of most of the crap. Apart from disabling JS etc. by default and blocking everything I possibly can I have no solution for the browsers, sadly, but everything else is gone. The Thinkpads all run minimalistic Arch Linux setups and mostly work as thin clients, so overall every system I use is snappy and responsive, for the first time in a decade. I shouldn't have needed a hardware upgrade to return to common sense in computing, but it is good to know that with some discipline and a low tolerance for garbage it is still (mostly) possible to have a reasonable computing experience. Now, if only there was a usable browser out there...
There's definitely a difference between the L5639 at 3Ghz and a i7-6700HQ (2.6Ghz base) in my laptop
Aside from that, 2008 was probably the biggest jump. Going from a mediocre laptop to a custom built ~$2500 desktop with a Core 2 Quad and (especially) 24" monitor was astounding. I'm never going to make a laptop my main machine ever again.
I notice a huge improvement in boot times with every (windows) OS upgrade I've gotten.
Everything else, moving from CPU generations to newer generations, _felt_ like it barely made a dent.
GPU rendering via CUDA also meant a massive jump.
The next time I compiled a kernel took something like 8 minutes. I couldn't believe my eyes as the steps flew by on the console.
Other than "adequate RAM" I'll go for "SSD" like everyone else. Now adequate RAM doesn't even matter all that much because swapping is so fast (relatively speaking).
There is also the idle nature of most computer usage. That next gen i7 will not load Facebook appreciably faster. CPU limited tasks could be only 5% of normal use. So a 20% improvement for 5% of the time.
I agree that laptops/batteries have gotten smaller at the same price/performance point, though. Very good for my company.
For me, every upgrade was at least a factor of two improvement to pretty much everything. It was a qualitative difference -- things that just didn't make sense to run were now reasonable to do, things that meant leaving it overnight could now be kicked off before going to lunch or whatever, things that required scheduling were now causes for slashdot breaks (age, remember?), and those minutes-long jobs were now something you could run and wait for without context switching.
I could go on for one more step -- except that's kind of where it ended, at least for reliable improvements. Stuff that was almost buttery smooth might become buttery smooth, but stuff that was buttery smooth already might start lagging some here and there.
These days upgrades are more about SSDs, RAM, decaying batteries that tip you over the edge, connectors, or maybe the collection of weird compatibility problems (eg 3 monitor support) that you've sort of worked around but the workarounds are breaking down, and you have this naive notion that the latest greatest laptop will magically be more robust. (Instead, you usually just trade over to a different set of problems.)
I'd agree that the SSD switch was the biggest boost in recent memory, and it's been about a decade since I've experienced one of those glorious upgrades of yore. And I miss them -- the heady excitement of everything being better is now replaced with a nervous inventory of everything I need to work or think might be improved, trying it all out in hopes of feeling some improvement or at least having it continue to work.
DOOM on a 386 was nearly unplayable. On a 486 it was perfect. Quake on a 486 was awful, was great on a Pentium, and was smooth as butter on a Pentium II.
These days, each generation is less than 20% faster in single-core performance. My desktop at home is an i7-3770k. In a couple months, it'll be 6 generations out of date, and yet it's still fast enough for everything I do, even gaming!
My CPU is 6 years old, yet doesn't feel like it. If you were running a 6 year old computer in the year 2000, you'd be running a 133 Mhz Pentium while everyone with a new system at the time would be on a 1 Ghz Pentium III. The performance difference was at least and order of magnitude and definitely noticeable.
1) A lot of the "a ha" magic really has been figured out in microprocessor arch (e.g. pipelining, superscalar, multi-level caching).
Previously, we were reaping the benefits of process shrinks AND microarch epiphanies. Now, only a slower former and less powerful versions of the latter (e.g. branch predictor tweaks).
2) We're discounting the "performance" that has instead been allocated to power. Up until the P4, we had the benefit of just saying "more power!" in pursuit of performance.
Now, not only do we have a power budget to stay within, but we're actively trying to decrease power draw for the same workload in mobile parts.
So instead of getting something twice as fast, we get something that has battery twice is long (or however the math works out).
We've essentially changed the metric of performance from "calculations per second" to "calculations per watt", which can be useful for mobile (battery life) and data centers (reducing heat and the need for massive air conditioners, plus the electric bill), but less useful for gamers that often need more single-core performance.
I'm still using that machine, i try out new stuff every few months. I still don't see the point. I load it down pretty heavily all the time and it still kicks right through it just like it did when i took it out of the box.
If it breaks, i'll just buy another similar one. They're not much over $500 now and it's an amazing amount of computer for that price even in 2018(19?). My only regret is not maxing out the ram and storage really, but that's on me.
I've never felt this way until now, i used to wish i could justify a new machine every year or two and often would. I think i didn't keep a laptop for over a year and a half until i got this one... But now i just don't care. I don't own a single machine with a newer cpu than this one, and my desktop still does great too with a 3770 and a couple SSDs.
Earlier this year. I upgraded (my personal computer) from a 2010 dual-core i5 (Arrandale) laptop to a 12-core Threadripper. My old laptop was still usable for single-thread tasks - I upped the memory to it's limit (8GB) and upgraded to SSD, and this contributed to my long upgrade cycle.
My workflow has changed since I bought the laptop in 2010 - I now tend to run multiple VMs and docker in parallel, and there I noticed a BIG uptick in performance. I could only run 2 VMs (slowly, with swapping) at most before the upgrade, post-upgrade, I'm now able to run 6 VMs with no signs of slow-down - I haven't tried to find the upper limit to number of VMs, but there's plenty of room for growth for additional RAM and SATA disks.
One more I would submit is the transition from DDR2 to DDR3. I do not yet have a DDR4 machine, but I am anticipating some kind of "aha! so this is what I have been missing out on" moment when the time comes.
Here's a fun project idea: Plot the boot times of machines running systemd (and some control) over time, and I'm willing to bet you'll see a substantial increase in boot time immediately subsequent to the systemd decision by the Debian TC.
I think the next major step will be for the humanity to learn make software more efficient and effective rather than throw more CPU cycles at the problem.
This article is from 2014 but hardly anything has changed.
https://www.comsol.com/blogs/havent-cpu-clock-speeds-increas...
https://superuser.com/questions/543702/why-are-newer-generat...
I disagree with this. I'm not going after you specifically, but I think this attitude is part of why the IT industry seems to reinvent the wheel every few years; there's this perception that we're GOING WHERE NOBODY HAS GONE BEFORE. No, most of us are not. Maybe a very few people in research labs are, or people really pushing at the raw edge of cryptography or mathematics, but the rest of us are basically cycling through the same ideas over and over, in different clothing.
I can't tell you how many times I've run into a problem and done some research, and found out that the optimal practical solution or algo was devised by some dude working at IBM in the 60s. (In fairness, some of those guys were really ahead of their time.) A person could make a very good living just strip-mining old ACM research papers from the 80s and selling the ideas in proof-of-concept form to the government, military, investors, or anyone else with no sense of history.
Sometimes I wonder how much further we'd get if we did a better job building on prior efforts and resisting the urge to clean-slate things quite so often.
The number one cost at most companies, is humans. Before electron what did you have for cross platform development? Java was there, but that never really took off for embedded UI inside the browser, and the browser became the primary target for most businesses in the last ten years.
Electron allows small teams to focus their most costly investment on developing toward one product being developed across many platforms.
I use electron apps that give a consistent experience, on macOS and Linux (not personally Windows, obviously as well), and then the same web app on Firefox, Chrome and Safari across those same platforms with the addition of my phone.
Yes, that comes at the cost of running a full browser, but it’s a logical choice when optimizing for the diverse computing landscape of today.
We’ll see.
At the cost of battery life and memory usage for the users....
The memory usage hasn’t been bad enough for me to notice it stealing resources from other apps that need it. I don’t think most users notice.
I’m not talking about you. It’s clear this is a big deal to you, and so I assume you don’t use any electron apps. I think that’s a gamble most companies would make, losing a small number of users, to have the potential to gain many more.
Slack - Shouldn’t.
Yes, it's easy to make crappy memory hogs with Electron, but it's also possible to make fully-featured, snappy, great applications. Such as VSCode.
My basic point there being that losing a small number of users due to these concerns is probably worth it.
I don't want to come across as someone who's hyper-defensive of Electron. I hate the fact that such a bloated piece of software has become the standard means for supporting cross-platform development. At the same time, I really don't see other viable options at this point in time.
I'm curious though, is there something you would recommend instead that meets these requirements with a single codebase (~90% shared code): target all major platforms (macOS, Windows, Linux) and target all major browsers (Chrome, Safari, Firefox, Edge, IE), (edit: and how could I forget, all major phone browsers) with nearly identical UI/UX?
What might be reasonable, at least for applications that aren't performance sensitive, is asking for a cross platform solution between desktop OSes and a different one across mobile platforms. Those exist, as you well know.
It's one thing to not want to use Electron app, it's another to say no one should use them. It rubs me the wrong way. There are a lot of things I'd never use, but they don't rile me to that level.
> The only reason people pretend otherwise is laziness and cost cutting - a context in which Electron seems reasonable as well, to the horror of technically literate users everywhere.
"Laziness and cost cutting" is just your label for trade-offs you don't agree with. That would apply to cross-platform Java/Swing apps or Gnome. The "technically literate" users don't have to worry about business considerations and engineering tradeoffs - the authors do. If memory usage is such a big deal, then the market will self-correct, I was told it's a meritocracy.
The reason this trend riles me so much is that we now have companies like Slack, which easily have enough resources to do an efficient desktop app on any platform they choose, releasing utter garbage that, far from merely wasting memory, takes up ridiculous amounts of CPU time (and therefore battery time and energy) to do the simplest things (like render emoji, or even a blinking cursor). The aggregate waste of resources is mind boggling, and we've gone far past the point where Electron was solely used as a quick solution for very resource constrained companies or single devs.
> "Laziness and cost cutting" is just your label for trade-offs you don't agree with. That would apply to cross-platform Java/Swing apps or Gnome. The "technically literate" users don't have to worry about business considerations and engineering tradeoffs - the authors do. If memory usage is such a big deal, then the market will self-correct, I was told it's a meritocracy.
It certainly does apply to Swing apps, Qt, Gnome, etc. I've always considered all three of those frameworks ludicrously bloated, by the way, but Electron has far exceeded my worst nightmares in that regard.
I'm not sure who told you the market is a meritocracy or why you believe it, but I don't see much evidence for that view, personally.
Again, this is the developer's perspective. As a user, I don't care at all how many others use my email client or text editor. What's more important to me is that it doesn't suck, especially on my personal computer which isn't quite as tricked out as the computer I use at work.
My point is as previously stated: there are multiple perspectives on this. For me as a user there is absolutely no benefit to the developer using Electron. I don't care if it took a million man years to put together; I care about my own resources. I'm not arguing with your perspective, just arguing that it's just that, one perspective, and that users aren't "looking at the wrong thing" when they complain about the result being crap.
> At the same time, I really don't see other viable options at this point in time.
Electron is five years old. Surely, developing a cross platform application was viable before that? Seems like an ironic statement considering that the larger part of Electron is Chromium, a cross platform GUI application predating Electron. There are plenty of libraries and frameworks that abstract OS specific stuff with regards to GUI, networking, file system handling etc.
> I'm curious though, is there something you would recommend instead that meets these requirements with a single codebase (~90% shared code): target all major platforms (macOS, Windows, Linux) and target all major browsers (Chrome, Safari, Firefox, Edge, IE), (edit: and how could I forget, all major phone browsers) with nearly identical UI/UX?
For one, your web app could be just that: a web app. There's no reason to have multiple copies of chromium running and littering your disk when you already have a browser.
Second, these UIs that look consistent across all platforms probably make the designers happy, but for the user (or at least me, personally) it is better if the interface is consistent with the design conventions of the platform it's running on.
But no, I don't really have an alternative to suggest if those are your goalposts. Targeting multiple platforms is just one of those things you have to suffer through with some consideration if you don't want your application to be a bloated webapp-bundled-with-a-browser.
That is ridiculous comment to start with. You might as well say that something's name has to start with an 'E' ends with 'n' and is 8 letter word and developed by Chrome browser team.
If it were not a software, I think it would right in category of 'blame the victim'. Seems you really believe it is users problem that they do not upgrade their phones and computers every couple of years.
Is it, really? Let's drop the web as one of the platforms we don't want to target. There are some reasonable options left, but when you add the web in, and just limit that to the major browsers, your requirements for support just got extremely more complex. Why do you want to target the web? because in general, that's the easiest place for user acquisition and on-boarding of a product.
And where did you get this: "Seems you really believe it is users problem that they do not upgrade their phones and computers every couple of years." This is a decision people need to make in how they target certain users. If your target user base has limited compute available, limited bandwidth, limited memory, etc., then of course you need to build software that meets your business requirements.
There are no victims here, there are users, potential users, and former pissed off users who've gone elsewhere. These are all business decisions you need to make as a developer. If you target Electron and by doing so piss off all your users and they ditch your software, then you screwed up. But I only see VSCode gaining in usage, even Slack being a horrible memory hog, they're still gaining customers.
In a nutshell, there are companies that are being very successful with this, success is hard to argue with, as much as you might dislike the way it's being achieved.
Before Electron there was xulrunner, whose main purpose in life was to be the cross platform application layer of the Firefox web browser.
Even though it was officially an unreleased "technology experiment", xulrunner was successfully used both internally at Mozilla to develop Thunderbird and Sunbird, and externally to develop other cross-platform desktop apps like TomTom Home, Uploadr, Nightingale, Songbird, Miro, Joost, Lotus Notes, etc.
But xulrunner was never as fully fleshed out and widely used as Electron, and Mozilla was never serious enough about supporting xulrunner as a platform for other applications, for it to be viable in the long term.
It never caught on and became a standard, and now it's obsolete.
https://en.wikipedia.org/wiki/XULRunner
>XULRunner is a "technology experiment", not a shipped product, meaning there are no "official" XULRunner releases, only stable builds based on the same code as a corresponding Firefox release.
>Mozilla stopped supporting the development of XULrunner in July 2015.
In order to solve the problem of every application shipping with its own web browser, there needs to be ONE standard Electron shell that can run all those simple apps and even the advanced ones, and apps need to be able safely include their own native code extensions.
Many people developing Electron apps explicitly and legitimately WANT their app to include the whole web browser, just so they don't have to expend any effort supporting other browsers.
Electron needs to mature and become stable enough that there can be one global install that runs all apps.
But that requires long term support, and a big enough well funded team working on it full time.
Just as all the successful corporations who built on top of OpenSSH have an moral obligation to support its developers, I think successful companies making money by shipping big fat Electron apps today have a moral obligation to support the development of a standard Electron shell.
Think of it as buying carbon credits.
> I think successful companies making money by shipping big fat Electron apps today have a moral obligation to support the development of a standard Electron shell.
I agree completely. The issue that I mainly see, with the possible exception of Linux, it's a drop in the bucket of number of users (and has it's own issues GDK vs. QT/KDE), the main adversaries have traditionally been the OS builders. They actively seem to be intentionally creating a walled garden model.
Alternatively, we've got closed systems like Slack that are slowly and pervasively getting rid of open systems like IRC.
I'm really not sure that there's an engineering time advantage in taking a reasonably solid existing piece of software and rewriting it in Electron. Is the technical debt really that large?
Really, I think the problems we have are political, and not technical. I include things like NIH syndrome in that group.
We seem to be blaming developers, but developers are just dealing with an ecosystem where this is the best common denominator. There's another comment in this thread, that points out that Electron is only 5 years old, and there is a lot of opportunity for performance and system utilization improvement.
IMHO if we want this to be better, then the OSes should make this easier and more performant for developers, and by extension, then their platforms will out perform and outsell the others.
I shouldn’t have to buy 8GB more RAM just because Companies X, Y, and Z only want to hire a few JavaScript programmers.
A very simple and straightforward model of action. Yet it has fueled almost every major non-military-funded technological advancement since at least the Industrial Revolution, inclusive.
(And if you consider the military a very special customer who has been graciously granted the disposal of the entire nation's profits ... then they are no exception after all.)
Why does development have to be cross platform?
If you are a startup with a small team, and you truly only have two or three developers, then I can understand this desire. I totally understand the "this was my side project" or "we're just four people in a garage" arguments. I get it.
But most products people complain about are from billion dollar corporations that could easily handle having totally different codebases for 4+ platforms clients, (with way more total lines, but only a little bit more complexity and cost) than one single cross-platform super-client. Humans are not expensive to these companies, and we're only talking about adding a few more.
From my own personal experience, writing a native Java Android version and a native Windows UWP C# version of the same app, the effort + maintenance of those two codebases combined was only a little bit more effort than writing one HTML+JS+Electron-ish project that spits out an Android and Windows version. Native code just isn't that difficult if you spend a little bit of time learning their APIs (I would argue it's actually simpler than modern web frontend).
I understand why companies choose Electron instead -- and I've even had to do so myself at times, I get that. It's great for small projects or low-budget projects or internal-only uses, etc.
But it's not unreasonable for people to complain a little bit when these large hyper-popular billion-dollar services (like Slack or Spotify or Twitter) cheap out a bit on their software in that way.
There was a time when I couldn't have two Adobe products like Photoshop and Illustrator open at the same time. They would just continue to hoard resources until my machine locked up and I had to reboot.
Even today I have a fairly robust system (i7 processor, 16GB of RAM and a 3GB Video card) and running multiple programs still makes all my fans kick in and start whining at me.
Well, locking is hard.
> The good news is that even though there is occasional unfairness, there is unlikely to be persistent unfairness. In order for a thread to steal the lock, it needs to hit the tiny window where the lock is available. In practice, a thread is unlikely to be this lucky repeatedly.
https://blogs.msdn.microsoft.com/oldnewthing/20170705-00/?p=...
> The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it. Take heed. You got it wrong. Admit it. Locking is _hard_.
This despite the kernel having a nice RCU mechanism inside as well as waitfree queues.
> the rule simply is that you MUST NOT release and immediately re-acquire the same spinlock on the same core, because as far as other cores are concerned, that's basically the same as never releasing it in the first place.
https://yarchive.net/comp/linux/spinlocks.html
Other mechanisms do exist of course.
What alternatives do you suggest?
s/concurrency/contention/
Would you expect 24 employees to write ONE email without 4 team leads and one department head?
Obviously NO!!
Your processors obviously need more management. I think Intel has the right offering for you, aka management engine.
Scrum’s not necessarily a bad anology. Here’s a different thought that this conversation has me now thinking about, what if we did think about development teams as CPU cores? We might discover weak points in the architecture of an organization, and recognize more quickly where we need to address bottlenecks. The bandwidth of the bus betweeen the cores might be too limited. The pipeline of work (backlog) might not be deep enough, and has a ton of branch statements (spikes) that may throw out the entire pipeline...
I have a golden rule for my systems. After booting into X and opening one terminal with htop and hiding kernel threads, everything should fit very comfortably in one screen. This constraint forces yourself to have a very simple setup. A few daemons, a window manager and a terminal. I do the rest of my computing in Emacs and Firefox.
I have several Arch or NixOS setups that could work on a 128 MB RAM setup, excluding Firefox. Plus, if something breaks, I know how to fix it.
Jamf's code quality and security model are abysmal. It blows my mind that apple recommends the use of Jamf to large Mac shops. Having peeked under the hood at previous employers, I was extremely disappointed. The product is insecure by design - not what one wants for device management that's given root privileges on machines containing corporate crown jewels. I highly doubt their operational paradigm for Jamf cloud has changed in the last year, either.
I apologize for going completely off-topic, but these products make my blood boil. Especially because I now work for a shop that uses both Jamf and McAfee for Mac "security". On a positive note, I removed the corporate mandated malware with ease - no boot to single-user required.
Those days are past. Now you have to be crazy to run one. Even setting security aside, I think antivirus packages are the number one source of instability and weird performance problems.
void
do_work(the_work_t *w)
{
/* for simplicity here, rather than e.g. w->contenders */
static _Atomic int contenders = 0;
contenders++;
for ( ; work_remaining(w); ) {
take_mutex(w->m);
do_some_work(w);
drop_mutex(w->m);
if (contenders > 1)
reschedule_this_thread();
}
contenders--;
}
One could of course only check for other contenders every few times through the for loop if reschedule_this_thread() is expensive or slow, or if contenders is especially hot.
contenders is explicitly not a locking mechanism and should not influence the policy of any code running while the mutex is held. It should also be a per-mutex counter.
In this particular case the lock was a kernel lock in kernel code so the OS would have to fix this, by making the locks fair (or occasionally fair).
How about:
while work:
if (contenders > 1)
{ reduce_priority; pri_reduced = true }
take_lock
if (pri_reduced)
{ unreduce_priority; pri_reduced = false }
do_work_quantum
drop_lock
endwhile
Of course, if reducing priority is too fast, this likely doesn't help; alternatively it could be too slow and what you get back in system latency is taken away in lowered throughput. That's probably not OK if you don't need the system latency to be low.
I wonder if (dramatically, even) reducing the priority of some of the original workload exposing the problem, not just when racing for a lock but when doing the actual work quanta, would help. My thought here is that your email-sending is higher-prirority and will at least push some of the workload out of the way in reasonable time, giving you back some responsiveness.
I'm surprised if Windows doesn't offer up a high-throughput/latency-tolerant QOS for threads.
A QOS does not directly help. The only thing I am aware of that can help is fair locks, or occasionally fair locks, so that the lock is given directly to the waiting thread, instead of being made available to all.
I have yet to hear of any other solutions.
In my case it was actually being triggered by a piece of monitor management software that came with the LG ultrawide monitor that I use. No significant CPU load, no memory issues, plenty of cores in an older HP workstation with a Xeon Processor, 48 gig of RAM and a Samsung SSD. When that display management software was running a text editor couldn't even keep up with displaying text as it was typed.
Edit: rereading some of the original 2 articles, yeah, I'm not going to be running the same kinds of tests - even if I did it'd take too long to develop the knowledge base to be able to interpret my results adequately.
I think that the answer is that they didn't. That was just one of the bazillion counters that came along for the ride. I believe that that counter has been removed in the latest OS, thus squishing this bug in another way.
I don't understand WMI, but it sounds really weird. One peculiarity is that once some program asks for counters "WMI refreshes the list of counters every 2 minutes until the WMI helper process closes due to inactivity."
So that's great. IT asks for some data, they get memory scans that they don't want, and those scans are repeated every two minutes for a while (ten minutes?) even though nobody is looking at the results.