Windows Timer Resolution: The Great Rule Change
randomascii.wordpress.com
randomascii.wordpress.com
- Slack (sometimes) sets the global timer to 1ms when it is in the foreground, but restores it in background
- Spotify sets the global timer to 1ms, no matter what. Even if it isn't playing.
- Skype sets 1ms, if started at Startup (which it defaults to), even though I am logged out and it just has a tray icon. But when I manually start it, it doesn't (always) set it to 1ms.
- VSCode will set it to 1ms when you are interacting with it, but will eventually revert to 15.6ms if left alone (even if it is still in foreground).
- Firefox doesn't appear to set it (on its own; I presume that if I opened a tab that was using a low setTimeout or requestAnimFrame it might).
Spotify is interesting. A lot of people probably have that app, and since it sets 1ms unconditionally, it would have been setting fast-timer mode prior to the 2004 update, which could inadvertently "speed up" whatever games people were running.
That includes my own game, which uses a foreground sleep of as low as 1ms to try to hit its time target, and I don't call timeBeginPeriod. I guess I'll find out when I get the 2004 update.
Yup, I wrote such a (small, freeware) game 15+ years ago. I wasn't aware of timeBeginPeriod at the time, but I observed that for some inexplicable reason, the game ran more smoothly when Winamp was running in the background. :-)
Our models of computer timers are woefully inadequate. These things execute billions of instructions per second. Why shouldn't we be able to schedule a timer at sub-millisecond resolution?
Answer: we can. But the APIs are very old and assume conditions no longer present. Or something like that. Anyway, they don't get the job done.
Everybody seems to start a hardware timer at some regular period, then simulate 'timer interrupts' for applications off that timer's interrupt. If you want 12.5ms but the ol' ticker is ticking at 1ms intervals, you get 13 or so depending on where in an interval you asked, it could be 12.
Even if nobody is using the timer, its ticking away wasting CPU time. So the tendency is, to make the period as long as possible without pissing everybody off.
Even back in the 1980's, I worked on an OS running on the 8086 with a service called PIT (Programmable Interval Timer). You said what interval you wanted; it programmed the hardware timer for that. If it was already running, and your interval was shorter than what remained, it would reprogram it for your short time, then when it went off it reprogrammed it for the remainder.
It kept a whole chain of scheduled expirations sorted by time. When the interrupt occurred it'd call the callback of the 1st entry and discard it. Then it'd reprogram the timer for the remaining time on the next.
It took into account the time of the callback; the time to take the interrupt and reprogram. And it achieved sub-millisecond scheduling even back on that old sad hardware.
And when nobody was using the timer, it didn't run! Zero wasted CPU.
Imagine how precise timers could be today, on our super duper gigahertz hardware.
But what do we get? We get broken, laggy, high-latency, late timer callbacks at some abominable minimum period. Sigh.
</rant>
it's not just the API, the whole operating system need to be aware of the need to operate at such frequency, because even if the clocks are faster, the software gets interrupted all the time. what good would a nanosecond api do if an interrupt can stall the cpu for a microsecond?
hence the need for a whole class of realtime operating systems. the next question would then be, why aren't all computers using a realtime operating system? because with the relaxed time constraints we get great returns in speed and cost.
I feel your frustration.
The problem is that what you describe is very hard to guarantee in a non-realtime OS. Modern CPUs are pretty damn fast, that's true, but they can't task switch on a dime. The delay between an IRQ being triggered and task being scheduled can vary pretty wildly, especially if the CPU is already under a heavy load.
You can still try your luck with nanosleep() and see how that treats you. It might work. It probably won't always work.
In general if you need to react so fast on some event you probably want to avoid polling at all costs. That's why most OSs are fine relaxing timer constraints for better overall performance. If you can't react on an external interrupt and you really need to be reactive busy-polling is still probably your best bet.
What I want to say is that we already have precise timers usable on modern hardware; as for why Windows doesn't use that properly, I have no idea, but I can guess that's some technical debt in the scheduler.
[1] https://www.kernel.org/doc/ols/2006/ols2006v1-pages-333-346....
(For those about to suggest how it should have been done, the application also stored an atomic revision counter, so the unit test was switched to that instead of a timestamp.)
https://docs.microsoft.com/en-us/windows/win32/api/sysinfoap...
The lack of information and response from Microsoft on this has been quite frustrating.
Win10 2004 timer changes are breaking.
Timer frequency adjustments (timeBeginPeriod) don't seem to affect Windows timers in the same process like they used to do!
Again. In Win10 2004, if you do timeBeginPeriod(1), timers in the same process (other than using the deprecated multimedia timer) seem to only trigger every 15-16 ms or so instead of 1 ms.
This is bad.
As a sidenote about Win10 2004, after timeBeginPeriod(1), Sleep(1) seems to take about 2 ms per call. Or at least it seems to take about 1950 ms to call Sleep(1) 1000 times when I tested it. Confusing.
There is an a ticket opened in the `Feedback Hub` app for it that I've been checking but there seems to be no acknowledgement from Microsoft about this, which is one of the more annoying aspects.
That's when I discovered that doing a "process more data if there is any, if not - sleep(1)" loop is a very bad way of doing it, as on Windows Server 2016 "sleep(1)" means "sleep 16ms". It all worked fine once the timer resolution was changed to 1ms, but yeah, the default value will screw you over if you have anything this time sensitive and are using sleeps or waits on windows.
Edit: reading your comment again I think I see the misunderstanding - sleep wasn't used to do something periodically, if that's what you want to do then indeed, sleep is the wrong tool.
Like, you are focusing on the sleep(1) part too much - what I want to do is "wait until there is more work available to process, and then wake up and start processing as soon as physically possible". On Windows the "as soon as physically possible" part is unfortunately 16ms in most cases, and that's just not acceptable for something as precise as audio work.
If this is merely to free up CPU when you have no work to do, then the actual timing of 1ms vs 16ms doesn't seem like the issue, right? Shouldn't you be instead waiting on a condition that gets set when work is queued? Like an event object, or an I/O completion port, or something that indicates work is ready? Instead of polling for the condition?
> On Windows the "as soon as physically possible" part is unfortunately 16ms in most cases, and that's just not acceptable for something as precise as audio work.
Aren't you contradicting yourself here? If you're doing audio don't you want periodic multimedia timers? Which are built for this purpose and actually also do deliver the resolution you need?
Again, as I said multiple times - the WaitForObject won't wake up any sooner than Sleep(1) will. Only a real proper hardware interrupt could wake up quicker - but that requires special hardware that just wasn't available in that scenario. If it's easier to think about that way, then please assume I said that WaitForObject "sleeps" way more than necessary due to the low timer resolution on Windows.
>>Aren't you contradicting yourself here? If you're doing audio don't you want periodic multimedia timers?
No, because like I said at the very beginning, I was writing a server application processing audio coming in from the clients. So essentially every time packets came in from client machines they had to be processed and sent back - that's why I had multiple threads "waiting" for work and they had to pick it up as soon as possible, with no guarantee how often that work was going to appear. It's not the same as picking up audio from a hardware device on a client machine which yes, has to be done periodically. It's a simple producer/consumer queue problem, but with it being audio data, 16ms wait times to process packets was not acceptable - pretty much no other work is that time sensitive(except for video I guess), yet it wasn't the kind of work that has to be done on a specific periodic timer.
But you don't need (and should neither need nor want!) such a guarantee to do this the way I'm saying! It sounds like you might just be unfamiliar with overlapped I/O? ReadFile(), WSARecvMsg(), etc. all take OVERLAPPED structures that let you pass a HANDLE to an event object that gets signaled. There's also RegisterWaitForSingleObject, WSAAsyncSelect, WSAEventSelect, CreateIoCompletionPort... you name it. Heck, if you just have a thread select() the old-fashioned way, I think it should still wake up when the data comes, without introducing a delay at all. Nowhere should it matter how fast the data is coming, or to force you to wait more than you need to. Am I missing something?
I think what you're missing is that all of the overlapped I/O scenarios completions you describe all originate from a hardware interrupt from some device, like a network adapter.
We were talking about timing, not about waiting for I/O completion.
The comment literally says "packets came in from client machines". And they wanted to service these immediately, not 16ms later. Which is exactly what I'm describing.
Timing requirements don't arise out of the void. Either they're externally mandated based on the wall clock (which is generally for human-oriented things like multimedia) in which case you should probably use something like a periodic multimedia timer, or they're based on other internal events (I/O, user input, etc.) that you can act on immediately. In neither case does Sleep() seem like the right solution, regardless of its accuracy...
Very few real-world exceptions exist to this. In fact the only example I can think of off the top of my head (barring ones whose solution is busy-looping) is "I need to write a debugger that waits for a memory location to change and lets me know ASAP", in which case, I guess polling memory might be the way to go? But it would seem you would want event-based actions...
I don't want to say that there definitely isn't a way to do this - but we never found it.
Sounds like people in gambiting's company are the same.
I/O completion ports is an API for getting better efficiency between kernel and userland I/O. Batching, better scheduling and avoiding WFMO bottleneck. It's great, but it doesn't really have anything to do with timers.
The bottom line here is that Windows timer behavior has changed. This is terrifying.
Except sometimes polling is your only option. Your scenario works great, if the hardware supports that use case and you're not using some intermediate software layer that prevents you from taking full advantage of the hardware.
Other scenario is when Windows is being used to control something time critical. I fully agree one shouldn't do that, but sometimes you just have to. The new timer change really hurts here.
In other words, we don't live in the perfect world, but still need to get the software working.
I also feel like we're somewhat talking past each other.
> It's kind of bizarre to see you trying to interject into it to argue with me as if you know his situation better than himself while even contradicting his descriptions in the process.
I do know highly similar cases I've had to deal with. Not audio processing, but similarly tight deadlines. It was very painful to get it working correctly.
Also sometimes you just need to do timing sensitive processing. I guess you could argue one shouldn't use Windows for that, but unfortunately sometimes we developers don't really have a choice. Doing what you're told pays the bills. :-)
I don't want to use a deprecated API call, timeSetEvent.
> Using Sleep() and depending on it...
Absolutely no one was using Sleep for timing. It just used to be that all Windows timer mechanisms are really the same thing, and Sleep(1) was an easy way to quickly test actual timing precision.
Obviously Windows 10/2004 changed all this.
That's not what dataflow is saying to use. The phrase is "events or timers", not "timer events".
gambiting describes this scenario:
The hardware (network) does interrupt the CPU and supplies an incoming packet to the receiver thread, which does not use a timer.
Then the receiver thread wants to send it to a worker thread.
It sounds like gambiting's implementation has the worker threads polling a software queue using a sleep-based timer, to pick up work from the receiver thread.
Unless there's something not being said in that picture, the receiver should be sending an event to a worker thread to wake the worker instantaneously, rather than writing to memory and letting the workers get around to polling it in their own sweet, timer-dependent, unnecessarily slow time.
Given that it's described as audio data which must be processed with low latency, it seems strange to insert an avoidable extra 1ms delay into the processing time...
This is kind of the point of events (and in windows, messages; not timers or timer-events). So that one thread can wake another thread immediately.
The whole "if your hardware supports it" reply seems irrelevant to the scenario gambiting has described, as the only relevant hardware sounds like network hardware. Either that's interrupting on packet arrival or it isn't, but either way, a sleep(1) loop in the worker threads won't make packet handling faster, and is almost certainly making the responses slower than they need to be, no matter which version of Windows and which timer rate.(+)
(+) (Except in some exotic multi-core setups, and it doesn't sound like that applies here).
I have kind of answered this before - you can signal on an event from the receiver thread, and it doesn't wake up the waiting threads instantaneously, even if they are waiting on a kernel-level object. The WaitForObject will only return at the next interrupt of a system timer.....so exactly at the same point when a Sleep(1) would have woken up too. There is no benefit to using the event architecture, because for those sub-16ms events it doesn't actually bypass the constraint of the system timer interrupt.
>>So that one thread can wake another thread immediately.
The problem here is that the immediate part isn't actually immediate, that's the crux of the whole issue.
This isn't normally the case. You should expect this not to be the case because programs would be insanely inefficient if this happened. My suspicion is there was something else going on. For example, maybe all CPUs were busy running threads (possibly with higher priority?) and so there was no CPU a waiting worker could be scheduled on. But it's not normally what's supposed to happen; it's pretty easy to demonstrate threads get notified practically immediately and don't wait for a time slice. Just run this example and you'll see threads getting notified in a few microseconds:
#include <process.h>
#include <tchar.h>
#include <Windows.h>
LARGE_INTEGER prev_time;
unsigned int CALLBACK worker(void *handle)
{
LARGE_INTEGER pc, pf;
QueryPerformanceFrequency(&pf);
WaitForSingleObject(handle, 5000);
QueryPerformanceCounter(&pc);
_tprintf(_T("%lu us\n"), (pc.QuadPart - prev_time.QuadPart) * 1000000LL / pf.QuadPart);
return 0;
}
int _tmain(int argc, TCHAR *argv[])
{
HANDLE handle = CreateEvent(NULL, FALSE, FALSE, NULL);
uintptr_t thd = _beginthreadex(NULL, 0, worker, handle, 0, NULL);
Sleep(100);
QueryPerformanceCounter(&prev_time);
SetEvent(handle);
WaitForSingleObject((HANDLE)thd, INFINITE);
}For a real-world measurement, see:
>> "Windows event object was used but it takes 6-7us from SetEvent() to WaitForSingleObject() return on my machine
The person measured 6-7 microseconds.
The system timer is not interrupting that fast.
You don't need to wait for timer tick.
IOW, parent was talking about timer wait, not event wait. Timer events are only processed at timer interrupt ticks.
No they weren't. Read parent again, my emphasis added:
>> you can signal on an event from the receiver thread, and it doesn't wake up the waiting threads instantaneously, even if they are waiting on a kernel-level object.
>> The WaitForObject will only return at the next interrupt of a system timer.....so exactly at the same point when a Sleep(1) would have woken up too. There is no benefit to using the event architecture, because for those sub-16ms events it doesn't actually bypass the constraint of the system timer interrupt.
They described their threads in another comment. There's a receiver thread which receives network packets, and sends them internally to processing threads. The processing threads use Sleep(1) in a polling loop, because they believe the receiving thread cannot send an event to wake a processing thread faster than Sleep(1) waits anyway, and they believe the system would need special interrupt hardware to make that inter-thread wakeup faster.
IOW, they are using a timer polling loop only because they found non-timer events take as long to wake the target thread as having the target thread poll every 16ms using Sleep(1).
But that's not how Windows behaves, if the application is coded correctly and the system is not overloaded, (and assuming there isn't a Windows bug).
My hunch is a race condition in the use of SendEvent/WaitForMultipleObjects in the test which produced that picture, because that's an expected effect if there is one. But it could also be CPU scheduler contention, which would not usually make the wake happen at the next timer tick, but would align it to some future timer tick in future, so the Sleep(1) version would show no apparent disadvantage. If it's CPU contention, there's probably significant latency variance (jitter) and therefore higher worst-case latency due to pre-emption.
Above is not correct.
SetEvent/WaitForSingleObject takes a few microseconds to wake the thread. Not milliseconds, microseconds.
WaitForSingleObject doesn't use a timer if you set infinite timeout.
It does not "sleep more than necessary due to the low timer resolution", because it does not use the timer.
Even if you set a timeout, it doesn't use the timer when the wake is triggered by SetEvent. The wake is immediate.
Wake triggered by SetEvent in a different thread doesn't use interrupts or need special hardware. It's purely kernel scheduling inside Windows.
For a real-world measurement, see for example:
>> "Windows event object was used but it takes 6-7us from SetEvent() to WaitForSingleObject() return on my machine
The person is annoyed that it takes 6-7 microseconds, and they want to get it lower.
Here's a Microsoft example of how to have one thread wake another:
https://docs.microsoft.com/en-us/windows/win32/sync/using-ev...
There are other ways for one thread to quickly wake another thread on Windows, but SetEvent/WaitForSingleObject is the easiest in a general purpose thread.
What I suspect happened in tests where it appeared that "WaitForObject "sleeps" way more than necessary" is that there was a race condition in the event trigger/wakeup logic between the two threads, causing the worker thread to depend on timeout behaviour of WaitForSingleObject instead of being woken reliably by SetEvent. That is is actually a very common bug in this kind of wakeup logic, and you have to understand race conditions to solve it. When the race condition is solved, SetEvent wakeups become consistently fast.
To add, it's pretty shocking there's nothing better in the kernel mode either. It's just as bad as in the user mode.
You actually have to generate an interrupt on a hardware device (like a PCI card) to get good timing performance out of Windows!
Correct me if I'm wrong, but if it's hardware based IRP completion (like DMA done, network card received a packet or whatever), your thread could wake up very soon after I/O completes, long before the next timer tick.
But this is not something you can use for timing, unless of course your hardware is built to do so.
It's not a lot but it adds up. If it's only on your servers then whatever, but if the code goes to clients then please wait on an event.
So no one would rely on Sleep for periodicity. Getting reliable timing performance out of Windows is... challenging.
So ultimately it's just simpler to talk about Sleep. Windows developers who have to deal with it understand that same mechanism (which used to be determined by the global timer frequency, which you could adjust by calling timeBeginPeriod) affects all of the different timers.
Really? On versions prior to 20H2? This gives me 16ms resolution for SetWaitableTimer but 1ms for timeSetEvent:
#include <tchar.h>
#include <Windows.h>
#include <MMSystem.h>
#pragma comment(lib, "winmm.lib")
static unsigned int prev_time = timeGetTime();
VOID CALLBACK callback(LPVOID lpArgToCompletionRoutine, DWORD dwTimerLowValue, DWORD dwTimerHighValue)
{ unsigned int now = timeGetTime(); _tprintf(_T("%u\n"), now - prev_time); prev_time = now; }
void CALLBACK callback(UINT uTimerID, UINT uMsg, DWORD_PTR dwUser, DWORD_PTR dw1, DWORD_PTR dw2) { return callback(NULL, 0, 0); }
int _tmain(int argc, TCHAR *argv[])
{
enum { msec = 1 };
HANDLE handle = CreateWaitableTimer(NULL, FALSE, NULL);
LARGE_INTEGER due = {};
SetWaitableTimer(handle, &due, msec, callback, NULL, FALSE);
for (int i = 0; SleepEx(100, TRUE) == WAIT_IO_COMPLETION && i < 30; ++i) { }
CloseHandle(handle);
MMRESULT id = timeSetEvent(msec, 1, callback, NULL, TIME_PERIODIC | TIME_KILL_SYNCHRONOUS);
Sleep(100);
timeKillEvent(id);
} > clockres && Temp
Maximum timer interval: 15.625 ms
Minimum timer interval: 0.500 ms
Current timer interval: 15.621 ms
0
15
16
...
16
15
2
1
2
...Re: your troubleshooting: wild guess, but one thing to look into (if you ever encounter it again) might be changing the "Processor scheduling" setting in sysdm.cpl -> Advanced -> Settings #1 -> Advanced to "Programs" instead of "Background services". Not sure if that affects timers, but I can't think of much else (besides the hypervisor or hard-coded kernel differences) that would cause this.
So, sure, if you change Windows timer interrupt frequency, that's what you get. It'll of course also affect Sleep while active. Back to the square one.
And very alarming.
On Win10/1809 timeBeginPeriod does affect SetWaitableTimer resolution, as I'd have expected.
On Win10/2004 timeBeginPeriod doesn't seem to affect SetWaitableTimer resolution?! This could certainly break things!
SetWaitableTimer documentation lets you understand timeBeginPeriod is expected to affect the resolution. Of course, depending on hardware... but in this case both tests were running on same hardware.
"You can change the resolution of your API with a call to the timeBeginPeriod and timeEndPeriod functions. How precise you can change the resolution depends on which hardware clock the particular API uses. For more information, check your hardware documentation."
https://docs.microsoft.com/en-us/windows/win32/api/synchapi/...
On 1809 same took about 1.3 seconds.
I'm at loss... Windows timers have yet again managed to do really weird, unexpected things.
I really hope this mess doesn't affect device drivers, or there could be a world of hurt coming up.
Edit: Never mind. Seems it might not be the APCs. Maybe some callee of Sleep(Ex) is just rounding up the wait interval to a time slice somewhere. Not sure... would need some digging.
If you don't want two of the process running at once for any reason, a check-do-sleep loop can be easier to reason than debouncing an interrupt based approach. Less efficient, but often efficient enough especially if you are just throwing something together quickly with a view to if being PoC or a temporary utility.
You sure? According to e.g. the article it results in "sleep anywhere between 1 and 16mSec". But this illustrates part of the problem here: arguably problematic use of Sleep() where other solutions are more appropriate and people not reading the documentation and/or not realizing what it means.
[1] https://developercommunity.visualstudio.com/content/problem/...
In my opinion this still remains the conclusion, as it has been for the past decades. I cannot remember when I read a bit on Sleep() behavior and timeBeginPeriod() but I remember that what I read was enough to make clear you just shouldn't rely on these (unless you're 100% sure the consequences are within your spec and will remain so), also not because the workarounds are also widely known (IIRC - things like using WaitForSingleObject if you need accurate Sleep).
You won't get any more timing accuracy with WFSO.
"Windows 10" is too much of a moving target, and it's about time Microsoft stopped trying to pretend to the public that it's all one operating system. If they're not going to usefully document potentially breaking changes, they should at least do us all the courtesy of bundling these changes up into a release that increments the major version number, and gives us the option of not upgrading for a few years instead of having the software as a service model forced upon us.
(Given what I've written above, it probably won't surprise you to find out I recently spent a decent chunk of time trying to find a workaround for a different undocumented change brought by Windows 10 version 2004).
Any chance you could share this so those of us who haven't dealt with it yet might have a heads-up?
Shouldn't this at least be per-core rather than global? Then most cores can keep scheduling at a low tick rate and only one or two have to take care of the jittery processes.
Like it might be much (even 1000x) slower to read QueryPerformanceCounter due to cross-CPU synchronization issues.
Seems like a good move to me - just a bit of a shame a few applications might break.
Why does it require that? Timeouts should normally be on the order of minutes. Why does Chrome need timer interrupts to happen many times per second?