Why isn't memset() async-signal-safe?
boston.conman.org
boston.conman.org
It's just that some kernels fail to do this (or did in the past), probably because the direction-flag requirement is less well known and most programs won't crash if it's neglected.
I don't know what historical architectures may have had a bug with this, but it's not true of x86. If memset isn't signal-safe on modern x86 linux, it's surely not because of EFLAGS state management.
https://lkml.org/lkml/2008/3/5/207
is the first message.
Basically: ABI implementation bug in kernel. https://lkml.org/lkml/2008/3/5/231
GCC started actually relying on the ABI being correct in version 4.3, folks noticed bug.
The alleged bug is that Linux didn't clear DF on signal entry. (This has nothing to do with what is saved or restored. Flags have to be saved and restored and, AFAIK, always were.) The x86 ABI is crystal clear: C functions are called with DF clear. Neither glibc nor Linux cleared it before calling a signal handler, so there was a bug.
But the bug was fixed in March 2008 for Linux 2.6.25. [1]
[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....
As for GNU/Linux, can you assure memset works that way in all hardware platforms supported by Linux?
Even it does, no one intending to write portable UNIX code can rely on it anyway.
I don't know, that's why I'm asking.
I'm not going to try to write code portable to all theoretically possible unices - I doubt anyone has managed that for nontrivial programs, and it's too hard to tell. Hell, I'm content to exclude a fair few systems I know exist (CHAR_BIT != 8, non-IEEE FP). If it's supported on all the platforms I've heard of I'll do it - it's just not practical to hope to be POSIX-complient in a language-lawyer sense.
In the early 2000's, the company I worked for was deploying UNIX based software for GNU/Linux, FreeBSD, HP-UX, Aix, Solaris and Windows (yes Windows, not a typo).
And after reading that thread, I'm not as convinced as I was a few minutes ago that this was an obvious kernel issue. Yes, the kernel mismatched the published ABI, but callee-vs-caller save and setup is not always consistent, and for good reason.
E.g. registers are usually callee-save, since there are many registers and small functions use few of them. (This is what the kernel assumed.) But rarely-set/often-used flags (such as the flag in question) may make more sense as caller-save and even caller-setup, since this reduces save/setup overhead to only the cases where the flag is actually set. (This is what the ABI dictated and GCC assumed.)
First, I don't think that's true nowadays. The discussion points that this (not restoring DF on signal return) is a kernel bug and should be fixed: https://lkml.org/lkml/2008/3/5/531
Second, the kernel tries to avoid doing too much work in the signal return code. It tries hard do _avoid_ heavy XSAVE and just preserve only the needed registers. This is the job of sigreturn(2) syscall btw. http://man7.org/linux/man-pages/man2/sigreturn.2.html
Third, there was a similar discussion about restoring SS (segment stack) register in x86-64 signal code. First patch proposed by Bryan Ford: https://lkml.org/lkml/2005/10/5/176 . Then 10 years later by Andy Lutomirski: https://lkml.org/lkml/2014/7/11/564
Last one was merged into mainline. Then reversed. Sadly. Then applied again: https://github.com/torvalds/linux/commit/6c25da5ad55d48c41b8...
The gist: if you modify SS register in the signal handler you are screwed. The only way around it is to install a trampoline using "the famous dosemu iret hack" described here:
http://www.x86-64.org/pipermail/discuss/2007-May/009913.html
(the site is down, can anyone find a mirror?)
On Linux use signalfd(2) whenever you can http://man7.org/linux/man-pages/man2/signalfd.2.html (Ie: put signal handling back in event loop). That's the only sane way of dealing with signals.
> The conclusion is that DF=1 in x86_64 64-bit code is extremely rare
Depends on how releases are managed, but that might be much much more practical an assumption.
I'm not sure what you mean. This issue is fixed, so SS works exactly the way you would expect it to, unless you do very strange things indeed in which case you might need to fiddle with uc_flags.
The issue is fixed indeed, as for Feb 17, 2016.
FWIW, I don't consider it sad that my first attempt was reverted. I inadvertently broke some assumptions that a real program (DOSEMU) was making, and Linux takes backwards compatibility quite seriously. The second version was better.
Interestingly, Xcode's OS X docs used to link to x86-64.org for the AMD64 ABI, but looking right now, the link is removed (but the text remains), I guess because the site was down for so long.
(Yes, I believe something like card marking is more performant than this in several ways on modern CPUs, but then you need to insert card marking instructions into your code generator... trapping SEGV is easy and only happens in the GC.)
Also there's all the debugging and process inspection stuff that works via signals, all of which depends on being able to really interrupt code rather than just stuff a message in a queue.
Or, if you just need a "dirty page" bit, add a dedicated mechanism for that, which can use hardware features to run much faster without having to trap.
For debugging, we can do much better than ptrace. Linux already has dedicated syscalls to read and write another process's memory. Add some mechanisms to read and write registers, and extend the process stop/continue mechanism to allow single-stepping and stop-on-event (such as stop-on-syscall, or BPF-based filtering). I don't see any reason why debugging a process needs to incorporate signals.
This has similar reentrancy issues to signals.
edit: technically of course full reentrancy (implied by async signal safety) is not strictly required, "only" a fully non-blocking implementation of every function called by the signalfd thread.
Async signal safety implies both reentrancy and non blocking algorithms [1]. You might not need reentrancy but you do need non-blocking. That's really a significant restriction as libraries with non-blocking guarantees are rare.
Out of process handling is a robust solution though.
http://stackoverflow.com/questions/13341870/signals-and-inte...
I have the intuition knowing the HW/ASM & the code but ignoring how POSIX works (I read stevens, but I am not having all my answers) that unices reimplement in SW what the HW does with wires.
Maybe this could be fixable at the HW level? (and maybe I guess requiring kernel privileges)?
https://randomascii.wordpress.com/2013/03/11/should-this-win...
That said, I'm confused why you replied to my comment above, since it's totally off-topic. This thread chain was asking how much state exists; maybe you were trying to reply to another thread?
Actually, you don't. I'm not up to date as to what's done in this direction by current kernels, but a while back, a patch was merged that disabled floating point and MMX operations by default for a process, and enabled it for a while only when some FP or MMX instruction lead to an exception, which then allowed the kernel to avoid saving and restoring FP state on processes that weren't actually doing any FP/MMX operations.
Actually, no, you don't, at least not on _all_ context switches. What you have to save is what the switched-to routine will overwrite. For something like an interrupt handler (where latency often matters very much), if you know it will only modify, for example, eflags and EAX, then you only need save on entry and restore on exit from the handler eflags and EAX. The registers that are not modified remain identical from entry to exit and time is saved by not pushing/popping them needlessly.
When some code is interrupted and the signal or interrupt handler calls memmove, that memmove will set up the direction flag for itself correctly. Its entire execution is nested within the handler. If it is interrupted by a nested interrupt, that nested one will restore the flag.
Now if an implementation of the memset function happens not to care about that flag, so that it changes direction from call to call, that's not a signal or interrupt problem! The flag can have arbitrary value in on entry to memcpy in an ordinary situation not involving threads or signals.
The moral is: never use the looping primitives on Intel without setting up the direction flag, if you care about reproducibility.
It goes without saying that the flag is part of the machine state; any machine context saving mechanism (for async situations) is broken if it neglects that flag.
The list of async-signal-safe functions is available here:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2...
It can not have arbitrary value on entry. x86-64 ABI mandates that the flag is cleared before any function is called (3.2.1 Registers and the Stack Frame: "The direction flag in the %eflags register must be clear on function entry, and on function return.")
I'm sorry, I do not follow. The conforming compiler expects that the functions are called with the clear flag. The non-conforming kernel on the other hand does not clear the flag before calling the signal handler which breaks it.
while(n--)
*m++ = c;
With a call to the built in memset()https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56888
The terrifying thing is while the programmer might know that memset() isn't async-signal-safe and not use it in a signal handler, the compiler may blissfully and silently optimize the code to use memset() anyways. Odd crashes or worse security leaks may result.
Reminds me a bit of old floating point implemented in software. Some implementations had global scratchpad registers. Very weird things happened if you did any floating point operations in multiple threads.
Really all signalfd does is provide an easy way to handle signals in an epoll driven application. If you have a different kind of main loop, then you are probably back to stuffing messages into queues, or dealing with all of the async-safe BS.
More recently, Vista had (accidentily?) 16-byte aligned stacks when using OpenMP with MingW, but the exact same binary crashes on Windows 10 when it tries to use unaligned SSE instructions.
I agree with colanderman; the kernel should be saving every register when the signal handler is entered.
But I won't restart a war that has ended the day x86 architecture won over 68K.
I herby claim that sigreturn() restores the user space state in kernel context: https://lwn.net/Articles/676803/
It actually really, really is. At least from a technical perspective on x64. Having language-level lexical scoping of system-level exceptions is incredibly useful and once you've grokked how NT's trap handler, PE .xdata sections, prologues and epilogues all work in concert, signals just seem barbaric.
Here's an example trapping an access violation:
//
// Prefault the page.
//
TRY_MAPPED_MEMORY_OP {
TraceStore->Rtl->PrefaultPages(PrefaultMemoryMap->NextAddress, 1);
} CATCH_EXCEPTION_ACCESS_VIOLATION {
//
// This will happen if servicing the prefault off-core has taken longer
// for the originating core (the one that submitted the prefault work)
// to consume the entire memory map, then *another* memory map, which
// will retire the memory map backing this prefault address, which
// results in the address being invalidated, which results in an access
// violation when we try and read/prefault it from the thread pool.
//
TraceStore->Stats->AccessViolationsEncounteredDuringAsyncPrefault++;
}
Or an alignment fault: FORCEINLINE
VOID
StoreXmm(
_In_ XMMWORD *Destination,
_In_ XMMWORD Source
)
{
TRY_SSE42_ALIGNED {
_mm_store_si128(Destination, Source);
} CATCH_EXCEPTION_ACCESS_VIOLATION {
_mm_storeu_si128(Destination, Source);
}
}
Or an illegal instruction: FORCEINLINE
VOID
StoreYmmFallbackXmm(
_In_ PYMMWORD Destination,
_In_ PXMMWORD Destination128Low,
_In_ PXMMWORD Destination128High,
_In_ YMMWORD Source,
_In_ XMMWORD Source128Low,
_In_ XMMWORD Source128High
)
{
TRY_AVX {
TRY_AVX_ALIGNED {
_mm256_store_si256(Destination, Source);
} CATCH_EXCEPTION_ILLEGAL_INSTRUCTION {
Store2Xmm(
Destination128Low,
Destination128High,
Source128Low,
Source128High
);
}
} CATCH_EXCEPTION_ACCESS_VIOLATION {
_mm256_storeu_si256(Destination, Source);
}
Or a page fault that has occurred against a memory map backed file (because, say, the underlying network drive has been disconnected): TRY_MAPPED_MEMORY_OP {
//
// Copy the caller's address range structure over.
//
__movsq((PDWORD64)NewAddressRange,
(PDWORD64)AddressRange,
sizeof(*NewAddressRange) >> 3);
//
// If there's an existing address range set, update its ValidTo
// timestamp.
//
if (TraceStore->AddressRange) {
TraceStore->AddressRange->Timestamp.ValidTo.QuadPart = (
Timestamp.QuadPart
);
}
//
// Update the trace store's address range pointer.
//
TraceStore->AddressRange = NewAddressRange;
} CATCH_STATUS_IN_PAGE_ERROR {
//
// We'll leak the address range we just allocated here, but a copy
// failure is indicative of much bigger issues (drive full, network
// map disappearing) than leaking ~32 bytes, so we don't attempt to
// roll back the allocation.
//
return FALSE;
}
Relevant macro definitions: #define TRY_AVX __try
#define TRY_AVX_ALIGNED __try
#define TRY_AVX_UNALIGNED __try
#define TRY_SSE42 __try
#define TRY_SSE42_ALIGNED __try
#define TRY_SSE42_UNALIGNED __try
#define TRY_MAPPED_MEMORY_OP __try
#define CATCH_EXCEPTION_ILLEGAL_INSTRUCTION __except( \
GetExceptionCode() == EXCEPTION_ILLEGAL_INSTRUCTION ? \
EXCEPTION_EXECUTE_HANDLER : \
EXCEPTION_CONTINUE_SEARCH \
)
#define CATCH_EXCEPTION_ACCESS_VIOLATION __except( \
GetExceptionCode() == EXCEPTION_ACCESS_VIOLATION ? \
EXCEPTION_EXECUTE_HANDLER : \
EXCEPTION_CONTINUE_SEARCH \
)
#define CATCH_STATUS_IN_PAGE_ERROR __except( \
GetExceptionCode() == STATUS_IN_PAGE_ERROR ? \
EXCEPTION_EXECUTE_HANDLER : \
EXCEPTION_CONTINUE_SEARCH \
)(For a long time, I thought that literally the only thing that POSIX didn't fuck up is the way they don't have any analogue to MAXIMUM_WAIT_OBJECTS. But then, after I gained more experience with POSIX, I realised that even if I don't understand how, this too was probably also something they got wrong.)
MAXIMUM_WAIT_OBJECTS is also terrible of course, but even apart from that Win32 is not bright, at all. The doc contains so little details (or worse is sometime even false) that the real doc when you start to ask serious questions is ReactOS, Wine, or even the NT/2000 source leaks, and then IDA. Compared to that, Posix is actually documented and implemented mostly correctly by tons of OSes. Its easy to ship an API with basically no spec (so nobody can tell you that there is a bug when they find one - actually even if there were some real specs about Win32 I think there is public no way to report bugs in there to MS!), and that does not have to be compatible with any other implementation...
[0] http://bojackhorseman.wikia.com/wiki/Hollywoo_Stars_and_Cele...!
Edit: Has been updated