Is there support on OS level for that? Something that rewrites existing binaries on the fly and paralelises where possible? Is it even possible?
You mean for a hash table? I don't think you'll be seeing that any time soon. Hash computation will almost certainly be faster on the primary cpu then just the scheduling and waiting overhead. And then the GPU isn't particularly good at any pointer chasing required for the rest of the lookup.