Does anyone know they type of setup these probes have? Is it running a custom OS on custom hardware or plain old Linux on a x86 processor?
No idea about OS.
But it is certainly custom hardware and a custom OS, because you need radiation-hardened chips (even for earth-satellites) and extreme reliability.
Are there any space exploration vehicles running Linux?