I think that's a proprietary in-house emulator known as POPS. The fact that Sony already had an in-house PlayStation emulator for the ARM-based Vita makes their decision to use something else for the PS Classic a bit strange.
They simply reused pops from the psp, with no major changes, so it still ran on a mips system.