Adding custom RISC-V instructions to an open-source rv32 SoC

The core should trap an illegal instruction on unrecognized custom instruction opcode, and the handler could call an emulator. Just like if one issued a floating point instruction on a core without the F extension. Of course that means whomever writes that trap handler and emulator needs to actually keep up, and custom opcodes are unlikely to be unique across all RISCV implementations. So it’s theoretically possible but you know that saying about theory and practice…

Custom opcodes have a protected spot in the opcode map; but there's nothing that prevents two different CPUs from using the same custom opcode pattern to mean two different instructions.

Conceivably, you could trap, look up something like the mvendor/march CSRs to triangulate CPUID, and Do the Right Thing, but you usually added a custom opcode because you believe your performance/use-case depends on it.

Basically, your Option #3: "don't distribute binaries with custom instructions." If it's so important, it should get pushed through the standards process and then the binaries can be portable.