I spent a summer doing something very similar to this at a major military radio manufacturer in the mid 90s - it turned out that one of their product lines from the late 70s used an entirely custom 8-bit CPU for which the instruction set had somehow been entirely lost. However, they still had the firmware on a stack of EPROMs. So, the mission was to reverse engineer the old CPU to reimplement it on a modern DSP. Turned out that you can get surprisingly far based on a frequency analysis of things that look like opcodes ("let's assume it has two accumulator registers; that loading is the most common instruction; etc."), making some educated guesses about how the designer would have allocated the opcode bits, and then plonking a HP logic state analyser straight over the top of the 32-pin DIL to check the hypothesis. Fun times :)