Mel Kaye's hack was actually performed on an RPC-4000, so the story goes.

When I was doing my CS degree and had plenty of time in my hands for that sort of thing, I had a look at the manuals for the two systems and tried to follow Ed Nather's story.

It looks like Ed Nather misremembered the story (in his defense it had been several decades since he'd seen it) because the whole hack rested on the machine having an index register bit "between the address and the operation code in the instruction word", but the RPC-4000 had the index register bit ("index tag") at the end of the instruction word [1] and nothing between the current instruction and operand address, which is, I think, the natural reading of "operation code" and "address" respectively. There's also nothing between the operand address and the next instruction address. The RPC-4000 word looks like this:

  [COMMAND][OPERAND ADDRESS][NEXT ADDRESS][X]
  |0     4||5    11|12   17||18 24|25  30||31|
                   |              |
             TRACK | SECTOR  TRACK|SECTOR

What Nather remembers could be an overflow of the NEXT ADDRES field, through the index tag, and into the COMMAND field. It depends on whether the RPC-4000 handled overflows by wrapping or saturation (I can't find that in the manual).

On the other hand, Nather's account states that the hack changed an instruction to a jump instruction- and the RPC-4000 does not have a jump instruction, because it doesn't need it: like Nather's story says, every instruction has its own GOTO- in the NEXT ADDRESS field.

Here's some alternative ways the hack could have played out, from what I can tell:

1. The hack used the "Branch control" facility.

The RPC-4000 had another facility, the "Branch control", an internal flip-flop with only one bit that was turned on when an arithmetic overflow was detected. Kaye could have used this instead of the index tag and Nather may have misremembered it as being the index tag.

2. The hack was actually done on the LGP-30

The hack might also have been possible to pull off on the LGP-30, that had two bits between the opcode and the operand address [2] and an uncontrolled jump instruction ("Unconditional transfer" in the manual). Kaye could have kept those two bits set and caused an overflow, as told in the story.

3. The hack was done on an RPC-4000 emulating an LGP-30.

Royal McBee had an emulator written for the RPC-4000, specifically to be able to run LGP-30 programs on the new machine [3]:

  My name is James William (Bill) Bryner ... In 1960 I was hired by Royal-McBee
  to write the assembler for the replacement to the LGP-30, the RPC-4000.

  Mel Kaye designed the RPC-4000 assembler. It was titled ROAR (Royal-McBee
  Optimizing Assembler Routine). Edward W. Dubbs and I programmed that
  assembler.  Following that, I wrote an LGP-30 simulator to run on the
  RPC-4000. This was meant to allow all programs written for the LGP-30 to be
  executed on the RPC-4000 without further programming. A drum computer
  simulating a drum computer is agonizingly slow!

I bet, the blackjack program that brought everyone to the Royal McBee booth would have been top of the line of the programs to be run on the RPC-4000. Perhaps, then, Nather was working on Mel Kaye's "port" of the original blackjack program, not on the RPC-4000 but on the LGP-30 emulator running on the RPC-4000. Judging from the descriptions of Mel Kaye's programs in Ed Nather's account, just having an emulator for the architecure would not necessarily mean that Kaye's programs would run without any changes on the RPC-4000.

I have no clear idea how any of the above could have worked. Just guessing.

_____________

[1] http://www.bitsavers.org/pdf/royalPrecision/RPC-4000/RPC-400...

[2] Here's an example from the LGP-30 manual, as in the article:

    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0
                           (-------)   (----------)(------------)
                           order bits   track bits     sector bits
                             = bring       = 20           = 00
                                       ( address = 2000 )

[3] http://ed-thelen.org/comp-hist/lgp-30.html#Historical%20Note...