Specifically, some objects in-game have pointers to code associated with them ("what to do if this block gets hit by a turtle shell", that sort of thing). The P-switch has one of these pointers assigned to a very special value by coincidence: its pointer points to the memory location where button presses are mapped. This pointer is never supposed to be followed, but by making a bunch of objects very carefully the authors can glitch the game into jumping to that memory address. Once execution is there, they can write a bootloader by making sure the button inputs on each frame correspond to the correct opcodes, letting them execute arbitrary code that they write in on the controller port.
I wasn't involved in the production of this TAS, so I'm not an expert, but that's my understanding of what's going on.