From what I understand he used Mario 64 decomp code.
The game was reverse engineered as 1-1 compilable to the original rom.
[1]:https://github.com/n64decomp/sm64
Fun fact: The people who did this moved on to Zelda 64 Ocarina of time and recently completed that project as well
There are ongoing projects by other people for Mario Kart 64, Goldeneye, Perfect-Dark, Banjo-Kazooie and others although I think the next target for completion may probably be either Majora's Mask or The Minish Cap.