This guys has clearly done a lot of work, building on top of the existing reverse disassembly of the Mario 64 codebase and has a great understanding of the heavily dissected 25 year old hardware. And that's where my paise stops.
The level of snark here is outstanding. Maybe its a cultural thing, maybe he's trying to hype up his Patreon, but he comes across to me as a complete egotistical a-hole. Does he really think the Nintendo programmers were 'smelling the grass' rather working 24/7 to get a ground-breaking game shipped. He does his best to call the team who made several of the most influential and important video games of all time incompetent.
Breaking down his snark:
- The RSP/RDP co-processor had microcode that the game programmers interacted with (you generated a rendering command list on the CPU and called a library function to start the RSP processing that scene). So any criticisms of things the Mario team could have moved to the RSP (ie billboard calculations) are short-comings of the microcode made available by the NCL driver/hardware team - post launch there were regular revisions to the microcode and optimizations made available.
- A lot of the matrix operations were provided in library form to game teams and you were encouraged (forced) to use them rather than rolling your own - there was probably a concern that an overly aggressive optimization would send bad data to the RSP/RDP and cause issues on current or future hardware revisions. The divide by zero checks were likely there for similar reasons - the RSP microcode could run in a mode that skipped near-plain clipping of triangles, which was faster but would blow up spectacularly (huge jaggie triangles all over the screen) if you had triangles that did indeed clip into the near plain.
- I have not seen the Mario 64 source but I imagine some of the lowest level code was written in mips assembly but has been decompiled into C. I may be wrong here, but if that is the case there were likely branch-delay-slot optimizations and cycle counted operations that have been trampled by the reverse engineering. I may be off-base here so would love to know if that is the case?
- Yes the gameplay code has unused variables, duplicated code, etc. But this was a brand new genre of game, there would have been a lot of experimentation, tuning, things would have been in flux until shipping and there certainly was no time to refactor and run the risk of breaking something.
- I suspect the debug flags in gameplay code were to avoid compiler bugs. The mips compiler would sometimes emit some back-back floating point operations that caused the N64 mips cpu to hang (but was fine for all other mips processors). That may have affected the gameplay code with optimizations enabled. Later there was a command-line tool you had to run on your game elf that would look for this bad combination of instructions (and you would then change the offending code until the compiler did not emit the 'bad' instruction combination). Gcc was not allowed by Nintendo until much later, and had its own issues.
- Older floating point units were less predicable with numerical accuracy, rounding errors etc. If early testing showed differences in game control/feel between the debug and optimized builds then I can see why the team would be more comfortable shipping with debug code rather than risking a bug that was not picked up until after you manufactured a million or two cartridges.
- The memory bank conflicts were well documented and games worked around/with that knowledge. Mario 64 was likely well under development before that information was known. As others have pointed out, using the memory expansion module is cheating - that gives you a whole bunch of un-conflicted memory banks to use. 2x the memory also gives a lot of optimization oppertunities.
- I don't see a 6x speedup. I see ~10% gains in most systems and a best case frame time improvement from 29 fps to 41 fps. Which is pointless. An NTSC tv runs at 60Hz, you either hit 60fps or 30fps, anything in-between results in horrible tearing of the screen or your frames bounce between 60 and 30 and movement feels horrific. The game was optimized to hit targets and to ship.
- His split screen optimization is crap! Reducing load on the RDP (rasterizer) by killing rendering area (big black bars on the screen) is lazier than most everything he snarked on. The slightly better approach would be to ab-use the TV output timing registers so that the screen buffer can be less than 320x240 but still fill the screen. You trade black bars for smeary sludge but on the target NTSC TV that was generally the way to go if you didn't want your game rejected by Nintendo QA. Or actually re-write the renderer and reduce the amount of pixel fill.
- And a minor nit pick, the sound processing was not done on the CPU (as he claimed). The cpu built audio command lists that the RCP then used to generate the output waveforms.