“Not janky” is a weird way of describing the N64’s graphics. Sure Mario looked good, but have you seen most other games on that platform?
If you took the same animated scene and rendered it on the PS1 and the N64 side by side, the N64 would look better hands down just because it has an FPU and perspective texture mapping.
[0] Polygon jittering caused by the PS1 only being capable of integer math, so there is no subpixel rendering and vertices effectively snap to a grid.
I thought the problem was that it only had 12 or 16-bit precision for vertex coords, which is not enough no matter whether you encode it as fixed-point or floating-point. Floats aren't magic.
The N64 in comparison had an accurate and essentially modern (well, "modern" before shaders) graphics pipeline. The deficiencies in it's graphics were not nearly enough graphics specific RAM (you only had 4kb total as a texture cache, half that if you were using some features! Though crazy people figured out you could swap in more graphics from the CARTRIDGE if you were careful) and a god awful bilinear filtering on all output.
https://filthypants.blogspot.com/2014/12/n64-3-point-texture...
Interestingly, the N64 actually had some sort of precursor in form of the RSP "microcode". Unfortunately there was initially no documentation, so most developers just used the code provided by Nintendo, which wasn't very optimized and didn't include advanced features. Only in the last years did homebrew people really push the limits here with "F3DEX3".
> and a god awful bilinear filtering on all output.
I think that's a frequent misconception. The texture filtering was fine, it arguably looks significantly worse when you disable it in an emulator or a recompilation project. The only problem was the small texture cache. The filtering had nothing to do with it. Hardware accelerated PC games at the time also supported texture filtering, but I don't think anyone considered disabling it, as it was an obvious improvement.
But aside from its small texture cache, the N64 also had a different problem related to its main memory bus. This was apparently a major bottleneck for most games, and it wasn't easy to debug at the time, so many games were not properly optimized to avoid the issue, and wasted a large part of the frame time with waiting for the memory bus. There is a way to debug it on a modern microcode though. This video goes into more detail toward the end: https://youtube.com/watch?v=SHXf8DoitGc
It was basically the haze console.
Wiggling is down to lack of precision and lack of subpixel rendering, unrelated to Z buffering. Z buffers are for hidden surface removal, if you see wiggling on a single triangle floating in a void, it's not a Z buffer problem.
When you see models clipping through themselves because the triangles can't hide each other, that's the lack of Z buffer.
MVG isn't exactly an expert on 3D graphics.
Jittery polygons refers to the artifacts you get when polygon vertices are snapped to integer pixel coordinates, rather than taking into account subpixel positions. Quake did not have this issue, despite not using floating-point calculation in it's rasterization. It did use floating point when texturing spans, after rasterization, but this was more of an optimization than a fundamental requirement for accurate texturing :)