And bankrupt yourself with 64 GB of banks because they were really expensive compared to just some CD's.

Yes, but bank switching isn't a magic bullet. The code that runs the engine and all core and currently active functions must be in the active bank at all times. You have to either segment your address space so part of each bank always points to the same memory, or duplicate code across every bank that needs it. Either way, your banks are now effectively much smaller than the real address space so you need more banks and more physical memory.

Then you have to do a lot of work to actually manage those banks. Switching banks is kind of expensive, so you need to optimize your memory layout to minimize switching. Then you have to actually manage those banks at runtime.

Really, it's impractical to have more than a few banks. Expanding the N64 out to 64GB is just ridiculous. At that point, you'd be better off putting a modern CPU in the cartridge, emulate the game, and just use the N64 as a glorified display adapter.