How come that Nintendo decided to use them (according to reverse engeneering finds)? If they are not suited, then Nintendo should know that right?
The N64 doesn’t run any OS. It’s just firmware that invokes a ROM which runs bare metal.
The Switch, however, does have an operating system.
There is around 20 years difference between the two games consoles. That’s 20 years of Moore’s law. 20 years of consumer expectations of fast processors and fancier graphics. And 20 years of evolution with developer tooling and thus their expectations.
You cannot compare the two consoles in the way you’re trying to. It’s like comparing a 1920s racing car to a 2020s F1 car and asking why they are so different. Simply put: because technology has advanced so much in that time it’s now possible to do stuff that wasn’t dreamt of before.
I don’t think there’s much to be gained in speculation about proprietary operating systems running on newer hardware though.