The N64 was an expensive platform to launch on though. It was a lot more capital intensive to do a game on the N64, because the cartridges had wholesale costs of $15-20 to produce, plus licensing fees to Nintendo. Sony on the other hand had a cheaper licensing model and CDs could be mass produced at pennies per copy.
The problem is simply that the RPG genre was dominated by Square and Enix. And Square famously abandoned Nintendo after they abandoned the CD format that was originally planned for the N64. In fact, Nintendo had licensed the CD tech from Sony for early prototypes of the N64, and may have inadvertently created their most formidable competitor by doing so. The CD format was superior for doing large pre-rendered graphics and for higher quality sound tracks and audio files. Since these components were more useful for complex storytelling and cutscenes, Square decided to stick with Sony after the split with Nintendo.
You can still play Castlevania: Symphony of the Night now and it looks pretty fantastic, whereas something like Tomb Raider has some compromises you'll need to accept...
Some further examples: https://www.youtube.com/watch?v=D9Nk2lTRD5Y
The fourth Monkey Island game used extremely ugly 3D graphics. There was no good reason. The 5th, much later, was licensed to Telltale and they used ugly 3D graphics. There was an improvement from 4 to 5, but neither game can be compared favorably to 3.
The point of 3D graphics is to make things nicer for the developers, not the players.
That being said, I think modern 3D games look fantastic when done correctly. It's just not the best format for some genres.
There were exceptions, of course; big-name publishers could publish sprite games for tentpole franchises, so we still got Street Fighter, SotN, Mega Man 8, etc. Rules only apply when the rulemaker can afford to enforce them. That's still true today, as we found out when it came out that Google waived the app store fees for Spotify.
However, lack of perspective correct texture filtering on PS1 was already ugly at the time.