Yeah that 4KB also had to hold the MIP maps as well, effectively halving that down to 2KB for the full size and then 2KB for all the down scaled pieces.

Ps1 also only had a 4KB cache but it could also dynamically manage it for larger textures but that came with a large performance penalty.

The blur on N64 was a combination of that and small cartridge size. It graphics setup did have some advantages, look at Mario 64 for example. The hills in the distance are usually just a few polygons with some low resolution textures stretched over them. The filtering and perspective correction means it is a very consistent image, you would not be able to do that on PS1 without it becoming a visual mess.

PS1 affine rendering was fast but limited, which for the release window was good enough.