In the past, each file/folder/icon on the OS X desktop consumed an additional non-trivial amount of memory. So it wasn't irrational then.

Today, almost everything is a trivial amount of memory.

Removing icons from my desktop sped up my old XP install. Noticeably.

Not that irrational.

Lots of filesystems slow down with tons of files on the same folder.

And your window manager has to draw all their icons and potentially render (and hopefully cache) previews when you see your Desktop, which is also costly.

I've seen OSs take their sweet time (4-10 seconds) to render all icons on a cluttered Desktop.