I think this hits the nail on the head. I don't think the new interface is bad per se, it's just optimized for a different goal than the one of most current visitors of archive.org. It would probably work best as a page dedicated to "dumb user discovery" - which might very well be an user-base worth pursuing.
This is one of those cases where creating user personas and trying to come up with different interfaces optimised for each of them would probably be the best approach.