It seems to me that the First Sale Doctrine makes library lending (among any other use of an individual instance of copyrighted material) legal.
Unless you're arguing that the First Sale Doctrine was only created because libraries are older than modern copyright, I don't think I agree.