It's just de facto legal because of the difficulty of detecting those instances and the presumably nil return in litigating them.
This seems like it would be fair use. SeanLuke's comment[1] does a brief analysis that looks correct.
It's clearly affecting the market for the original as you are removing yourself from the market for it. Creating a copy for your own consumption harms the copyright holder's ability to market a reproduction to you.
My rough understanding is that normally there has to be some minimal amount of damage and the effect has to be measurable. I am not familiar with the court cases but the term has been quote elsewhere in the comments, De minimis[1].
I also think the argument can be made that the person who makes a personal reproduction was never in the market for the original so there is not commercial impact. Or at least it is an easy argument to make.