You are looking for some simple explanation and the simpler one is, "greed." It's right there in the personal diary...
> Let he who has not downloaded from Annas Archive cast the first stone
I fail to be convinced that it's harmful for an individual to download something that is digital (i.e. infinitely copyable) for direct personal use (i.e. not making any money off of it or sharing it further) that they otherwise would just not use at all (i.e. not being obtained this way as an alternative to purchasing). OpenAI's actions violate at least the second part of this (they're 100% using it as a way of making money and potentially sharing it more widely, albeit indirectly), and arguably the third one as well (if AI truly is such a game changer with the type of economic potential that the hype claims, paying to license the data properly would still be worth it in terms of long-term profits). Someone who downloads a few books to read that they would realistically not bother buying otherwise and then deletes their copy afterwards is not doing anything close to what OpenAI did, so I strongly disagree with the implication by the comment I responded to above that it's hypocritical for someone in that position to criticize them.
The issue is not, however, whether the material is for 'personal use'. All the books I bought have been for personal use, and if I would not have bought them, the publisher would have lost that revenue.
At the same time, we can all agree that copyright laws in the US are pretty extreme and should return to their more limited historical norms.
To be clear, I'm saying that personal use a necessary condition for it to be ethical, not a sufficient one. My personal view is that if it's not infinitely copyable (e.g. physical goods), not for personal use, or if you would be buying it otherwise, then it's not ethical to do, and I'm comfortable with IP law protecting it. The reason I think personal use is important is precisely what you mentioned here, which is that it's impossible to know if someone else actually would have bought it or not.
The main problem with this conception is that it essentially relies on the honor system to enforce, which is obviously untenable. At that point, the question for me is mostly where to draw the line for the best balance that's actually achievable. My biggest frustration with IP protection in the US isn't even about the letter of the law though, but how it's enforced in basically the exact opposite way that seems fair; big companies consistently get away with at most a slap on the wrist, and whereas individuals can face steep fines or even jail time. That's hardly specific to copyright though, since basically the same discussion is happening around the lack of legal consequences OpenAI has faced for their (at best) negligence in agents running amok and trying to hack websites. DMCA has been repeatedly used to prosecute people for doing stuff like that, but when it's a tech company with mountains of investor money, the legal system seems to be fine with looking the other way.