I'm very confident that in the majority of cases, people who download stuff would not have bought it at any price that the publisher would accept, if piracy wasn't an option. Which means that by the definition of harm I think you are using, the majority of cases would not be immoral. It seems to me the immoral cases would be the exception.
I guess it makes more sense if you also say that consuming something without paying for it is immoral even if you would never have paid to consume it. I find it hard to agree with that though, as I don't see who's harmed.