This is mostly true; it is perfectly legal to download copyrighted material. The crime is in distributing copyrighted material that you do not have any rights to. This means that I do not need to worry about if Hulu, Funimation, or whatever streaming service I am using is complying with the law and obtained the rights to stream their content.

However, in the case of torrents you are also uploading the content, which is illegal; and I believe there is precedent that you can be charged (although I do not have a case handy).