Not parts, but the entire thing. Colour propagates through causality. The article I refer to explains pretty clearly about what it means. It's the intent and provenance, not the bits, that are important. If youtube-dl gets classified as illegal, then any trivial modification to it will get the same treatment. Even if you end up slowly replacing every bit of code, if you forked off youtube-dl and didn't change the functionality, that's still essentially youtube-dl.
(You can argue that after enough work done, the ship of Theseus isn't the same ship that sailed into the dock. But the important part is that it's still the ship of Theseus, no matter how many parts you iteratively replace.)
Consider cases like going after someone who took GPL code and republished it as proprietary, or plagiarism, or copyright infringement itself. For the law, it doesn't matter whether or not the bits you have are identical to those of the protected work; what matters is how did you get them. It's the same principle at work here.
> In this case the "meant for copyright infringement" part seems to be some of the unit tests.
In this case, unit tests are evidence that the whole project is meant for copyright infringement. The offending entries serve to establish intent.