If by rights, you explicitly mean "The right to be paid" then I think you're probably correct.
Although I'd note they're not really interested in having the musician get paid. Mainly just members of the RIAA.
Now - I have some sympathy for that view - I'd also like to get paid, and while I think the RIAA has little real value in the world, some of it's members certainly are working on the marketing and promotion front.
That said, I use many of these downloading services to save conference talks from youtube. So they're attacking a service I use legally for a useful purpose. At some point, ethically, I start to lose ANY sympathy for them as an organization.
Basically - https://www.youtube.com/watch?v=ZNS7IqmZBbA&has_verified=1