If YouTube provides a service that removes ads ONLY for $4.99/month, they would already have my money. However they have to bundle it with YouTube Music and other stuff into "Premium", then never mind.
It was exactly what you want, a much lower priced subscription which just removed ads, and didn't include any other premium features. I happily paid for it.
It was discontinued this month.