Disabling background play in the YouTube app is driven from the licensing deals they have with record labels. If they don't (make a reasonable attempt to) show the ads they don't have permission to show the video/play the music.
I suspect they could theoretically allow background play for other content but that would be confusing to users.