Can someone create malicious ads for YouTube to take over a browser?

You clicked an ad, not YouTube. You never went to YouTube.

To be specific: you clicked an ad that says 'www.youtube.com' next to the title - NOT the url bar. Advertisers can put any domain they want there. It does not need to be a real site. It does not need to be a destination. It is just a user-defined text string and has nothing to do with YouTube.

Effectively, you are thinking that you visited YouTube when in reality you just clicked a link that's

    <a href=malware>youtube.com</a>