If you serve any third party logic, such as advertisements, your code is just as potentially compromised.
Funnily enough, I don't actually think I do serve third party JS, though. Don't serve ads, don't use external telemetry, don't use JS CDNs. I don't think you have to go quite as far as I do, though - I imagine if your ads are Google AdSense or something, you're probably going to be fine.