> swarm of drones
That swarm of drones will almost certainly be spotted by the AEGIS air defense network (the cruisers and destroyers that escort the carrier), and shot down. Cruisers and Destroyers serve as the "screening" role for the Carrier, you will always see the cruiser/destroyers first in the Carrier Strike Group no matter what angle you approach from (and vice versa, they will see you first before you get to the Carrier).
This gives the cruisers/destroyers a chance to shoot down the drone swarm (or whatever air platform you're using). If its too much for the sea-based air defense systems to handle, they'll radio for assistance from the Carrier and F18 / F35 jets will be dispatched, beyond the horizon, before the swarm even sees the Carrier or pinpoints its location.
That's the difficulty of fighting a Carrier. Not the Carrier itself per se, but the entire team that works together. AEGIS cruisers have the best radar systems the US Navy can offer, and their large power plants can make their RADAR systems more sensitive / higher power than anything you can possibly hope to put into a drone.