Smaller drones have weaker propulsion and can't keep up with the ship its supposed to provide a detection bubble around. Being underwater and having to deal with additional drag also doesn't help. The large size of an aircraft carrier belies the fact that it is the fastest ship in the carrier group and can outrun all of its escorts. Larger ship = more space to pack powerful engines.

That's also a major issue for a lot of other drone swarm schemes such as using drone swarms to counter fighter aircraft. Smaller drones don't have the propulsion to keep up with fighters, and once you make drones large enough to hold the engines that would let them keep up with a fighter you might as well build a fighter instead.