The idea of drone swarms doesn't go well together with aerodynamics and basic physical intuition. If you shrink an aircraft down, the aerodynamic cross-section (i.e the drag force) scales with the area (scale^2), but your engine thrust is going to drop roughly by the decrease in volume (scale^3).

So you end up losing maximum airspeed and fuel efficiency (in terms of the mass you're moving) the smaller you go. Unless the drones in your swarm were really big, it doesn't work out.

Although, I imagine we'll see some smaller, unmanned jet fighters in the future (assuming someone figures out how to control something like that remotely, or autonomously). A smaller aircraft has the advantage of a smaller radar cross-section and being more difficult to hit. Doing away with the pilot cuts out a lot of weight and frees up room for a larger engine and fuel tank, offseting the downsides of the smaller size somewhat. There should be a sweet spot where that works out.