That's true but they can take (nearly) straight paths to their destination without stopping and idling, and are smaller and lighter.
And engine idling is not a huge deal compared to the amount of thrust that must be expended constantly to keep a drone in the air (along with it's payload).
You'll gain a little on pathing straighter and then lose everything on upward thrust and repeat trips.
Bear in mind, the most efficient way to freight things: Trains. Trains are bigger and heavier, but they are the most efficient at moving things from one place to another.
The benefit you get from a drone potentially is package delivery in the time it takes to order a pizza, because it's automated direct shipping of one item at a time. But drones do not, in any way, offer the promise of fuel efficiency.
I think you just described the potential usefulness of drones in terms of energy used for a delivery. A drone only has to carry itself and it's package - compared to a car that has to carry it's weight, the driver's weight, the packages in the vehicle, etc.
I think it boils down to this: How many/weight of items being delivered to energy efficiency is on a curve. Sure trains are the most efficient method of transporting goods, but there's a reason a train doesn't deliver our packages to our doors. Building such an infrastructure would be insane.
That's why current delivery is tiered to the quantity/weight of the goods you're shipping.
I have a feeling drones will be more efficient for small packages, but larger ones will still be best by car.
As you say, the problem with trains isn't efficiency, it's infrastructure for last mile to our doors. However, we already have that for vans: Roads and driveways to our door are already built.
If I need a cell phone delivered to my front door, which is going to be more efficient: The van that already is delivering a larger box two doors down dropping it off, or a separate trip from the warehouse to my house from effectively, a small aircraft?