Things that fly, almost by definition, are inherently less energy efficient than things that roll on the ground.
And engine idling is not a huge deal compared to the amount of thrust that must be expended constantly to keep a drone in the air (along with it's payload).
You'll gain a little on pathing straighter and then lose everything on upward thrust and repeat trips.
Bear in mind, the most efficient way to freight things: Trains. Trains are bigger and heavier, but they are the most efficient at moving things from one place to another.
The benefit you get from a drone potentially is package delivery in the time it takes to order a pizza, because it's automated direct shipping of one item at a time. But drones do not, in any way, offer the promise of fuel efficiency.
I think you just described the potential usefulness of drones in terms of energy used for a delivery. A drone only has to carry itself and it's package - compared to a car that has to carry it's weight, the driver's weight, the packages in the vehicle, etc.
I think it boils down to this: How many/weight of items being delivered to energy efficiency is on a curve. Sure trains are the most efficient method of transporting goods, but there's a reason a train doesn't deliver our packages to our doors. Building such an infrastructure would be insane.
That's why current delivery is tiered to the quantity/weight of the goods you're shipping.
I have a feeling drones will be more efficient for small packages, but larger ones will still be best by car.
As you say, the problem with trains isn't efficiency, it's infrastructure for last mile to our doors. However, we already have that for vans: Roads and driveways to our door are already built.
If I need a cell phone delivered to my front door, which is going to be more efficient: The van that already is delivering a larger box two doors down dropping it off, or a separate trip from the warehouse to my house from effectively, a small aircraft?
Personally I'd like to see a truck that drives around the neighborhood like a little drone-aircraft-carrier, launching drones with packages for a short flight so that the truck doesn't have to stop. I don't think the technical challenges of that are any more complex than simple drone delivery.
If there's one thing drone delivery is not, it's "simple". All of the work we did taming and mapping roadways and navigation has to be re-implemented for aerial flight. Every address needs to be mapped to a correct "spot" for a drone to offload it's package. Drones from competing companies need to figure out how to not collide with each other.
I was going to make a sarcastic comment about my 1 sq ft amazon box carrying my pair of 2 AA batteries crushing someone, but I see that heavy lifter drones can carry up to 500 lbs. I have no idea why someone is next day ordering a large fridge from amazon, but a drone could both carry it and easily kill someone with it.
Now imagine some terrorist hacking Amazon network and doing that deliberately. I can even imagine a not-so-US-friendly country, like Russia, or North Korea doing that.