Factories ship in bulk. Returns ship, typically, as singletons.
The cost to put one extra box into a shipping container containing thousands of identical units, the pick up of which is scheduled months in advance (so utilization of the overall cargo ship is near ideal), is negligible. Similarly, the cost to break down the incoming shipment, warehouse it and check it in is small, because the factory ensures the correct bar codes are on the outside of the boxes. The worker checking them in is completely stateless - scan a barcode, stick the box in the rack with a light, move on, thus the worker is not specialized and cheaper. From there individual orders are scheduled and collated overnight into trucks that carry hundreds of boxes to individual cities, which are then broken down into the last mile courier.
Now, to arrange the reverse logistics within a bounded amount of time (less than two weeks, say), you are likely to have to send an empty truck to the end address; if you're lucky the route has other packages, but you can't count on it. The reverse logistics sender may not be there when the pick up arrives, so you either have to arrange multiple pickup or risk leaving the item unguarded. Then the box arrives at the local warehouse, with no specific routing information to where the rest of the inventory is, so you have a specialist inspect the box and also make sure there isn't fraud going on - eg people ordering phones and returning similarly weighted bricks. Then if you want that box to make it all the way back to the overseas factory where it was produced, you're probably booking it on an airplane, and not a cargo ship.
20x is actually on the low end. A wall wart power supply might run an incremental cost of about $1 to ship from Shenzhen to your door. Sending that same thing back to the factory by FedEx - a process that takes about a week with customs and pick up delays - probably north of $50 for the 'rack rate' retail shipper.