"cost of operating retail outlet"
The cost of the retail outlet though is spread among many goods. While there are stores that are certainly devoted to one product (and run by the manufacturer or designer) there are more stores that contain many designers. So what what matters it the incremental cost of adding one extra product to the store and what their markup is on the goods.
Also, you will most certainly sell more total products (and lower your costs) by having, in general, multiple channels for those products.
Lastly, as the point I made in another reply here said, you can't discount the amount of what you are paying that goes to the designer for lending their name to the product or getting involved. Tiger Woods is paid to put his name on golf clubs. Ralph Lauren makes shirts and all sorts of goods. People like and buy the label. Some of it is about quality and some is about cache. By eliminating the retail outlet you only remove one part of the costs.
"We eliminate the middleman" is a pitch that has been around for a long time. And yet, people still manage to do business selling through retail channels at the point of purchase. And there is certainly room for both.