As far as I know, the only ongoing costs for the publisher (not immediately recouped via conference admission) would be running their own website and providing downloads. These hardly seem like massive expenses to me.
I don't see any fundamental reason why the model in CS couldn't work for other disciplines even if you don't want to adopt conferences; people are willing to step up and do the work, so why is so much money going to the middle men?