I was involved with arXiv.org and we estimated costs somewhere around $5 a paper, however we were not doing any typesetting (mostly we got submissions in TeX) and no peer review. It may sound blasphemous, but it is not clear how much value you really get out of peer review.
One key is that arXiv was running on a very large scale, if you are considering the cost of the programmers, hardware and such, it is just as expensive to run a small "journal" as it is a large one.
The reason I don't work there anymore has to do with the extreme difficulty we had in finding that $5 a paper. After years they've finally been able to get other academic institutions to put some chips in the pot, but it seems to me that it is much easier to find money for a boondoggle than it is for a lean operation.