I use nano or vim as my editor mainly. The setup I use on Linux is much the same. I don't notice much of a difference in terms of performance between OS X or Linux for the text-based and command-line related stuff that I do. I could sit down and be happy in either OS X or Linux and it wouldn't matter to me which one I'm using. I also use mutt for my email and cmus for playing music in both environments so I am seldom out of the command line for anything.
Although I do find apt-get much more efficient and I can get packages faster and with less fuss with Linux.
I do also use brew on OS X.
As far as I'm concerned, for what I do, there isn't a lot of difference ... I hardly ever use XCode, though.