You could start with macports or homebrew. Those tools allows you to install open source software in OS X.

One example is XQuartz the X11 port for OS X http://xquartz.macosforge.org/trac/wiki/Releases

If you know what are you doing probably this is a good point to start, and if you need other libraries then you should look there too.