A large part of that, from what I've encountered (by no means a representative sample), is due to Unix makefiles expecting/requiring X in specific-location-Y, or making things stringly-typed[1], or outright artificially restricting the systems it thinks it can run on, failing fake sanity checks up-front. Or other bad code.
But yes. OSX is not a direct match to other Unix-like systems, and a lot of source compiles not tweaked to handle OSX (many are) will require changing a couple things. Though Qt worked just fine for me, so who knows.
[1]: http://stackoverflow.com/questions/2349378/new-programming-j...