"Yeah, nah" means one of two things. Most frequently it means "I agree with you that the answer is no", but it can also be used to say "yes I can see how that point is valid, but I disagree with the conclusion". Tone and context are used to differentiate between the two uses, particularly how one says the "nah" (in agreement or not). So for example: "it doesn't look like it's going to rain today aye" could be answered with a "yeah, nah" to indicate that one agrees that it won't rain today, or a "yeah, nah" to indicate that, while it may look like it won't rain, the respondent still believes it could rain.

I believe "nah, yeah" is used in an analogous way, but in my experience it's less commonly said than "yeah, nah".