It doesn't answer the question. The questioner is asking for your help in making a decision. If you are completely neutral about the two, then most people answer with the phrase, "Either one is fine". Saying just "yes" sounds like you misunderstood the question --- unless you say it with a wry smile, showing that you mean to be funny in your answer.
(I suppose occasionally someone could ask the question with a different inflection, with the last word at a higher pitch, meaning, "Are either of these fine with you?" rather than, "Please pick one.")
> Similarly with the boy or girl question, "no" may be stillborn.
The sex is set at conception.