Which actually makes sense given your premises. A good evolutionary "strategy" would be to evolve brains that can make choices far more complex (literally, inelligently) than what can be pre-encoded by genes that change slowly and mutate randomly.
In summary, you are implying that when you have an affair, it's not a choice, but it's determined by your genes. Thus, it's not a moral issue; you have no choice. Thus, you can justify having any particular affair by claiming "genes," and you should just forgive any particular affair your wife has for the same reason. And I'm saying, no, it's a choice.