Avoid all wheat, oats and soy products for a week. See how you feel.

Turned out I had a gluten sensitivity for the longest time, and I felt loads better going off all gluten.

Soy also acts as a phytoestrogen in the male body - could potentially be counteracting testosterone production.