I think research shows the most important genes are involved in producing dopamine. Two of the genes that have been found to be implicated most commonly are DAT1 and DRD4, although others are being discovered all the time.
This means that brain signals are not carried correctly, and the signals may be carried to the wrong part of the brain. This, in part, leads to the well known symptoms of ADHD afaik.
So memory definitely a secondary issue.