But clearly the neurological disorders are rare, as Brazil has had saturation level Zika exposure and a noted uptick in birth defects, it has not had a similarly sized (or even noted that I can find) uptick in neurological disorders.

This is made even more interesting because the set of people who have contracted the virus and have brains, is significantly larger than the set of people who have contracted the virus and are pregnant (which is a proper subset of the above group). So from a risk perspective it seems less of an issue than West Nile in men/non-pregnant women.