1) malnutrition at a young age. Meaning under 3 years old or so. Malnutricion during breast feeding or pregnancy is the most damaging. After a few years it mostly stops mattering.
2) some chemical contaminants, most notably (and commonly) lead. There are others (and worse ones), like Cadmium, but they're not very common at all.
3) genetic predisposition. Most pronounced, but far from the only effect, is Down syndrome. Of course there are genetic problems, like epilepsy, that are bad enough to make you an idiot.
4) any contact whatsoever with child services has a strong negative effect. If the child is removed from their parents, the effect magnifies dramatically. But children treated do far worse than untreated children in the same situation (and if you look at what happens in practice in such treatment, such as removing the children from school for years, this cannot come as a great surprise)
One almost-exception: voluntary treatment involving the mother (mostly by the mother) has only a slight negative effect. Father does not seem to matter if they're part of treatment. Still does not have a positive effect though. (Absense of a father has a strong negative effect though. I would love to see studies on if you see the same with mothers where the father is the primary caregiver)
I would say that attempting to fix children, in whatever way, other than advising the mother without any other option, is the negative. Children do best when following through on whatever way they develop. Attempting to change things is really bad.
5) drugs, and psychoactive medication are a strong negative, if taken over longer periods (obviously for strong drugs, longer period can be months. For most psychiatric drugs the period is something like a year)
6) lack of attention is a strong negative. Less so than attention by youth services, but it is decisively not good.
That can mean by parents, by peers, by teachers, by ...
At this point though we're talking about things that will definitely leave your IQ at over 90. It won't be catastrophic, whereas child services sometimes produce from normal children that people that have problems like not being able to count at age 17.
And then there are things that don't matter, but are widely perceived to matter.
(All of these have the same caveat. Make them bad enough and of course it will matter. If abuse means you don't survive, of course it will affect you. It essentially never does)
1) getting abused does not influence intelligence (Concrete example: getting scolded daily from age 3 by your mother for not cleaning your room because she's drunk will not make you an idiot)
2) quality of life does not matter. If anything it has a slight negative correlation: worse living conditions will actually make you (a little bit) smarter. (again, there is a level where is most definitely does matter, but that level is very low)