I've had the same issue. Even when I take low doses of 400iu, vitamin D causes extreme anxiety and mood issues that only go away a couple of days after I stop taking it. I've had to completely cut it out and can't take supplements that contain it. I've also taken various brands so I would assume the additives would be different but who knows. If you search this symptom online, you'll find a lot of people with the same reaction but never any explanation.
Definitely interested to see if anyone has had this same issue and was able to fix it.
Anyway, just a word of caution to anyone reading. Because even 400iu makes me feel very uncomfortable. I don't want to know what would happen at 4,000iu.