All I've heard is the danger of calcium precipitating out of your blood and causing arterial blockages. Apparently D3 potentiates this so that the concentration of blood-borne calcium doesn't need to be as high for this to begin.
I've always been wary of going up to max-dose levels for that reason.