That can happen if 1) you have an older dimmer not rated for LEDs 2) the dimmer is mis-wired; some are very specific where you hookup the hot and load wires especially if you have a 3-way circuit for the light.