Hate to say it but have you tried turning it on and off again? IIRC sometimes the state of alerts got stuck in a weird place and resulted in the incorrect state. Toggling it might put it back into the desired state.

I'd also check and make sure Updates is turned off (delete the dependabot.yml).