I did not know this could have been life threatening. That's... comforting...
[1] https://qz.com/677980/opioids-arent-the-most-dangerous-drug-...
[2] https://en.wikipedia.org/wiki/Benzodiazepine_withdrawal_synd...
Doctors are often ignorant of the need to taper benzos over a long-term period, and not rapidly. Some might see you coming to them with a problem quitting an often abused drug as a sign that you need to be taken off of it as soon as possible. Sometimes instantly.
Of course you should always let your doctor know when you're taking less medication than you're prescribed. I suggest consulting with a specialist before trying to quit with a GP or your run-of-the-mill psychiatrist.