They, in large doses, stop your breathing which is close enough and will kill you (like you later said), unlike, say a benzodiazepine which you can take to your hearts content and will just knock you out senseless without killing you (unless you add alcohol or another GABA receptor drug).
They also can send you into a hypotensive crisis if you're hypovolemic, which also will kill you. So I would really be hesistant to call them "quite safe, as drugs go" because they are definitely on the dangerous spectrum. We could list a ton of drugs I'd rank higher on the safety spectrum than opioids in terms of both short and long-term effects, with opioids only "redeeming" quality being their instant reverse agent (naloxone) or partial reverse agents (pentazocine or buprenorphine).