The Bill of Rights (first ten amendments to the constitution) is all about protecting us against abuses of government officials. It presumes people in government could be bad, and we need protecting from them. By allowing a government official to seize all our private conversations it could allow a bad government official to go fishing and find things things that they can use against their enemies. There are many, many laws. Many of them aren't enforced, but could be.
For example in some states adultery is illegal, even a felony in some states. Not just in small states, but in New York, Massachusetts, and Michigan. In Massachusetts there hasn't been an adultery conviction since 1983. But if law enforcement was allowed to decrypt an iPhone, and find private bedroom things, it could be used as an attack on a political enemy.
It isn't only "bad people" that should be worried about the government. I think pretty much everyone agrees Martin Luther King Jr. was a good person. But the government at the time didn't think so. When he was killed the first people on the scene were the police officers who were doing surveillance of him from across the street. That is the kind of government we need to be protected from.
The only real defence against government abuse is strengthening democratic institutions, respect for human rights, strong oversight, transparency etc.. End-to-end encryption is quite a shallow/weak protection since a truly tyrannical government can easily make it illegal at any time. A lot of government abuse is also possible without breaking encryption.
So I think the benefit of end-to-end encryption in protecting against abusive governments is often over-stated. But what about the costs? For example technological progress is making the world more dangerous. Should it be easy for a bunch of people to use unbreakable encryption to coordinate the development extremely dangerous genetically engineered bio-weapons from their homes? If one such person is caught should it be really impossible for 5 judges to allow the police to find all those they've been talking to?