It is absolutely a crazy idea.

Physical analogies don't really work in this situation because of the scale, and the payout.

A physical master key for a building has a few hundred thousand/a few million people that could potentially access it. The payout is low (i.e. the motivation is low on average)

An encryption backdoor to phones has a few billion people that could potentially access it. From anywhere in the world. The payout is huge (access to all iPhones).

Multiple entire governments would dedicate tens of millions of dollars and thousands of people to gain access to a ubiquitous backdoor on something like a phone. The same just isn't true with your building analogy -- they are completely different.