This reminds me of some tech journo who lost all his iCloud photos including all his photos and videos of his newborn because some scriptkiddles wanted to pwn him and steal his Twitter handle.

They basically used a series of escalation of seemingly innocuous personal data to eventually take over everything.

IIRC they somehow got his last 4 credit card details from Amazon then used that to get through the Apple account recovery flow from there they then had his entire key chain.

So "what is a passport number really used for" is like that but on steroids.

I also recall someone's daily life turned into a nightmare because someone successfully socially engineered their power company and maybe phone company to generally harass them. So while I also have no intuition for why or how a passport number is sensitive I rationally know better and am very careful about even how I discard old boarding passes etc

EDIT also worth adding that as a foreigner/non native my passport number is also often used in place for a social security number