Like, just last week the browsers had to remove a certificate authority from their root cert programs because Kazakhstan was issuing certificates to MiTM traffic. A TOFU model would make it a lot harder to detect and remediate this sort of attack and lots of other relevant attack vectors.
We'd also need to re-solve a bunch of adjacent problems like revocation, renewal/rotation, and transparency, which would probably mean re-introducing the sorts of centralized architectural components and processes that I'm assuming you're trying to eliminate with TOFU.