> I never feel quite at ease with sets being some universal foundation

That's not a big issue nowadays since there is univalent foundations, which is based on ∞-groupoids instead of sets.