Now that I think about it, most double stroked characters really represent "concrete" objects, not generic ones. Like ℙ for projective space, or 𝔸 for affine space. Also 𝔽 is often used together with an index 𝔽_q for the field with q elements.
On the other hand, Wikipedia[1] agrees with you, that 𝕏 is "occasionally" used for metric spaces.