I don't mind them visually non-matching, but different materials/thickness, etc. is annoying

I've actually had quite a few people positively comment on my unmatched socks, including on dates. I now wear unmatched socks on purpose.

This. Been doing this for almost two decades.

99.9 percent of the time nobody will notice. You almost have to draw attention to it deliberately before anyone notices. Any nobody ever cares - the sort of person who would you probably wouldn't want to be around anyway.