“Cant’t take a joke” is a terrible excuse for making people uncomfortable.
People are sensitive, impostor syndrome is extremely common. The people who work for you are always going to be worried about their job at some level. Don’t take advantage of your power to make a joke that is, at best, not funny and at worst incredibly anxiety-inducing.