Whether swearing sounds childish is a cultural convention, particularly in the case of such a mild swear as “shit.” My experience is that it is not something that carries this baggage you describe for the typical person. Of course we’re both commenting from inside our bubbles.