In my experience, the people in my friend group are wary of AI because of the potential it has to eliminate jobs. I think the generally negative connotation comes from this "replacement of human effort" interpretation. If it can be done cheaper and easier with AI, that will naturally supplant effort a human could have made.

Since non-techies are often the people being displaced in this scenario, it doesn't sit well with them when AI shows up in a product they like.

(For what it's worth, I mostly agree with this sentiment.)