Honestly, I don't necessarily agree. For example in Japanese 'iie' sounds very similar to 'yes' or 'yeah' but it actually means the opposite, it means 'no', whereas 'hai' means 'yes'.
If we want to talk about individual phonemes caused by emotional reactions, there might be some truth behind what you're saying, however as soon a we enter the realm of "this word sounds soft so it's positive" and "this word sounds hard hence it's negative" everything collapses.