I'm a German living in one of the heaviest wine regions of Europe. Social drinking is a way of life here, there are constant festivals, and it's normal to drink moderately when with friends. Honestly, as horrifying as that may sound to other cultures, it works quite well. However, there are some mostly unspoken rules that must be met for drinking to 'work' well, from my experience:

Not having a drink when together with drinking friends must be fine and accepted by everyone. You must be able to NOT drink when you don't want to. Your consumption must not unacceptably interfere with your life or the life of others. You must be able to enjoy a drink in moderation without escalating further. If you drink excessively, make sure it's only once in a while and in a safe environment.

In your case, there is apparently a fine line between using alcohol to slightly enrich your experience on the one end and feeling "like garbage" on the other. If you can't ride that line because your consumption always escalates, it may be best to cut out alcohol completely - at least for a while - by reasoning about the inevitable outcome when you're still sober. I have two friends who never drink, for that exact reason.