One summer between junior and senior years, I took a road trip to Montreal and tried out my French. I was thoroughly embarrassed and lived the OPs nightmare. The residents I spoke with seemed actively offended by my French and would smile condescendingly and speak to me in English.
During senior year, I was able to spend a week in France. It was a totally different experience. I spoke nothing but French for the week and was able to feed and otherwise care for myself. The locals seemed to genuinely appreciate my efforts to speak the language. The capstone was later in the week when a 2am knock on my door woke me up. A younger member of our group was having serious homesickness and wanted to call home, but couldn't figure out how to make the call. I went to the lobby with her to ask the desk clerk how to make the call. I ended up having a great conversation with the desk clerk who was only a few years older than me and was in Paris for university. We chatted for about 30 minutes about life, things we liked, cool things to see in Paris, etc. It was awesome!
For OP, give it a shot. The only way to get better is to do it!