I agree. People will understand you, but they will also judge you. In my experience, Germans are much more ready to notice "wrong German" than English speakers do with "wrong English". This is just a sad fact.
My aunt, an American who has lived in German-speaking countries for 10+ years and can say almost everything she'd want to tells me she still gets treated less seriously because of subtle mistakes like that in her speech.