I am a native english speaker (born/raised in California) and also very self conscious about the way I sound for different reasons. I suspect many of us have this kind of internal battle, even if it seems relatively insignificant to other people.

FWIW, I think your accent is interesting and enjoyable to hear.