I do the same thing. There's a particular way of speaking in the North of New Zealand that is associated with poor Maori - when I'm talking to my dad's family, I switch back to talking like that. I mask it in almost all other situations, but it's caught a few friends out at times. One told me it was like talking to a completely different person.
My dad is a tradesperson, and his influence is the one thing I have never been able to completely hide, even in professional settings. It's gotten me a bit of grief over the years, as my response to stress is often to start swearing like a builder.
On one hand, I like being able to code-switch easily. On the other hand, I feel like the only reason I can do it is because I had to, and the inability to completely mask my lower class upbringing has negatively impacted my opportunities at times.
At this point, I just try to own it. If I don't, it's just persisting the negative stereotypes of people of Maori descent never making something of themselves. Gotta start breaking those biases down somehow.