In my experience, it pays to be open with product managers and QA about what you are working on, how you are implementing it, what the risks and benefits are, etc. For example, it helps QA to focus their testing efforts on the areas that are most likely to be broken by a refactor. And having some understanding of how an app is built and where the problem points in the code lie helps product managers to develop a sense of how much effort a certain change might take, and reduces the chance that they will be surprised by some behavior in the app as the result of a change.
And if someone on the team wants to question me about the cost/benefits of a certain refactor and discuss alternatives, I think that is totally fair game. Same way that if I want to ask a question about the cost/benefits of a new feature and discuss alternatives, I feel empowered to do so. It's not about asking permission so much as it is about developing a shared understanding of the costs and benefits of the decisions that we make, and working together to optimize them.