Since you write the implementation of the method inside your struct's package, you have the right to manipulate private data there as you would do with any function. I don't understand the problem.
If external packages could add methods to a struct, thus turning private fields into effectively-public fields, the whole struct (all its fields) potentially becomes public. How do you maintain a package if your defined API (public fields/methods) is ignored and instead everything is made available to external packages? To keep compatibility, you could never refactor the package, instead you have to keep all private fields to not break external packages that use them.
I wish I could add methods from outside of my package. I currently run into cyclical dependency issues when I try and define something like...
func (g Game) GetPlayers() []Player { ... }
func (p Player) GetGame() Game { ... }
While having Game and Player in separate packages. It's not the end of the world having them in the same package, but the amount of methods does pile up...