What are the best practices to ensure I don't mess up? E.g. always have current project/cluster name in your prompt. What other things would you recommend?
I'm afraid of accidentally running something on wrong gcp project / cluster combination out of mistake which will inevitably happen, taking down prod or something.
I think GitOps as a philosophy helps here a bit. All changes to your cluster(s) go through git with a PR / review. Generally speaking you wouldn't be running kubectl or any CLI tools on your dev box where you hope you're working against the correct cluster or namespace.
I also like the idea of having a temporary test cluster that you routinely spin up and destroy for testing any type of Kubernetes upgrades or cluster-wide changes. This way you get a real result of how things will pan out before performing it on an important cluster.
I have some bash functions which get/set these defaults, so I will typically make sure I know what I am running against before executing a command.
If your DevOps setup is good, you should not be able to take down prod and it should be easy to rebuild. There should be controls, especially for deletion in prod. There is also rollback like capabilities.