You could release a well tested new feature with perfect ease and see nominal stats for hours after, and still have a problem you don’t see.
Maybe your new Calculator Widget 2000 works great for everyone except Big Enterprise Customer Bob. And maybe Bob doesn’t log on until 7 hours later because you’re in London and he’s in San Francisco. When he starts up the Widget, his mixture of data causes a pathological database behaviour. It’s a performance mistake in your code (that’s hard to find even with rigorous unit testing and manual QA).
Knock on effects follow like your task workers getting overloaded. So you have to saunter back in late on a Friday to fix it. You tiredly misdiagnose it as a usage spike, maybe some big new customer came on. You spin up 20 new worker nodes. Doesn’t help enough, tasks are executing too slowly. You figure it’s one of those mysterious intermittent performance issues in GKE. You begin to upgrade the cluster software in a desperate attempt, your thinking muddled by too many Friday dinner whiskys, to resolve it through the traditional GKE incantations. But cluster upgrades take forever and when you finally see the real problem and roll back your code it’s 2 am and your partner fell asleep on the couch and your kid wakes up at 6 am and God why did you deploy your well tested small new feature on a Friday?