If you're going off of "feel" instead of statistics then you're doing it wrong. Period.
I disagree, at least to a point. Take an experienced Operations/monitoring guy who's been around the block more than once, then sit him in front of the monitoring utilities for a new company developing a new service.
Then, take a total newbie and put him in the same place.
Train them both to equal skill on your tools and operations.
Who do you think will make the most proper calls? Why?
At this point, those statistics and that documentation do not exist yet. What constitutes a "false positive" vs a "drop everything and spin up more VMs and get on the load balancer" can be more of an art than a science, especially when you're first starting out.
As hokey as this sounds, certain systems have a "personality" that varies between installations and companies, that nothing short of day to day use will educate one in.
But the original article was talking about A/B testing, and that is the context that I was thinking about. There you both can and should use statistics.
There are plenty of places that require intuition and these are the places that errors are often introduced. Statistics is an art.
If you are the one doing the collecting and analyzing and you just think it feels like enough data but you can't actually reason to that point, then yes, you are doing it wrong.