In the language of do-calculus, counterfactuals have a very specific meaning. Ferenc gives some great examples here: https://www.inference.vc/causal-inference-3-counterfactuals/

The main idea: Let Y be the set {y, not y} and X be the set {x, not x}. y denotes timing out, and x denotes “Kubernetes not starting the pod”. We have some distribution P(Y=y|X=x) (in this case, the probability of “timing out” (y) given “Kubernetes not starting the pod” x. The counter factual distribution is NOT the same as P(Y= not y | X= not x) (the “probability of not timing out given Kubernetes starting the pod”). The counterfactual is P(Y’=not y| X=x, Y=y, X’=not x), or “the probability of not timing out given that it did originally time out when Kubernetes did not start the pod, and given that this time it did start the pod”.

This aligns roughly with what you are saying above.