Most of the complaints about UB-based optimization that I've seen are from when the compiler uses UB to assume some type restraint that the programmer wishes it did not. E.g.:

  int v = *p;
  if (p == NULL) { /* ... */ }

The compiler uses UB to annotate the type information on p to indicate it's non-null, and then performs type-directed optimization.