This is a huge and under-appreciated concern. It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.
A goal like "improve clickthrough rates" is easy to measure, but without some human insight it's all too easy to achieve it at the cost of overall success. Did you decrease time-on-page? Maybe your visitors feel mislead. Did you decrease conversion rate? Maybe your new visitors don't actually want your product. And so on, indefinitely, including lots of side effects you might not have convenient statistics for.
I have a depressing sensation that at least half of corporate data science consists of abusing Goodhart's Law - finding a useful metric and then naively optimizing for it until it's no longer representative of business success.