I cannot disagree more. IMO AUC is a rare great metric: it's principled, useful, universally applicable (e.g. invariant to class imbalances), and easy to explain adequately to non-statistician ("probability of choosing a positive sample over a negative one").
>What you really want is two separate steps: estimate the probability that instance A belongs in classification X, and then a decision step where you decide how to classify A based on a loss function (this varies depending on how harmful a false positive is).
Yes, but those are two inherently separate steps and should be measured separately. AUC is a metric of the first step (the "model"). The second step is a business decision and will often be made separately from the modeling process, by different people, with a different cadence, and with a different goal in mind.
For example if I am designing a model to find a disease, I just want to make the best prediction I can, which is cleanly measured by AUC. Then, when it comes to actual diagnosis, someone else will choose cutoffs based on various factors like false positive costs (treatment cost, human toll), false negative cost (disease damage, death toll), supply of treatment, etc. I can picture scenarios where the same model is used for decades, but the cutoffs change seasonally.