My take: I’ve published in well-regarded mathematical journals and the culture is definitely hard to explain to people outside of math. For example, it took more than two years to get my key graduate paper published in Foundations of Computational Mathematics, a highly regarded journal. The paper currently has over 100 citations, which (last I checked) is a couple times higher than the average citation count for the journal. In short, it’s a great, impactful work for a graduate student. But in a field like cell biology, this would be considered a pretty weak showing.
Given the long timelines and low citation counts, it’s not surprising that it’s so easy to manipulate the numbers. It is kinda ironic that mathematicians have this issue with numbers though.