I want to be clear that I would phrase that as “based on consensus,” and not as bullshit. I have seen so many doomed/failed projects/models that have come up because people say “well, it’s math, so it has to be right.”
Sorry if I'm missing your point?
A ethicist will never be caught using real inputs at all, precise or not.
A Psychologist will tell you 86% of people taking treatment X recovered. But miss out that 85.8% of people on placebo also recovered. Those numbers will be from a study of 10 people.
Mathiness is a term coined by Nobel prize winner economist Paul Romer to label a specific misuse of mathematics in economic analyses. An author committed to the norms of science should use mathematical reasoning to clarify their analyses. By contrast, "mathiness" is not intended to clarify, but instead to mislead. According to Romer, some researchers use unrealistic assumptions and strained interpretations of their results in order to push an ideological agenda, and use a smokescreen of fancy mathematics to disguise their intentions.
Your model can be completely internally consistent and allow you to study things in the context of your assumptions perfectly, but that doesn’t mean it doesn’t have critical flaws that disqualify any learning about the system you’ve modeled.
The lack of understanding about what implications a representational choice have for the kinds of relationships a mathematical model can describe, and the lack of understanding concerning what are and aren’t critical domain components to model that’s difficult.
People often forget that there are two translation steps that have to take place. Domain -> Math -> Domain. That second one is too often implicit or assumed to follow, especially in ML.