The discussion also is at high risk of becoming politicized, where more credence will be given to salient first-person accounts that happen to fit well into some prevalent social narrative of the times.
If you can prove that a lot of care was taken to avoid these types of bias in extracting summaries from a collection of qualitative data, then it can be valuable. But often this means you have to be so cautious and err on the side of rejecting all but the most consistent summaries from the data that you run afoul of critics who want the data to be used to support their preferred political narrative.
This risk of subverting qualitative data like this into a disingenuous claim that something is data-driven is really high, in addition to all the usual cognitive biases that lead people to read what they want into narrative accounts.
We have engaged on related ideas before[1], and I think your position represents something that I find quite common and deeply unsettling about modern tech culture, but which I don't yet feel I can fully characterize.
There is something about the need for everything to be "data-driven" and metrics-focused that I think represents an abrogation of moral responsibility to form, apply, and maintain our own value systems. It is easier and we would apparently rather assign this messy work to some external process or algorithm or metric, which can then be argued to be perfectly rational, correct, data-driven, unbiased[2].
To me this is a dangerous fantasy, closely related to scientism or the general failure to accept any reality outside of what can be measured, counted, or quantified. This is a characteristic failure (in my opinion) of people in their twenties and of STEM majors in our times, so it's not surprising that it would be a characteristic failure of tech, where both those categories are over-represented.
It also seems correlated with a general ignorance of and arrogance towards the humanities among tech workers.
[1]: https://news.ycombinator.com/item?id=21097061
[2]: A related symptom of this disease is in hiring, where companies come up with systems that look very logical and bias free on paper, for example having "objective" scoring criteria and technical challenges (which end up being almost random number generators, but with some bias that tends to reinforce existing imbalances in the industry, e.g. towards recent graduates), and equally "objective" ways of combining results from hiring committee members or interviewers that tend to create and enforce conformity and exclude divergent viewpoints, while giving the appearance of being bias-free, data-driven, and rational. In fact it's a great way that entire companies fool themselves and come up with terrible processes, and then they whine about how hard it is to maintain diversity (of gender, age, whatever) in their ranks. The people that design these systems aren't doing this as some kind of great conspiracy; as far as I can see, they're not evil, just blind.
I think it's a big (and almost certainly invalid) assumption that in the absence of metrics, people "apply and maintain" a consistent value system. It seems to me inconsistent with a belief in the destructive effect of the wrong metrics.
If you think that metrics lead to bad results, it's completely inconsistent to suggest that not having metrics is better. When you have metrics, the harm comes from people ignoring what really should be optimized in favor of what is being measured. If you don't have any metrics, then you're only decreasing the pressure to do whatever may be the right thing.
It's like, no matter how bad the government is, the only solution is better government, not eliminating it.
So, I think your general attitude is ironically capturing the essence of a common intellectual failing of the people you are stereotyping. You know, the glib libertarian tech worker.
What on earth makes you think I'm making this assumption? I never said anything of the sort.
I said that people are abrogating their responsibility to do difficult work (figuring out if the ship is going in the right direction or not) by trying to offload this onto metrics, which are theoretically objective and unbiased.
Nowhere did I ever say anything about what people not over-using metrics are doing.
> If you don't have any metrics, then you're only decreasing the pressure to do whatever may be the right thing.
That is not necessarily true at all. Having no metrics at all, just like having no government, means that you must fill the void with something. You must decide for yourself how to act, probably based on a subjective judgement of what the situation is combined with your personal values.
How can one "fill the void" without measuring anything? Personal values may vary, but I can't see the concept of determining the consequences of your actions as being an optional one of them. It seems to me that if you are definitely rejecting that, then you have no personal values. Even if you say you do.
When we talk about metrics, especially in a business context, we are talking about numerical quantities that can be tracked and unambiguously measured. A metric that can't be measured, or that nobody agrees on the measurement of, is no good as a metric, in this context. Are we agreed on that part of the definition of a metric?
> determining the consequences of your actions
That is an entirely different thing.
> measuring anything
And that is yet another entirely separate concept. You can weigh a course of action without measuring anything.
> It seems to me that if you are definitely rejecting that, then you have no personal values.
In most situations in life, you must make a decision without any chance to know the results of your actions. Can you know in advance the result of picking a career, or taking a job, getting married, moving to another country, or voting for a particular candidate? Of course not. The number of decisions in life that are repeated, and where a numerical, measurable value is available to guide your decision making over time (which is what is meant by tracking metrics in a business context) are a vanishingly small minority of the decisions we must make.
Unless you calculate a utility function before crossing the street, or having breakfast, I don't think you really mean to argue for what you appear to be arguing for.
https://www.amazon.com/How-Measure-Anything-Intangibles-Busi...
From the reviews, the book seems to have some light statistics and introduction to Monte Carlo methods. I don't think it would have much to teach me about statistics or philosophy. It might be useful for someone without a background in statistics who is forced to navigate a business environment where people value metrics above all else, but I'm arguing against such environments.
You wrote "In most situations in life, you must make a decision without any chance to know the results of your actions". So it seems to me you were clearly expressing an idea of what a metric is, in rejecting the need for metrics, that you now attribute to me and say it's the wrong definition.
Unless I'm missing a subtlety like perhaps consequences are not results?
I agree that you probably wouldn't learn anything from that book.
No. You said a bunch of things, that to me are almost entirely unrelated, and I responded to several of them.
One of them was about determining the consequences of actions and I responded to that because it was part of your argument about having personal values.
Before any of that, you wrote: "How can one "fill the void" without measuring anything? Personal values may vary, but I can't see the concept of determining the consequences of your actions as being an optional one of them."
The implication is that if you reject metrics you still have to measure and determine the consequences of your actions, so metrics are unavoidable. If metrics are not the same thing as "determining the consequences of your actions" to you, then you answered your own question of how to "fill the void": you determine the consequences of your actions without metrics.
The question of whether metrics are a good way to determine the consequences of actions is separate from the question of consequentialism, or whether actions should be chosen by looking at consequences alone. You seem to be assuming both and I'm challenging both.
You weren't clearly expressing an idea of what a metric is, gotcha. Very helpful.