I think it is because people are really bad at logic. Seriously. Most of the arguments on here are coming from a very bad set of assumptions. The major assumption here is, like the OP said, some Sci-Fi sensor data. Even as I tried to explain this people are still thinking way too exact data ("Shih Tzu" vs "small dog"). People are correct to counter these claims, but when they do, they usually go overboard. They say the data is useless or "they already have it, so it is effectively useless." Verification is a powerful tool.
But I think a large amount really comes down to the fact that people don't think from the statistics or mathematical viewpoint (despite claims). I've read probably more on stats and prob than the vast majority of people here yet I would never remotely claim I'm good at stats. It is fucking hard, yet so many think it is so easy (be wary of anyone that suggests it is easy). So here's some things I notice:
- People think we need crazy specific details to find trends, but you can extract signal from noise with a lot of data (especially diversified data) and very careful insights. The counter to this isn't to just dismiss but to ask if simpler data can be useful[0]. We feel the need to take a drastically opposing argument rather than a nuanced argument.
- Thinking that there are singular causal factors (see the arguments "it is about robots, not data" when it is clearly both). We talk about PCA all the time but a lot of our arguments create frameworks where we're only discussing what is the dominant factor and pretending that others don't matter[1].
- Being nowhere near familiar with emergence and how it plays a role in data and our lives (I suspect lack of emergence knowledge is why so many believe in "deep state" or conspiracies).
- Not framing things from a probabilistic viewpoint. I see this in security arguments (usually among non-experts but tech literate: see the Signal community forums or here). Most security is probabilistic in nature and about putting bounds, not guarantees. The classic example is remotely wiping a phone. If you wipe, there's a probability that an adversary hasn't gotten the data first. If it isn't wiped, said adversary has all the time in the world. Everything, and I mean literally everything, is a probability. What we call truth just has tight bounds.
- Zero sum fallacy. Many people think the vast majority of games are zero sum, when very few really are. We see this a lot in economics (value can in fact be added to the system. It is only zero sum in a infinitesimal point in time). A tide raises or sinks ships equally, it does not raise some and sink others to the point where we have a balanced system.
- Over simplification and thinking higher order approximations aren't necessary for "good enough."[1] People think that most probability distributions in the wild are Normal(ish) when they aren't (most have heavy tails). This is all caused by creating a "spherical cow in a vacuum" framework. An oversimplification of a problem isn't necessarily a good approximation and can often lead you in the wrong direction! Specifically the major challenges we face today, we need higher order terms to even get a reasonable approximation. For those math inclined, think of the Taylor series approximation of e^x (1 + x + x^2/2! +...+ x^n/n!). y = 1 isn't a good approximation except in a very narrow region (around x=0). y = x + 1 is even a worse approximation (depending on your region of interest)! Even a 4th order approximation is only useful on the bounds [-1, 1] (8th order will get us to [-2,2], maybe [-3,3]), but diverges quickly beyond that. If we're concerned about x<0 then the first order approximation is better than the first 5 (y=1 is closer to e^-100 than the 5th order x=-100). But if we're concerned with x>0 then the higher orders are even better.
The real Trojan horse is how we've structured a belief that these things don't matter. That we think simple answers are good ("you don't know something unless you can explain it to a child" is bullshit). I suspect that this is evolutionary as this framework has allowed us to solve most of the issues related to survival. The problem is that our modern society is much more complicated than that and we have effectively solved these survival hardships. The problems we face are now so complicated that our simple frameworks are no match for solving them. The above are things that most people struggle with yet a single one can quickly ruin our frameworks. I'd argue that we see all of these points showing up in the arguments of this post (this rant is still rooted in the topic, just meta. I'm writing this so we can have better arguments).
[0] To help, let me give a very clear example of something almost trivial to determine but highly useful. Suppose your roomba constantly bumps into things near a door and those things move every single day. It is very likely that those are shoes. We now know to advertise a shoe rack so the person can organize their shoes at their door. Yes, there are more complicated examples where we can get more intimate details of ones life and sell more products, but the simplicity here is that there is noise that we can extract signal out of and in a way that purchasing behavior or online behavior wouldn't capture.