Hacker News Reader

Comment by jandrewrogers | Hacker News Reader

There is a lot to be skeptical about in this story.

I seriously doubt they have 3B unique devices, and are instead counting the same devices several times under different ids from different sources. They aren't remotely tracking 3B devices in "real-time" sourcing data in this way. And even if they were, I guarantee they don't have the technical infrastructure to land that data. The "geofencing NSA and CIA" shtick was old news a decade ago, much cleverer things were already being done.

This is marketing targeted at people that know nothing about this kind of business. There are literally dozens of mediocre companies doing the same thing with the same kinds of press releases.

View on HN

Yeah it's all passive location datasets. Financial firms have been buying up defunct apps with location data for years to infer trends. Intelligence firms have been doing the same. Then they package and resell between each other to get more data sets. Nothing new here, but a good shock story for The Intercept.

It sounds like you know this kind of business? What should the takeaway be for the rest of us that are not in the know?

People severely underestimate the velocity and volume of data implied by this if you actually did it, never mind having to run analytics processes at the same scale alongside it. We are talking about bespoke state-of-the-art data infrastructure platforms. You can't support anything like this with open source software, not that it stops people from trying. Glorified data brokers are usually not bastions of world-class software engineering, and if they were, they wouldn't be in such a low margin business. Most of these companies are just recycling the same low-quality and stale data sets.

The question I always ask, when evaluating companies making these types of claims, is "what hard computer science problems did you solve to make this possible". If you've actually done what is claimed like in the above, it will be an interesting list. In practice, this question usually elicits confusion.

There are legions of dubious companies making claims like this, which you can safely ignore. Their data quality is so poor that they would have difficulty violating most peoples' privacy even if they wanted to. The couple orgs with the technical expertise to actually pull it off competently don't talk about it.

Actually, off the shelf open source and commercial data processing and analysis software can be made to do this stuff, if you're willing to spend $$ on AWS or similar infrastructure. This kind of analysis is easily partitionable, and while you're right that it would be hard to do this for all 3bn phones all the time, it's relatively straightforward to identify hundreds or thousands of candidate devices and then do the needed analysis across a huge data set.

(Built and sold a Data Broker, it's not a low margin biz btw, we had >85% gross margins because of how cheap the source raw data is, and we were doing double-digit millions $$ in revenue).

There is open source software that can do online indexing across space/time/entity and cross-domain entity resolution in real-time on petabytes of telemetry data per day? The data never stops flowing and it isn't trivially partitionable in an analytically useful way -- graph-like joins figure prominently. It isn't a problem you can solve simply by throwing hardware at it.

I've been paid to characterize the quality of this type of data from well-known brokers by discerning customers, and in my experience it is all varying degrees of rubbish from which almost no generalizable insights are possible. Even with data that is much higher quality than what you can get from app SDKs and adtech, ground-truthing experiments show that it requires very sophisticated analytics to build something resembling a generalizable model from which insights can be reliably extracted.

I know there is a lot of money in selling data like this but it is essentially a scam, promising insights that aren't really derivable given the data provided. The data quality has also become much worse over time, for a variety of reasons, which is arguably a bigger limitation these days than lack of a suitable platform.

There are that many phones though. That information exists due to structure of mobile internet infrastructure.

Reply on news.ycombinator.com