Hacker News Reader

Comment by jandrewrogers | Hacker News Reader

Almost all global sensing data published by the US receives a lot of negative pushback from other countries around the world. I don’t think most people are aware of this.

The reasons are myriad. It makes it harder for other governments to control narratives in their own countries. It undermines efforts of governments to develop their own capacity; the US gives it away for free but those countries are not the customer and it does not serve them per se. It sets a much higher bar for domestic implementation than they have the capacity to implement. There is a sense the US exploits this data for their own ends. It doesn’t just irritate China, it irritates everyone.

Making the issue more political, the US edits and censors the data it publishes for its own purposes. This isn’t a secret but it taints the perception of US neutrality when making this data available.

The geopolitics and realpolitik of international sensing data is not clean.

View on HN

So pointing out that the sky is blue when it is indeed blue would be a form of propaganda as well?

> It undermines efforts of governments to develop their own capacity

If the government is so intent of concealing the problem that it is falsifying basic measurements then surely those efforts are not worth much?

We’re not talking about information that is in any way subjective, relative or biased.

it’s like a government trying to pretend that climate changes isn’t happening by getting pissed at other countries that have accurate thermometers in their embassies..

What is political in sharing scientific measurements? Can you point to a specific person who is irritated by it? Why it irritates you personally?

The comment you're responding to literally spells out the reasons.

Literally literally, not figuratively literally.

If you collect novel domestic sensing data and share it with the world, that is exploitable as intelligence against your country. If another country gives you their domestic sensing data, it may be manipulated to effect some other national objective. These are not idle concerns because both have been done many times by many countries. It is a dual use technology. There is a lot of paranoia around the sharing of sensing data sets between countries because much of it is easy to abuse.

It is a “default deny” environment. The majority of international data sharing deals I’ve seen, and I’ve seen quite a few, never happen because they can’t get past what governments see as the political risks. The scientific mission barely even enters the discussion when it comes to getting sign-off from governments. When they do give their approval, it often comes with conditions that impede the scientific mission. I’ve also seen data sharing denied for unrelated petty geopolitical reasons between governments quite a lot.

It is less obvious in the US because so much sensing data resides there; there is still a lot of scientific sensing data in other countries that would be useful in the US if you could get it. Outside the US people feel it more acutely because of the relative paucity of available domestic data.

I’m not making an endorsement of any type, I’m just saying this is how it works internationally in my experience. You may have simple scientific missions but the approvals come from people that have a different agenda. It is a pain in the ass and really demotivating if you need that data for your research.

China lost their minds over it. It’s very sensitive politically because it undermines the official narrative.

I mean, Trump said on covid, 'if we stop testing, we'd have fewer cases'. People who deal in a steady stream of lies don't want the inconvenience of facts like scientific measurements. Same reason all the climate data is being removed.

> Making the issue more political, the US edits and censors the data it publishes for its own purposes. This isn’t a secret but it taints the perception of US neutrality when making this data available.

First I have heard of this, what's the source for the US editing & censoring global sensing data?

They remove things not germane to the purpose of the data they publish. For example, USGS seismic data is noticeably bereft of most seismic events that are not geological in nature or sometimes related to mining (though in some sources they often scrub the mining ones too). Events of military interest like weapons tests, some target getting blown up in the middle of nowhere that may never make the news, etc is removed.

There are a ton of artifacts that show up in other sensing systems that are indicative of interesting or sensitive things that are outside the scope of their purpose, and these too may be edited from the data.

The people deciding what constitutes an event that should be scrubbed is pretty opaque AFAIK. It is official policy and sensing companies that do a lot of work with the government seem to follow similar guidelines.

Due to the proliferation of crowd sourced and alternative sensing platforms, I would argue that this is increasingly an exercise in futility. Nonetheless people still view many of the US sensing data sources as authoritative for all practical purposes. There are countries with laws dictating that some alternative data they control must be treated as authoritative for all purposes for their country, but that US data is sitting out there.

That's a lot of words but still no source.

Well, you could go to the website where they clearly state that some data is reviewed before publication and may be removed or modified. It is a frequently asked question. Or you could find an obvious counter-examples in the data, since it is public. The detection and flagging of anomalous events for review has been automated for decades, also publicly mentioned. I don’t assume everyone knows, I’ve been working with government sensing data for 20 years, but they are quite explicit if you look.

What they remove is a secret AFAICT but if you are an expert in the sensing modality it becomes obvious what should be in the data but isn’t. There are now businesses that specialize in differentially finding or reconstructing things that have been removed or modified in sensing feeds, so the effectiveness has diminished greatly.

> That's a lot of words but still no source.

Less flippantly, I literally went to the USGS FAQ to find this question https://www.usgs.gov/faqs/does-latest-earthquakes-map-show-n... where it says mining events are not reviewed, but if they are identified, they are still included! So ... I'll ask more explicitly:

Can you provide an explicit LINK to anything that supports your claim? Thanks.

Like others nearby, I’d like to see something more specific.

I’m willing to believe that in some silos (like relatively high-cadence seismographs) there might be some censoring. For example, it’s believable that siting of permanent stations is nudged away from some sensitive areas. Also, more believable in the past (say, 1980s) than the present.

Related, I’m sure that some sensors aren’t allowed to be flown over some areas (e.g., certain military bases) in the US.

However, you are claiming a broad based program of censoring US scientific data - gathered by the government or by government contractors. Like you, I’ve worked in this space for a long time. But I have not seen what you describe.

I wonder if we are working under different definitions of “censor” (see military base remark above)?

For people’s reference, the US-sponsored seismograph network is under EarthScope (https://www.earthscope.org/gsn/).

Your remarks caught my notice because I have personally worked with GNSS (lower cadence than seismograph) data, and personally know people who placed the sensors, wrote the data assimilation algorithms it uses, and set up the data pipeline. This data is not censored. (Although, famously, it was, before GPS was opened up.) I’m trying to find a way to rectify these two viewpoints.

That raised my eyebrow also.

In the past I've heard similar statements and had people point at 'cooked' raw data as evidence of editing.

'Cooked" generally means raw data with warts removed, the raw data is still available, the cooked data is what's on offer as the primary data of record - typically it may have had sensor errors and saturated bursts removed, undergone light smoothing filtering, and perhaps been geolocated to earth coords rather than retaining raw instrument attitudes, etc.

'Censoring' can mean 'no longer linked on public webpages for easy downloading' - generally the raw and cooked data is still on servers and accessable by direct FTP.

I'd be interested to know what specifically the GP actually meant by that throwaway assertion.

Less doublespeak version: The US edits and censors the data it publishes for its own purposes. Thus the data is not viewed as neutral.

The neutrality of some entity is not set by default by God and then “tainted” by their very own actions. The neutrality or lack thereof can be reasoned about directly.

The irony is that the US isn’t actively encouraging other countries to use this data, quite the opposite. But if it is public to anyone it is public to everyone. Countries became accustomed to freeloading on this data because they didn’t have their own, even though it wasn’t for them or designed to be fit for their purposes. The intended purpose of the data may not be the same purpose that other countries are trying to use it for.

The censoring and editing often isn’t nefarious, it is a useful data cleaning exercise that reflects why the data was collected. If they simply published the raw feed then they would be doing a disservice to their actual customers.

> The censoring and editing often isn’t nefarious, it is a useful data cleaning exercise that reflects why the data was collected. If they simply published the raw feed then they would be doing a disservice to their actual customers.

The way you worded it made it sound self-serving and underhanded to my interpretation. :p

I mean data cleaning is all neutral. I’m not gonna complain about that.

Current USA leadership is not concerned with negative pushback from abroad.

Not concerned about negative feedback from democratic countries that take care of their citizens.

However USA leadership seem very concerned about keeping happy face when talking to dictators. That tweet by Elon saying "this is what competent leadership looks like - of picture of Sergei "novichock" Lavrov and Mohammed "bonesaw" bin Salman.

Or how T could not say Putin is a dictator. Or anything bad about him at all. Never mind under Putin that country has no freedom of speech of or freedom of economy.

I wonder what data on the US by others is there

Some but a lot less than you might think. The US invests a lot in global sensing, in some cases operating redundant independent networks while many countries don’t even have domestic sensing. This is a consistent complaint of non-US scientists that work with sensing data; it is difficult to get their governments to invest in domestic sensor networks when the US kind of does it for you, so even many developed countries have limited domestic data. In Europe in particular, there are additional privacy concerns that stymie more creative approaches at aggregating these types of measurements by other means.

When one of these countries does build out a domestic sensor network, they are often unwilling to share the data because it is seen as advantaging the Americans. Resistance to data sharing is quite high, for many reasons, so if you need to do anything that spans many countries you often end up falling back on whatever the Americans can give you.

The geopolitics around data sharing has been a significant hindrance to scientific activities like trying to build accurate and detailed climate and environmental models. Natural processes don’t recognize national borders.

The only global sensor networks operated by several countries independently are related to weather, and even then there are fewer than people probably imagine.

Reply on news.ycombinator.com