Hacker News Reader

Comment by apatters | Hacker News Reader

There is a difference though. A system implemented by Equifax would presumably look at your social media habits or phone usage and say, "Sorry, we're denying you a loan because our data indicates that you may default."

The Chinese social credit system looks at your social media and phone usage and says, "Sorry, we're denying you a loan because you were critical of the government."

It is fair to criticize the prospective Equifax system and the concerns around pervasive data collection are valid. But the Chinese system is in a whole different, far more evil league.

> While English-language media describe China’s social credit system as a Black Mirror-like authoritarian implementation, Chinese social media users seem to focus more on the advantages than the burdens.

This is a remarkably tone-deaf observation given that talking about the burdens will get them imposed on you.

View on HN

Since equifax is a US company, the data is only one robed person away from being used as, for example, "you have been deemed a 'national security' risk and so now you cannot enter/leave the country and now require additional attention from some certain 3 letter acronym gov. organization"

Such a system, once in place, is really easy to change to filter for any kind of criteria. One must prevent the infrastructure for large scale surveillance. It's not okay just because you agree with its planned mode of use.

As per the case of the 1930's Dutch census data used by Nazi's to murder Jews. I doubt anyone in the Dutch government imagined such a thing for a moment, and yet - catastrophe and horror.

We must be very careful.

You may not even need to change it.

Fed with enough data, a ML system should be able to derive “We’re denying you a loan because you’re likely to default on it because you’re likely to lose your job because you’re critical of the government”.

It likely wouldn’t be able to express that to humans that way, though, so humans would say “We’re denying you a loan because the computer says we should”

How do you imagine such an Equifax system would regard criticism of said Equifax system? Seems like the sort of thing that makes you a default risk, doesn't it.

Maybe? The point of the system is to profit/ensure no default/etc. So if the system determined criticism actually did make you more likely to default, sure. If not, no.

That may be the stated goal of the system. Any system, however, has an underlying goal of sustaining itself. If that's by accomplishing its stated goal, then great. If it's by suppressing criticism of itself, then that's just as good.

> Equifax would presumably look at your social media habits or phone usage and

... and exercise and driving telemetry data and family medical and criminal history and...

At a certain threshold the distinction with the Chinese system becomes irrelevant given the impact on an individual.

> The Chinese social credit system looks at your social media and phone usage and says, "Sorry, we're denying you a loan because you were critical of the government."

That's a common misconception of how Chinese social credit works. China doesn't really care if you're criticizing the government as long as if they don't think of you as a threat. If they do, they'll simply imprison you. If the government is only a little suspicious, they wouldn't want to send you on guard by lowering your credit score.

Other comments about Equifax say "Let's wait until sinister intents kick in, I'm telling you"

This one says "Nope, the Chinese system wouldn't use the credit system for non benign reason. They don't care"

Different standard?

> It is fair to criticize the prospective Equifax system and the concerns around pervasive data collection are valid. But the Chinese system is in a whole different, far more evil league.

I think calling it outright 'evil' is unfair.

The Chinese model at least comes from a well-intentioned place. It's a ham-fisted version of the "nudge" programs you see in places like insurance plans, with a bolder underlying goal to create a better society. Their definition of "better" may not be yours, but the goal is not inherently malicious.

The American take of this will have the exact same flaws-- the data will eventually be mishandled, the models will have holes you can drive a bus through-- but it's justified in the sake of "we'll get slightly more accurate pricing for credit and insurance".

Personally, I find the push to allocating resources based on this kind of data interesting and could possibly be more efficient than the way it has been.

That being said, since these systems are abstraction on debt, the failure mode at the margins means that companies that become reliant on funds from these loans will be increasingly sensitive to people at the margins who will be denied such loans.

I also find it interesting that the public will also have increasingly more access to the data to be able to retroactively understand the factors that determine the go/no go (especially as these systems become increasingly automated). Combined with the above, it will be increasingly possible to create ways to dirty the waters in the signals for a variety of desired outcomes for 3rd party actors.

To understand the Chinese system you need to understand where legitimacy comes from in China. The Chinese government employs 50m people, it is a participatory state - everyone will know someone employed by the government. Personal obligations mesh people together to the state, migration and demographic change have disrupted connections and threaten this mechanism of legitimacy - the social credit system could be seen as a reaction and reinforcement of this. I think it is important that people try and see what is going on as it is rather than as a caricature because otherwise the lessons will be lost in the noise, in this case I think that the Equifax / capitalist system is remarkably similar to the Chinese system. Not only are people constrained by the predictions of loan repayment (and by the way that's a can of worms in terms of how & what) but also people are tracked and shamed by their opinions and habits online; and these are wilfully misquoted and misinterpreted to generate social constraints (the recent story about the estate agent who was falsely accused as a bigot and an adulterer : http://www.gizmodo.co.uk/2018/07/when-a-stranger-decides-to-...)

My concern is that the truly evil have stealth and pretence on their side, while people who are developing the behaviours and norms that will be necessary for us to cope with the societal challenges we face - for example people who espouse radical anti-consumerism - may be driven out of debate by the norms and norm maintenance mechanisms that are accidentally and with good intention being developed in the west.

Equifax: "sorry, we're denying you a loan because an algorithm no one understands said to"

They will feed your behavior into a machine learning algorithm, which will probably develop biases.

I always wondered: what if redlining worked from a financial perspective?

If it turned out that Ethnic Group A is on average 5% more reliable about paying their debt, this is valuable information, but the banks are legally bound to ignore it.

So they've probably spent a lot of time trying to produce similar metrics through proxy measurements. Seek out a profession or purchasing pattern that suggests a given ethnicity without outright saying it.

I suspect AI is, from a legal CYA perspective, the greatest proxy of them all. "It's a black box, we don't/can't know how it produces its results... yet mysteriously it biases exactly how we would if we could overtly discriminate."

What if politically active people are determined to be at high risk of default?

Actually China restricts people from boarding planes and trains, if they have a low score.

US can do the same thing... using the patriot act.

Reply on news.ycombinator.com