China's brightest children are being recruited to weaponize AI
scmp.com
scmp.com
These threads have been getting noticeably worse lately and the worst sections of this one are vile.
I'm not saying that everyone should have nukes, I'm saying the EU, US, China and Russia should be making those treaties right now instead of later
With nukes we can have inspectors and we can lock down uranium. Killbots are too diffuse and too multi-use. I'm all for arms control / non-proliferation, but I really don't see how it works for these things. Also, we already have weaponized AI. We have for decades. We're now talking about a matter of degree.
I'm far, far more worried about the mass weaponization of civilian systems (self-driving car, etc) via cyber attack than I am about killbots, but they're both symptoms of a different problem:
As technological growth continues the space of potential combinations of methods accelerates. This ever expanding search space results in unpredictable threats and increasingly asymmetric attack economies.
I've been meaning to write an article on it, but I'm having trouble pulling together the math because it's so abstract. But that's the general idea.
1. Hard to quantify incentive structures.
2. Chaotic / cascading decisions and adaptations.
3. Growing search space that's influenced by information restriction.
4. Uncertainty of future arrangements and options.
5. Uncertainty of deterrence efficacy.
6. Even some aspects of agency problems.
I've gone through some of the papers out there but, frankly, nothing I'd consider a serious mathematical investigation. I don't need the input data so I thought this would be easy. I only need the math or computer code.
The possibility space is limited by sanity checks of the actors - technically a terror group could start putting live rattlesnakes into snake in a can prank devices on people's doorsteps but it would be logistically difficult and incredibly stupid.
Beyond that one interesting thing about attackers are how memetic they are over what is appealing to their personal image as opposed to effectiveness. Like the symbolism matters more.
Terrorists in the US could cause lasting damage and disruption by infrastructure attacks like going around and emptying a few AR-15 clips into unguarded transformers and substations or train track sabotage around oil freight trains yet they don't at all. Similarly looking at the historic trends between bombings vs mass shooting vs vehicular murder plots seems to be more cultural than availability related.
I agree that symbolism seemingly matters more. I was actually once on a podcast years ago arguing that terrorism was essentially a solved or non-existent problem because armed with nothing more than a rented truck terrorist could plow through crowds. And in the US it was even easier! They could buy weapons at gun shows! Why did they need to hijack planes?
Ultimately, though, my arguments didn't hold. The terrorists did exactly what I thought they would do if they really wanted to cause damage.
Just because X could Y (but doesn't) doesn't mean X won't ever Y.
I'm not worried about ISIL style attacks where a couple hundred people die. Frankly cutting out one can of cola a week would probably save more lives in the West.
I'm worried about mass attack. We can skate around it all we like, but we're one bad Windows / Dropbox / Tesla / Linux / Cisco / QNX update from hundreds of thousands or millions of people dying. The Windows worm that preceded the electrical grid blackout in the mid 2000s was the inadvertent cause. If an accidental worm can take out that much we should be much more worried about the scale of the threat.
We've also never been able to secure the boarder. With drones this gets even harder because capture is no longer a real deterrent.
Security is an arms race that favours the offense over time. A theatre that strongly favours the offense requires deterrence and intelligence for dominance, but the deranged can't be deterred and I don't think intelligence will work forever. Ultimately the arms race needs to end.
I don't know the solution, but my intuition points to GATACA and UBI. But even with an optimistic estimate I still consider that a stop-gap.
The issue with regulating AI kill bots is that much of the technology that could be used for good like self-driving cars can also be weaponized. There's really no way to progress without risk. The only other option would be to ban AI research.
Might be the Amish and Unabomber were right about technology
No need to speculate about killer AI bots when you've got plain old famine, drought and resource wars.
While the human body and psychological context is preprogrammed up to a point from our genetic makeup, we are capable at learning.
The same mind that was good enough to craft better stone tools sufficed to send people to other worlds.
The same societies that practiced slavery and genocide reached a point where these became non-accetable.
While as a species we are inherently capable of great horror we are also capable at great acts of compassion and love.
The fact that we are constantly teetering on the edge of animal chaos makes it only more important to try to be a decent human being. And not to listen to your amygdala so much.
AI is nothing like that. It's just software. Once it is out there, it is going to be copied, and the resources involved are minuscule.
International arms control treaties haven't historically been worth the paper they are printed on, so there are probably a great many things that would be more worthwhile to direct energies towards.
I know this can be seen as whataboutism but I feel the Western world might have gotten desensitized to its own activities in the sector, while looking warily at possible geopolitical competitors.
Pointing out that the US does the very same doesn't necessarily discredit or diminish the problem with China's actions. If anything it shows we should be worried about these things on a global scale.
I think the main issue is that those actions aren't written about or criticised when done by the western powers. Hence why these article seems so hypocritical - they try to paint actions accepted and normal in western world as somehow evil when done by the eastern part. It doesn't mean they're not bad - but it does seem like the articles are just propaganda and not an objective look.
So, the default assumption is that killbots controlled by your country are safe to you, but killbots made by a foreign power are dangerous.
After all, it was created as a US propaganda term for saying that it is hypocritical for the USSR to propagandise the hypocrisy of the USA, when both governments are hypocrites.
Outside of that, it is called making comparisons and it is a completely valid thing to be doing.
Making comparisons is valid, but it doesn't make you incapable of responding to the original arguments.
In any country on earth, you be pretty much 'vetted for loyalty' before they let you close to their sensitive intelligence or tech related work.
There are many other things you learn and things you can go on to do as [engineering student] at [university]. Doing a [resume-boosting challenge] is one small part of your universe.
On the other hand, when you are entering a special program for weapons development and giving press releases about how long you've been studying about guns... your intentions and the expectations for you are very different.
The existence of these types of programs isn't troubling. It's the belligerent geopolitics of the state that are troubling.
The US is, however, involved in overthrowing scores of other country’s regimes.
Should we get human-level AI, there is a good chance it will be made in China. As I don't think the Chinese are a nation of heartless automatons, I think spreading alignment ideas in China is not at all a hopeless task.
that is just a cheap excuse repeatedly used in almost all high tech areas -
1. because the Chinese government is bad, China is not allowed to join projects like the international space station.
2. because the Chinese government is bad, allowing China to have access to any semiconductor techs from the west is not acceptable.
3. because the Chinese government is bad, exporting nuclear power station tech to China is now banned.
4. because the Chinese government is bad, Huawei's phones are banned, ZTE was not allowed to buy any US made parts.
5. because the Chinese government is bad, Alipay's acquisition of MoneyGram was banned.
6. because the Chinese government is bad, Chinese universities working on China's weapon programs are banned from buying DELL desktop computers.
when US banned exporting any high tech product to China, what to do when there is a trade imbalance? get Trump into the whitehouse, cry hard for the unfairness in trade!
give me a break, please come back with some better excuses.
I'm pretty comfortable with the idea that free societies should not economically support oppressive ones.
We imprison more of our own people than them and we wage war all over the world, but people don't see it in their daily life so it's just background noise.
Maybe the basis should be cooperation and understanding rather than demonization and fear.
That is a great idea. Except it's not really what's at play here. If it were, then the US would ban all imports from China, and not just ones where it feels it needs to maintain a strategic advantage. It would also stop supporting Saudis.
I believe what GP is trying to communicate is that US struggle against Chinese government for maintaining top superpower status, thinly veiled as concern for human rights, is unnecessarily hurting Chinese people.
I believe it's way too harmful to simply categorize government into "free" vs "bad/corrupt/authoritarian/oppressive". There're a large number of other factors to consider. I agree Chinese government has tons of problems, just like US government has tons of problems of its own, just different.
At the bottom line, Chinese government is making tons of efforts to improve the economy of the country and improve the life of all the citizens. "According to the World Bank, more than 500 million people were lifted out of extreme poverty as China's poverty rate fell from 88 percent in 1981 to 6.5 percent in 2012, as measured by the percentage of people living on the equivalent of US$1.90 or less per day in 2011 purchasing price parity terms."
https://en.wikipedia.org/wiki/Poverty_in_China
> Do you not believe the Chinese government to be corrupt, authoritarian, and/or oppressive?
I agree there're corruption in Chinese government. Anti-corruption is a one of the main focus of Chinese government in the past few years. Moreover, corruption has nothing to do with 'freedom' vs 'non-freedom'. There are way too many heavily corrupted democratic government in the world.
Is it authoritarian? Yes there is only one party in power - the CPC. The party has close to 100 million members throughout the country, which is close to 1/10 of the entire population. You can't possibly consider a party like that to be a single entity. There's also semi-democratic process: People elect representatives and they form National People's Congress - highest organ of state power in China.
Is it oppressive? Well, media censorship is a thing, and if you had to go on the street and protest the government in the same way you'd protest Trump, the government might invite you to "have a talk". But people still talk shit about the government all the time in private and on social media. The comments are likely to get deleted / banned. That sucks, but it is not something that could hugely impact the quality of life of regular citizens. You can still hear lots of dissident voices on the internet, and the government would often respond to that in positive ways.
https://www.nytimes.com/2018/11/06/world/asia/china-detentio...
https://www.forbes.com/sites/ewelinaochab/2018/10/16/organ-h...
https://www.nytimes.com/2017/07/13/world/asia/liu-xiaobo-dea...
https://sg.news.yahoo.com/beijing-encroachment-hong-kong-fre...
If you believe China lifting people out of poverty excuses a laundry-list of human rights abuses we'll have to agree to disagree.
I've also expressed my opinion about some other issues like Xinjiang in another thread if you're interested:
https://news.ycombinator.com/item?id=18439175
I'm not saying there aren't tons of problems with Chinese government. But from the point of view of a Chinese citizen, the human right issues are mostly excuses that western media use to attack Chinese government; a lots of those are false claims; and even for the parts that'd be truth, it impacts very few amount of people and it's generally not a big concern for vast majority of people living in China no matter Han or not (Not saying violating human right is fine but the issue has much lower priority for most people)
To avoid continuing a flamewar we can just agree to disagree. Yes, Uighurs, Tibetians, and Falun Gong practitioners are minorities. The fact that they are minorities does not affect how I feel about the way they are treated.
If the rest of the internet is going up in flames about this, that's more reason, not less, to take better care of HN.
I remember when I was very little, so at least 20 years ago, there were no toilets in the village I was born in. It has not remained that case since then. I would say your information is pretty stale.
Since when does progressivism mean "try to understand and respect other cultures, except the Chinese, fuck them"? When did that last part get added?
You'd think that after all the recorded historical examples of populations being duped into believing some other group of people is subhuman - countless cases since the Ancient Rome to the XX century, all of them discussed on history lessons in western schools - you'd think people would be less willing to accept another round of the same propaganda classifying a whole society as inferior. It's sad that we still can't avoid that trap.
I've been to China and met Chinese people. They're normal. People, like everyone else, with problems like everyone else, with dreams and fears and passions, like everyone else.
Thats a ridiculous argument: they are 1.5MM, im pretty sure you can find a few milions smart enough who fit the description of the parent comment.
Its not racist or whatnot.. every nation has its smart fanatics
> “We are looking for other qualities such as creative thinking, willingness to fight, a persistence when facing challenges,” he said. “A passion for developing new weapons is a must … and they must also be patriots.”
I think it's fair to say the students they are recruiting can't have any misgivings about the fact that their projects will have military applications.
He was involved in the Brtitish chemical weapons program. There are some interesting arguments in that little volume, which apply equally well to AI killerbots.
Sure, anything they patent gives a foreign government a starting point and the U.S. government can use secrecy orders to prevent patents from being public or even flat out locking them down via things like the Invention Secrecy Act but I think OpenAI, and similar initiatives, will still considerably benefit mankind when they actually start to develop 'AI' with practical applications and certainly if they develop AGI.
For example with OpenAI this work they're doing with learning dexterity with the robotic hands could drastically assist amputees in the very near future by learning quickly from input from the user to make considerably more functional artificial limbs.
Calling them brightest seems overly glossy.
The US did this quite a bit, especially during the height of the Cold War. In fact you go could so far as to say that America's elite schools operate (or did at least then) as intellectual feeders for the defense industry.
https://www.imdb.com/title/tt0089886/
(Ninja edited to add the top paragraph...)
To say they are fervent about righting the wrongs and perceived wrongs of the past is an understatement. It's pretty much a driving force of life with many people. It's a powerful dynamic. And it's one that's held close to the chest by Chinese students studying here in the US but in China it's freely talked about, written about, and taken for granted as the right thing.
On our (the west) side we also have racism and nationalism. But I think our more diverse (talking about points of view, not race here) and more activist anti-racism voices are a tempering force that China lacks.
Relating this to weaponization of AI, I think people like Stuart Russell are naive to think that watching Slaughterbots would give these students pause. On the contrary, it would excite them, because it's showing exactly the kind of thing they would be delighted to develop. Not that I know these students, but I'm aware how China drills bitterness into its people about their past humiliations, and uses this to develop racist nationalism. Pretty scary.
From my perspective, I didn't really see it that way. Like any country, the history curriculum focuses on China's history, focusing primarily on ancient history (pre-Qing dynasty) for 3 semesters with one semester of modern Chinese history (Qing to present). While there certainly was a nationalistic tendency in many topics I would regard the information as historically accurate (as compared to courses about east Asian history here in the US). And in reality, my many friends in China with whom I've maintained contact over the years do not have this sort of mentality and are against censorship and much of the nationalistic messaging.
Another interesting note that people here have no knowledge of is professors in China are often times anti-CCP. Many professors take this to the extreme and in papers will often just give a bad grade if it mentions any good policy by the CCP.
On a semi-related note, the comments many make here on HN about China make me extremely uncomfortable. Not because they touch on sensitive topics but because they smear an entire population of people as liars, thieves, and mindless followers. Replace the word "China" and "Chinese" with "Italy" and "Italian" in many comments and they could have been written back in the Ellis Island days.
Except that's not what I said or meant. Yet you felt the need to put it in quotes.
When you notice examples of this, we'd appreciate it if you (or anyone) would email us at hn@ycombinator.com so we can at least post moderating comments in the thread. We can't come close to seeing all the comments, and sometimes I notice wretched stuff days later that it pains me to have missed at the time.
> China has cultivated in its people a very troubling mental complex centered on national pride and driven by the Century of Humiliation.
I’d say this is correct to a small extent — the history lessons in particular, took that stance when it’s covering contents about Chinese history from late Qing dynasty to the end of WW2. But the education also renders China as a peaceful nation, and the rise of China would benefit the entire world, which majority of the Chinese population agrees.
> To say they are fervent about righting the wrongs and perceived wrongs of the past is an understatement
It would be a correct statement if “righting the wrong” simply means “making China a great country again” so that “centuries of humiliation would never happen again”. Yes majority of Chinese citizens do have national pride, not unlike people any other country I’ve seen. Except maybe some extreme nationalists, most Chinese citizens would love to see China becoming stronger peacefully and be helpful to under-developed parts of the world.
> But I think our more diverse (talking about points of view, not race here) and more activist anti-racism voices are a tempering force that China lacks.
There are very little discussion about racism in China compared to US, simply because it’s less relevant. - Most Chinese don’t ever interact with a different race. But diversity in oppinions in other topics are very common. I’d say it’s virtually impossible to keep a billion citizens on the same page.
Just as an example, at the core of China’s “nationalism education” (don’t know what should I call it) is the anti-Japanese war which was part of WW2. Like South Korea, plenty of Chinese hate Japan for not acknowledging the war crimes they’ve done and trying to erase those facts from their history books. There had been several movement to boycott Japanese products in the past two decades. At the same time, Japanese culture actually have a huge influence in China, especially among young people. Japan is also one of the most popular foreign destinations Chinese people would love to travel to.
All the dynasties founded in the region, no matter if it was mostly Han people or minority groups, are considered part of China's history now. Just that Han had much higher weight in history books.
In a modern context, the main theme from textbooks is that all the minority groups live peacefully together and united as one nation, no matter what the history is. This is mostly true in reality, except maybe a small number of people in the west might want separation. You can find people from all minority groups living in large cities in the east of China, where the economy is most advanced. They mostly appear not much different from Han people, even though they might still carry some traditions from their own culture. Sometimes Han people might get curious when they meet someone from a minority group, but mostly they won't treat them any differently.
Being a part minority group is generally a good thing in China -> Among other things, they get bonus points in college entrance exams, which is a very big deal for vast majority of Chinese.
And that is the mentality difference between the US and China. US (having no real enemies) citizens see development of these weapons as potentially being used against its own citizens. Whereas Chinese citizens sees these weapons being to protect themselves from external threats.
There are hundreds of thousands of Chinese students in the US whose view on nationalism differs from individual to individual. Not only is the characterization of a purported hidden motif not warranted, it is reminiscent of Yellow Peril.
Nationalism is certainly alive in China. But Chinese foreign policy is only assertive only in its immediate periphery. To think that they are harboring this deep seeded grudge against the West that would threaten our physical safety is ludicrous.
This is a good point which seems to hold for now, at least with regard to exertion of military force. It’s mostly defensive although there is disagreement about the boundaries of what is to be defended.
Longer term though there are concerning matters like the extension of predatory loans to African governments in exchange for mineral and other development rights. Ironically similar to some of the abuses China suffered in its past.
So from that perspective I don't understand why 'natch' is receiving so much protest here. Totalitarian regimes are as easy to predict as a linear function and the Chinese "government" is totalitarian. The goal is to maximize power by all means necessary and - to err on the side of caution - also all means available and obtainable.
Humanitarian concerns exclusively play a role in governments and societies where at least some level of democratic processes and transparency can be carried out.
The bottom line is - expect the worst.
I know what the Chinese government has been doing and I was born and raised in China. However, that doesn't make me the immoral, senseless and vengeful entity he is trying to portray in his comment.
I know it's easy to generalize a certain group of people with whatever you hear from the news or read on the Internet, especially if you've not had too much meaningful interactions with individuals from that group. So I don't blame him, as I was somewhat similar before I came to the US and get to know the US people.
Secondly, nobody is immoral, nobody is vengeful or senseless - but having said that - the enemies of the state of course do not deserve moral standards as they don't adhere by those standards themselves ... "just watch the news if you want to know how many children they killed with their latest terror attack!".
The truth is - everybody loves slaughter bots as long as the right people get slaughtered - and who is that? Well, for the majority it's those labelled bad in the information sources they consume ... and are Chinese news independent? No, they aren't.
The Americans and Israelis love their drones - which aren't philosophically different from slaughter bots - and as long as those remove terrorists or taliban nobody would question those.
If China were staying 'beneath' the west, like they're 'supposed to', we wouldn't have scare articles every week in the economist and the NYT, and we wouldn't have this comment at the top of the thread.
You're demonizing them as scary others and proving the point of hardliners there.
> China drills bitterness into its people about their past humiliations, and uses this to develop racist nationalism
I don't believe you meant to, but you crossed into nationalistic and racial flamewar with this comment. Imagine how it must feel to read something like this if you're on the receiving end. It's surprising how many of the replies below managed to remain thoughtful. When I read those comments and compare them to yours, it's clear who the model HN users are in this case.
I don't think you were being malicious, I think you underestimated the difference between musing in a personal conversation and broadcasting to potentially millions of people. For example, it isn't enough to protest that you didn't mean all Chinese, just the ones who <insert caveat here>. That is far too crude to undo the impact such claims make on the ones who are being categorized. If you can imagine yourself in that position, this is not hard to see; yet it is hard to imagine, because we're mostly so certain that none of it could ever be justified in our own case, so it feels like no big deal.
One interesting thing: since most nations are tiny, and they are so utterly dependent on playing ninjitsu against others because they have no hard power ... the though doesn't occur to them. It becomes part of the national identity.
Canada->US Scotland->England Nordics (i.e. Norwegian)->Germany/France
We're a little bit more meek (for better or worse) by default, not just in geopolitics, but culturally as well.
Peace is kept through a balance of power.
I hope you didn't use the word "other" to imply the US has a policy of "peaceful globalism". That would be wrong.