DARPA open sources resources to aid evaluation of adversarial AI defenses | Hacker News Reader

DARPA open sources resources to aid evaluation of adversarial AI defenses

darpa.mil

134 pointsinfodocket28 comments

“ART provides tools that enable developers and researchers to defend and evaluate their ML models and applications against a number of adversarial threats, such as evasion, poisoning, extraction, and inference.”

The first two attacks, evasion & poisoning highlight the incredible importance of having high quality data when training models. Evasion is false-negatives that are allowed because the model did not have a diverse enough selection of training data and poisoning can occur when the data sources are not well vetted. Data quality is probably the single biggest problem with ML models, and I wish we’d see more of a focus on it.

I'm less familiar with poisoning, but at least for test-time robustness, the current benchmark for image classifiers is AutoAttack [0,1]. It's an ensemble of adaptive & parameter-free gradient-based white-box and gradient-free black-box attacks. Submitted academic work is typically considered incomplete without an evaluation on AA (and sometimes deepfool [2]). It's good to see that both are included in ART.

[0] https://arxiv.org/abs/2003.01690

[1] https://github.com/fra31/auto-attack

[2] https://arxiv.org/abs/1511.04599

I am very glad to see this. I looked for techniques to counter adversarial ai, and I was disappointed to find a lot of useless approaches and nothing actually useful. Many people have published ideas, without seriously trying to attack them. I hope someone can identify better approaches.

So is "adversarial" the A in Generative Adversarial Networks (GANs)? Are they just tiptoeing around the term deep fake?

I realize that IBM is the US government's IT department but their involvement doesn't instill a great deal of confidence that anything this program has created is more than a heavily documented dumpster file

The IBM Watson group have been at the forefront of AI research and generally operate separately from the IBM umbrella. I know a few people in the group, and they are a brilliant bunch.

I don't doubt that they hire some brilliant people, but I haven't encountered their product in the real world doing anything of note. It always seemed to me that IBM created some parlor tricks for PR, but that Google, Facebook, Tesla, Microsoft, et cetera have been years ahead of IBM at every turn.

You know I'd love to read a good article on the inner workings of Watson, especially in the early years (2008-2012).

I remember reading news pieces about Jeopardy and discounting it as a cheap statistical party trick, especially given how it was about the same time terms like "big data" started proliferating in the media.

I only became convinced about the potential of machine learning much later, when CNNs starting being impressively good at classification. This prompted me to look further into it and realize it's not the latest business buzzword but an emerging field.

A book about IBM and Nazi Germany https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

What is your point?

Probably that blinkered commercial justifications have been shown to contribute heavily to negative outcomes, and that funding by government for military purposes is historically a key channel through which these outcomes are generated. The example of IBM during the Holocaust is given.

Yes, there are counter-examples like the internet.

In other words, engineers should consider what they may be contributing to ethically before doing so. The justifications "If I don't do it, someone else will" and "But the building blocks are already public" and so on are tired and morally invalid.

To a nontrivial extent FAANG have similar stigmas.

> morally invalid

Genuinely curious, because that sounds so definitive; so what's the way to determine whether a viewpoint is morally valid or invalid? I'm not talking about this specific example about military-commercial funding, but how to consider statements like "If I don't do it, someone else will" on its own. The justifications are forms of reasoning, so I imagine there's an overarching principle to evaluate these ways of reasoning?

Indeed. It's called the asshole precept, aka. "Do unto others", "Leave the world a better place", "Ahimsa", etc. Most cultures have a version of it, including agglomerating meta-traditions such as Baháʼí, which is testament to its popularity across time and space.

That's a great answer. I can see it that way now. Thanks.

Reply on news.ycombinator.com