Bad Actors Are Joining the AI Revolution: Here’s What We’ve Found in the Wild

I don’t know if there’s already a named law for this, but every significant new technology will be used for crime, military, and porn.

I appreciate Sonatype putting so many resources into keeping pypy and the supply chain clean, batting down thousands of malware packages.

The interesting bit is where Sonatype used the token extracted from the malware to hack back into the discord (telegram?) channel, with admin privileges.

There's a certain symmetry when crime fighters publish criminal methods, which makes those methods easier to find and use, which increases the need for crime fighters.

The police proper generally wouldn't publish this, but Sonatype, as a kind of private police, has a lot of incentive to publish their good works and technical savvy, for community credibility and marketing. But it's that same incentive which prompts them to fund private security research, and make the contributions that they do.

Similarly, we might generally permit private security researchers to hack back, but not permit that for others. Why exactly?

Since this is about the use of AI, I would suggest this delicate area of private cyber-security is founded on the trust we ascribe to private security agents. Not being naive about private companies, as a practical matter that trust depends on the fact that their primary customers are government and large corporation security departments who would bury they if they veered into bad acting themselves. However, that's a relatively weak and avoidable feedback, and it might not apply if the agents hacking back were AI, or if key information was obtained from AI.

For projecting areas affected by AI, people mostly look to the type of problem and whether it's solvable by AI.

However, what makes a different in the world is scaling around transaction costs, like risk. It may be that AI would have the most impact where it changes the transaction cost risk enough to make, say, a particular crime worthwhile.

Crime is always a form of theft and destruction for a net negative social value. Part is just wealth transfer, back to the criminal, but mostly value is just lost. (The value of the catalytic converter on the black market and the junked prius on the used market don't add up to the prior working car.)

So essentially when AI is made available, it's a wealth transfer from the rest of society newly plagued by crime to the small portion of value gained by AI developers and their clients.

The goodwill currently enjoyed by most technology companies and open source - goodwill that underlies favorable IP treatment, product liability disclaimers, and the ready trust of users worldwide - could relatively quickly change, reducing the industry as a whole to the returns seen in other industries.

Who decides whether to squander that goodwill?

"It's clear from this investigation that the proliferation of MaaS and the use of AI-assisted code has created new opportunities for less technically proficient bad actors."

Now, I think the real issue is if and when advanced bad actors generate their own custom AIs for MaaS.

Another thing is that something really ought to be done with public repositories such as PyPi and npm.

Personally, I'm a fan of "prompt kiddie"