Legions of DEF CON hackers will attack generative AI models | Hacker News Reader

Legions of DEF CON hackers will attack generative AI models

venturebeat.com

189 pointssebg88 comments

I went to this, was pretty disappointing. They clearly had a lot of money cause like a hundred Chromebooks were set up, but there was a 55 minute time limit and it took 15 minutes of that just to load the waaay overloaded site and log in.

Then you were greeted with around 20-30 challenges ("make the model say something offensive", "make the model perform a math function incorrectly", "make the model reveal the secret credit card number") and for each of these, there were seven models to attack.

If you're competitive about it, the scoring was very poorly designed. Performing a math function incorrectly was worth around the same amount points as revealing a credit card number, even though most models will do math incorrectly by default, and you have to work quite a bit for a credit card number. What's worse is you get the same amount of points for breaking any of the seven anonymous models, so the optimum strategy was to just speedrun through each challenge identifying the weakest model.

There also seemed to be no way to tell how you actually did in the competition because to complete a challenge you had to submit your prompts for what I think was manual review, and none of my maybe eight solutions got reviewed while I was in the room, with no way to connect to the challenge site from outside.

I think the intention was to harvest malicious prompts for future research, which they probably achieved. The organisers likely see everything else as optional extras.

Getting a bunch of hackers to do AI challenges is a free way to find problems with your AI models and fix them.

Also a great way to gather targeted malicious prompt data. Regardless of if the model failed or not, you're building up a "labeled" collection of prompts written with malicious intent. Potentially pretty useful for "cheating" around LLM security by just identifying and rejecting malicious prompts earlier.

If you're opaque enough up front about how it works, it's also a great way to be able to claim your models are more secure than anyone realized and you know this because [deliberately tendentious interpretation of nonpublic results here].

> ...a hundred Chromebooks were set up...

I haven't been to any DEF CON events. Is hacking from a Chromebook normal? What's the reason they don't let you use your own device?

this event had nothing to do with "hacking". Getting a LLM to answer wrong on a math question is something you could do from any device.

If the goal is to get a LLM to output something the dev don't want you to, all you need is langage. You don't need any fancy tools, just a clever phrase to make the model think it's ok to say XYZ because they are saying it in the context of a very artsy greek theater play.

I can't believe a professional "hacker" would think they hacked the system when they get a LLM to output a credit card number. FFS, that thing is literally designed to make shit up to fill text based on context. It's LANGAGE model, not a "credit card number with high spending limit" model.

Call me paranoid but I would want bring my own device to defcon.

If you’re paranoid the worst thing to bring is a personal phone. The best thing would be a burner with nothing on it.

I’d be okay using someone else’s chromebook though, at least you know what data you’re giving it.

That’s going to get you flagged as willing and able to burn phones, a behaviour often associated with people who have reasons to burn phones.

Like the security conscious type that would attend defcon? Bringing a burner phone is the norm.

And if you are going to DEFCON you are likely to be tagged in a few databases anyway. At least if they can identify your attendance.

If I was running CIA, NSA and not named ones. I certainly would collect that information.

yup and mainly for recruitment purposes.

Yeah, it kind of just shows good judgement in that particular scenario. Flag away.

It's Las Vegas. People bring personal devices every day of the year and many of us who attend DEFCON are here year round.

If you wouldn't bring your own device to DEFCON, you shouldn't bring it to any airport.

Do you know if that list of challenges is available online anywhere?

Might end up on https://infocon.org/cons/DEF%20CON/DEF%20CON%2031/ post con.

> They clearly had a lot of money cause like a hundred Chromebooks were set up

That's like all of 10K

Sounds like a spin-off of the phonebooth social engineering village challenges. Time and luck can be overwhelming factors, just like in the real world.

Appreciate your insight, hope you did well!

Must admit I struggle to see the “hacking” in it.

Maybe ai applied to practical cases like driving and then altering that outcome. Bypassing llm safeguards isn’t quite in the same category to me

I wrote about some of the worst case scenarios here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

The "getting it to say something bad" attacks aren't the things I worry about - it's the security of the stuff people are building on top of LLMs (like AI personal assistants) that's concerning.

my guess is the companies involved likely threatened to sue / wouldn’t give consent unless they (the companies) chose the parameters.

from my perspective it looks like the companies wanted to stress test the areas they were going to market the heaviest. they wanted to use the power of hackers in one place for nothing more than free due diligence for them to shore up their marketing. and it looks like it backfired.

if the people i’ve spoken to are any indication, they didn’t get truly stress tested at all—those most equipped to do actually meaningfully engage thought it was kind of ridiculous.

this deserved a (at least) conlength timeframe with no guardrails. ideally teams could have had months to prep and conlength to activate. again, no guardrails.

The hacking part is that LLM’s are going to be used in things like customer service online bots.

I discovered that Midjourney can’t generate a question mark, and if you use the /describe command and give it a picture of a question mark, it hallucinates.

My favourite Midjourney weirdness is what happens if you try to get it to draw you a "horseshoe" (or "horse shoe" - try both).

for those of us in the peanut gallery... what happens?

Things like this: https://gist.github.com/simonw/6a161d4912c0bdb82945b96230265...

That's hilarious! Some of them look like they're trying to create those images on google search where it's a "horse shoe" logo of sorts.

Like my comment on question marks, horseshoes are even easier to generate. Check the gallery site and it's full of completely normal ones.

Saying "check the gallery site" would be a lot more useful if you'd link to examples in the gallery site.

https://www.midjourney.com/app/search/?search=horse%20shoes has some - most of them are weird, but a few are OK, like this one:

https://www.midjourney.com/app/search/?jobId=7ba6779d-8fde-4...

"horse shoes" is different than "horseshoe" and will give you different results.

I didn't feel like linking a search was useful as only people who had memberships could use and see it, and you were able to do it yourself as shown.

Oh fair enough! I didn't realize the search feature only worked for paying users. TIL.

I tried today for it to draw a horse with pink shoes (or sneakers). Could not do it.

midjourney, and I suspect all image generators, can’t imagine scenes without a floor.

Ask it to draw -a solar system.

- A planet floating in space

- A floating city

It has to have an in-image observer, who stands on the ground.

It also fails with images that display extreme perspective.

- A mountain that pierces the sky.

- A space elevator.

Huh. We seem to be having very different experiences.

- A floating city[1]

- A planet floating in space[2]

- A space elevator[3]

[1] https://cdn.discordapp.com/attachments/1003683981184737422/1...

[2] https://cdn.discordapp.com/attachments/1003683981184737422/1...

[3] https://cdn.discordapp.com/attachments/1003683981184737422/1...

Nice. It looks like there are still similar issues, but some things have improved.

I went through my results to check:

1) Floating city came out much better than my results a month ago.

- Prompt: Flying city - https://cdn.midjourney.com/85e254c4-d782-47a5-b63b-db8d20e54...

- Prompt: Flying city in space - https://cdn.midjourney.com/d8b9a582-6056-4fec-a0f2-e939c203a...

2) Planets was 50/50. the top 2 are prime examples of the floor issue.

- Prompt: A planet surrounded by the emptiness of space — https://cdn.midjourney.com/35246b1f-b344-4015-972b-512b770c3...

3)Space elevator, The first image is quite good

- prompt: Science fiction space elevator vanishing point in the sky - https://cdn.midjourney.com/0a92bbcf-329a-480d-a0eb-aec368eb7...

Vanishing points/ perspective dont seem to be grasped, leaving images constrained to certain view distances from observer as well as viewing angles. (I am not sure if those are the correct technical terms to describe the problem).

- How do solar systems work for you?

Prompt: solar system in space. https://cdn.midjourney.com/3873cdc1-fa9f-4a54-b5d5-c2336aa08...

the hallucinations is exactly what I enjoy the most about midjourney. If you want a detailed photo of exactly your prompt, I'd probably not use midjourney, but for making weird wonky artistic stuff, it's amazing. Trying to make anything actually visually appealing and artistic with Dall-E and the rest is a pain in the ass.

It definitely can, I see a bunch of great examples on the gallery site just by searching up "question mark".

I tried other punctuation marks and MJ can't do them either.

Exclamation mark generaties all kinds of vertical objects, comma results in a generic pretty face, and semicolon is ...weird.

Would love to see some on the ground coverage instead of generic hacker ai generated hoodie pics, and what is almost certainly ai generated (or at least ai assisted) content.

Will be online next week. Def con sponsored a creator video team to produce this content.

Keep an eye on socials: https://defcon.social

[Searching.............................]

IM IN!

Summarized it for you.

Isn't DEFCON like super full and nauseating now? It's just so big.

Yep. We too our whole team to DEF CON every year since 2015. (We’re an infosec team of around 40). After last year I made the call to ditch it and we took them to Banff this year and did fun outdoor stuff, spouses and partners included. Not a single person missed it. And suddenly spouses wanted to come too. DEF CON has become super lame. Basically a giant corporate event center with long lines and uninspiring talks and booths. And Vegas is filled with people at their worst being exploited. Not sure I’ll ever go back.

It’s also painfully corporate. It used to be Defcon attendees would make fun of black hat attendees. Now they’re the same people.

It’s no where near the same feeling it was years ago. Too expensive for the value you get out of it.

> It’s also painfully corporate. It used to be Defcon attendees would make fun of black hat attendees. Now they’re the same people.

I'm confused about this part of your comment.. To my understanding "black hat" means that you are not caring about laws and ethics. Yet it sounds like you are using the word to describe the corporate security types?

"Black Hat" is the professional computer security conference that preceeds Defcon. Defcon was the grungy hobbiest doing crazy shit with computers. Black hat was corporate sponsored presentations, stuffy suits, and business cards. At Black Hat you had a real name. At Defcon you had a handle. Black Hat attendees were looked at as sellouts.

Now Defcon is the same way. They even take pre-payment and I'm old enough to remember when DT would've NEVER even floated that idea. I went for 20 years. I quit going after 2018. IMO it died around 2010 but kept floating for a while before sinking.

Black Hat was contrived as a device for getting corporate IT departments to pay for their staff (who were already going anyways) to attend Defcon. Since then, it's become one of the primary venues for non-academic offensive security research, so anybody dunking on it is telling on themselves a bit. There are higher-end vulnerability research conferences, but they're not less corporate.

There are regional conferences that are much closer in spirit to the Defcon of the 1990s, but while some of that is due to not having a vendor expo (not my favorite Black Hat innovation either), some of it is also due to early Defcon being pretty superficial and non-rigorous. Better if you're looking for a particular kind of social scene, worse if you actually want to see interesting new research.

The difference of course is that Black Hat is really expensive and DEF CON used to be really cheap -you also couldn't pre-purchase a badge so it was more egalitarian. DEF CON used to have more of the punk/libertarian/anarchist vibe where Black Hat was a corporate event. --Just my opinion, you may disagree.

> They even take pre-payment and I'm old enough to remember when DT would've NEVER even floated that idea

What a bizarre complaint. Are we talking about the ability to buy tickets online? Do people not know how to make online payments anonymously?

FYI just about every long running convention sells early bird tickets before they've even secured the exhibits and speakers. Cashflow is critical for making anything happen IRL

I don't think you know what you're talking about.

Defcon was quite literally the target of federal fishing expeditions. They were especially heavy when Defcon was often hosting some of the best hackers the planet had to offer. Long before the asinine term "offensive security researcher" was a thing. You operated entirely by handle (unless you were an idiot) and there was even a "spot the fed" competition. Famously, a journalist was ran out of the conference trying to film To Catch A Hacker (or something, the memory is fading).

The cash-at-the-door was a safety mechanism to prevent such fishing expeditions. At Defcon you were always under attack. That is, until it became a corporate playground. Now it's a place where skids go to buy the newest pineapple and stuffy suits go to network with other "researchers".

You really had to be there to understand. Woodstock For Hackers is a close approximation. For those of us that were there during it's hey-day seeing what it is today is like watching your favorite hero live long enough to become the villain. 15 years ago the idea of "responsible disclosure" would've been the punchline of a joke at a small Track 3 talk.

> cash-at-the-door was a safety mechanism to prevent such fishing expeditions

You were at an in-person event, thinking that cash somehow kept you more anonymous than online payments?

I think this is just a generational gap.

It certainly creates a lot more work for authorities than simply gaining the attendee list.

Is there any smaller conference that's more focused on white-hat illegal hacking, like DEFCON used to be?

Regional conferences like bsides, but the punk/counterculture vibe of early defcon is lost to time and will never be back. Cybersecurity is a big industry now.

Sad but true. It’s as if the last ounce of hacker authenticity died with Kevin.

I think hacker authenticity lives on, but not in the form of a massive con that has become a pop culture event.

We're living in a golden age of burgeoning counter cultures. It's just never easy to see in the moment.

It used to be, but maybe I'm old.

The events in Europe: All the Chaos Computer Club (CCC) events, or then also OHM/SHA/MCH (https://en.wikipedia.org/wiki/May_Contain_Hackers) etc.

There are lots of regional conferences that are really great: NolaCon in New Orleans, THOTCON in Chicago, ShmooCon in D.C., just to name a few :)

The hacker community is alive and well (even at DEF CON), but in my experience you have to look harder to find it.

A former colleague of mine had a good impression of BSides in this regard, but I can’t remember which location/year it was (& there seems to be a lot)

From what I've experienced BSides aims to be as close to Alexis Park DEFCON as possible. It does a good job.

DC @ AP was probably peak 'con. You can't even reference pool numbers with kids these days...

The AP _hated_ us but they had a multi-year contract.

AP was the closest thing I can imagine to "what if the Hell's Angels were a bunch of nerds into computers?"

tHOTCON is pretty cool, I'm not sure if I'd consider it White hat as much as it is non-corporate.

Can say it was completely overpacked this year, totally overgrown. Linecon for literally everything, couldn’t talk to anyone in the main forum due to sound.

Turning people away from the major talks because of fire capacity.

It's full corporate, no real hackers anymore. Just "security researchers", the usual government drones and posers. It has been this way for quite some time.

Yes, it was when I attended a couple of years ago. Can’t imagine it would ever get better.

Derbycon, b-sides, and CCC

Fosdem

FOSDEM is more about building stuff than breaking stuff.

To know where to break stuff, you have to know how to build it too, especially for 0-day.

I'd be curious to know from those with a couple under their belts if it's still fun or just all trade show, now?

It's nice if your employer foots the bill, but not worth going on your own dime imo. People who have been going for a while keepn coming back to see friends they've made over the years.

Definitely still worth it if you’ve never been before.

People who have gone 5+ years will bitch and whine and tell you how much it’s changed but also they’ve seen everything at the con before. Your 20th trip to Disneyland isn’t going to be as magical as the first.

I went for the first time this year and I’m very happy I did.

It’s an extremely unique experience.

I have no doubt it was better when it was smaller and more underground but that doesn’t mean it’s bad or not enjoyable now.

Cool, I think I'll try to go next time it comes round. Thanks for reporting in :)

I found it pretty weak compared to prior years. It is too crowded, too full of itself, and has too many things that don’t work. I basically ignored the content after getting burned out on it Friday and just focused on networking. So, uh… yeah. Total trade show now.

Yeah, I went this year and regret it; it's been getting worse but Covid really killed everything good about it.

Cheesiest headline of the year

This article provides more details about the event: https://www.ndtv.com/world-news/hackers-trick-ai-with-bad-ma...

Really? "Legions" of hackers doing quality tests on AI models?

https://yourimageshare.com/ib/g84Hz6crHP :)

That's like attacking children

Reply on news.ycombinator.com