OpenAI Cybersecurity Grant Program | Hacker News Reader

OpenAI Cybersecurity Grant Program

openai.com

87 pointsstaranjeet37 comments

I miss the 90s security groups which were both funky and respectable and (most importantly) devoid of corporate lingo.

Semi underground stuff like Phrack, 2600 and various punky groups who surprisingly did good security research and open to participation..

Today's cutting edge security research feels very silod in the products.

Oh, the nostalgia... the associated EFNET channels were fun.

I wouldn't even say it's surprisingly good research at this point. A lot of the basics being taught in universities still cite Phrack for example.

A bit related: OWASP is working on a Top 10 for large language models.

OWASP Top 10 for Large Language Model Applications

https://owasp.org/www-project-top-10-for-large-language-mode...

The draft of the Top 10

https://owasp.org/www-project-top-10-for-large-language-mode...

Is that a ranked list?

My guess is sort of. If they really wanted it to be ranked they might have the number in the ID repeated at the start of each of them.

I thought this would be about a bug bounty program, which it isn't, but it did lead me to check if there is an OpenAI bug bounty program, which there is: https://bugcrowd.com/openai

Does this bug bounty include passthrough issues like the ones here?

https://simonwillison.net/2023/May/2/prompt-injection-explai...

No, it explicitly does not include prompt injection attacks for obvious reasons.

The obvious reason that they don't want to be responsible for the ensuing clusterfuck?

Likely because prompt injection is basically instruction injection, aka. asking it to do anything…

Unless the prompt somehow bypasses a meaningful security boundary (and not just some best effort moderation) I don't see why they would care at all.

For areas that require a great deal of confidence in conclusions, like forensics and incident triage, I can only see generative AI making matters worse. There’s already a ton of over-automation and humans making mistakes because they trust their tools too much. Giving that industry access to more tools that confidently give wrong answers almost seems irresponsible.

I am excited for AI-based threat intelligence products, though.

I could see the exact opposite happening. Forensics and IR often require a ton of complex information parsing that an LLM could accomplish very quickly and then point an analyst in the right direction to verify. It's not meant to be a panacea just an accelerant.

Threat intel has purported to use AI for ages and I remain unconvinced of its effectiveness. Program analysis to support TI could be exciting however.

Agree, but having the LLM explain how it arrived at the output will be crucial for provenance and anti-hallucination purposes. Judges don’t care for fabricated evidence or citations.

This language bothers me deeply. So many people ask the system itself to describe what it is doing, for which it is absolutely incapable of doing. The system has no access to its internals.

Even academics are asking the LLM itself to describe what it's doing like idiots. I know that academia's intellect is slipping due to allowing too many people in, but this type of stupidity is just mind boggling.

I like to preserve skepticism here as well, though probably blaming them won’t help the issue.

But absolutely agreed, just listened to Gary Marcus’s podcast episode on how LLMs are already capable of making dad jokes, but they can’t do neither “deep” humor generation nor “deep” humor explanation very well, in part because they lack all of the cultural and experiential (“touchy-feely”) context of the humans, and an even bigger part since they are optimized for generation, not explanation, while humans are almost the opposite.

I think you misunderstood my comment or I didn’t articulate the idea clearly. I’m simply arguing for a verbose or debug mode that can be toggled on to show how output was derived, not expecting an LLM to have awareness or understanding of itself or it’s internal workings.

That's fair, an encoder system for which the full attention values and weights available is a useful tool.

IMO encoder models are far more powerful and useful for devs than any of the decoder models.

> I am excited for AI-based threat intelligence products, though.

Have examples of existing product or project in this space that are interesting to you?

There’s traceable.ai, for one (not affiliated, but testing it). I’d say none of traditional ML products are very good, at all (at least among the ones available publicly). Most of them are just snake oil. But I’d err on the side of plausibility that LLMs might finally add fuel to that, at least on the pentesting side (might, not yet will).

Additionally, even OpenAI’s own advertisement “paper” acknowledges they couldn’t make it work for pentesting well (hence probably why they created this bounty program, among other reasons).

FWIW there’s this paper that just came out, and promises a low hallucination factor (not quite for pentesting, but this could be a useful starting point):

“Gorilla: Large Language Model Connected with Massive APIs” https://arxiv.org/abs/2305.15334

Looks like a good initiative.

Cybersecurity is certainly one of the near term (now?) AI related risks.

This should help encourage some defensive progress.

Wait 'til they find out that the kind of stuff they build is way easier to apply to attack than to defense...

Yup. Defense is mostly technical nuance. Attacks outside of academia often rely on the weakest links, distracted or scared humans. Chatgpt and LLMs like it can easily mass phish or otherwise thousands of targets in a matter of seconds. Can also be used for OSINT on human targets... Combine the two and ransomware will be a true cottage industry ran by script kiddies.

That said there are things they could do to prevent this. They won't do them, it'd be costly research... But they could.

We don’t seem to be quite there yet, but things are moving in that direction, unfortunately (see the Gorilla paper, for example). I still think to tailor the tool to each particular API requires human intelligence (and even that is hard), but a shotgun approach might be more “successful” even if it’s 5% effective.

That said, more traditional approaches (non-ML) are probably more effective still.

What about AI security itself, everyone is afraid of rogue AI, but a "dead" AI or "locked up" AI can't do harm. I.e. Have some sort of monitoring system to monitor AI systems, and a sort of kill switch that can disable it from the internet, or if need be trigger a small onboard EMP device to fully nuke the device.

It could give some safety if skynet becomes a thing, of course super AI will learn to disable it in a week, but still maybe there's a path there to mitigation of future threats?

Funny how it's all going William-Gibson-Sprawl-shaped. The best defensive ice (Intrusion Countermeasures Electronic) was written by AIs who constantly updated it, and the best icebreakers were also written by AI.

I'd be very surprised if secretive agencies within the USA, China, Israel, Russia and others weren't going full-bore on the offensive side (which OpenAI says this particular grant program is not involved in, at least, but I wonder what the black-budget NSA-TAO side of things looks like at present).

Indeed any AI defense capabilities will necessarily need to include some offensive abilities, which for the public are mostly censored right now (eg try asking for any of the obvious public PoCs which it almost certainly was trained on) I find it to be one of the worst aspects of the current alignment….

I don’t think it’s a serious limitation though, since the open-source LLMs like Alpaca/LLaMA have already “won” the mass adoption among engineers, and are only getting better (already better than GPT4 in some areas).

One of exciting outputs of LLMs will be, I hope, to automatically port large software projects to other languages.

Free startup idea:

1. Create something that monitors an app's network connections

2. Use the data to auto-generate integration tests for every endpoint including edge cases, error states, etc

3. Have user validate and accept the integration tests

(end there for self-serve tier, and for the enterprise tier...)

4. Rewrite app into another language, framework, or simply refactor code to be cleaner, more consistent, convert callbacks to promises or async/await, etc.

... key step being #3, need a really good test suite to validate the new code works the same as the old.

> We will grant in increments of $10,000 USD from a fund of $1M USD, in the form of API credits, direct funding and/or equivalents.

Does "increments of $10k" mean that a single grant can be greater than $10k? Or are grants limited to that amount?

Valid question. I read "increments of £10k" to mean 100 projects/groups or less.

thank you for posting this here, wouldn't have found it otherwise

why exactly did i get a downvote on this? HN is against thanking people for posting stuff?

This is clearly a play for getting access to organizations' logs for them to train their models on - probably in preparation for a future cyber related offering.

Reply on news.ycombinator.com