I feel like in near future the meme about "it's all just chatbots emailing each other" will actually be true. I wonder when I will receive my first AI generated email and will I bother to respond to it at all
We're in the "fuck it we ball" era.
Not even the many reports of prompt injection and takeover due to IA misfeatures can cheer me up anymore.
It is all incredibly sad.
Oh hey, I know this one! It’s humans and a phishing test, they’ll click on all the links and enter information without checking the domain properly!
Personally, I’d like systems that aren’t open to attack and can be depended upon. But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), not any software written in languages without memory safety, not even the ones with (Log4j comes to mind), not the packages in many package managers, not other humans and sure as hell not the token prediction machines. What a world.
We all probably live with a 0.XX% chance of getting pwned any given day.
Clearly you feel alarmed, but it's important to base these "alarm" feelings on actual evidence and real concrete proof of something being bad. You clearly don't have a proper understanding of the exploit, so please take a moment to re-read what actually happened and how it would be exploited in practice, particularly the "the scope of this attack is smaller than it sounds" comment chain: https://news.ycombinator.com/item?id=49496918
Overall, I agree with you though, and it's a healthy perspective to be safer rather than sorrier, so living with the assumption that getting pwned any day is a non-zero chance/risk is probably the best approach and what I personally do too.
We'll see how it goes, but the product in question (Conveyor) is a downloadable tool that's got deliberately unobfuscated bytecode in it, with lots of detailed logging. AI is perfectly capable of reverse engineering it and in fact this bug report contained such a reversing. So even if someone tricks it into revealing source code or similar, they won't get anything that isn't already obtainable via other methods. This isn't a SaaS where security through obscurity might conceivably help, or where the codebase might contain credentials by mistake.
It's a developer tool and this level of trust helps customers debug their own problems quickly. If someone wants to break the law, they'll get a legal answer, but it's never been a problem.
The bot in question cannot write to master though, only open up pull requests from its own isolated repository.
It's a bet on modern models being more resistant to confusion attacks than they were before. The harness setup also makes it very clear to the model where input comes from. This might be a bad bet, but if it's not, then it's helpful for customers to get help right away.