I'm not sure it's anything to fret about. Someone who has the ability to inject a prompt into your AI probably has the ability to run arbitrary code as your user. The prompt injection is the strictly less worrying part of the exposure you have.
I'm not sure it's anything to fret about. Someone who has the ability to inject a prompt into your AI probably has the ability to run arbitrary code as your user. The prompt injection is the strictly less worrying part of the exposure you have.
The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue.
The author admitted there were personal and professional consequences in their blog post despite the small surface area.
They literally use that example in the decision. Quote: "The example usually given by those who would punish speech is the case of one who falsely shouts fire in a crowded theatre.
This is, however, a classic case where speech is brigaded with action. ... They are indeed insep- arable and a prosecution can be launched for the overt acts actually caused. Apart from rare instances of that kind, speech is, I think, immune from prosecution."[0]
That is to say, shouting fire in a crowded theater with the intent to cause harm is actually one of the few cases were it actually would be illegal based on that decision.
[0] https://tile.loc.gov/storage-services/service/ll/usrep/usrep...
> Ultimately, whether it is legal in the United States to falsely shout "fire" in a theater depends on the circumstances in which it is done and the consequences of doing it. The act of shouting "fire" when there are no reasonable grounds for believing one exists is not in itself a crime, and nor would it be rendered a crime merely by having been carried out inside a theatre, crowded or otherwise. If it causes a stampede and someone is killed as a result, then the act could amount to a crime, such as involuntary manslaughter, assuming the other elements of that crime are made out. Similarly, state laws such as Colorado Revised Statute § 18-8-111 classify knowingly "false reporting of an emergency," including false alarms of fire, as a misdemeanor if the occupants of the building are caused to be evacuated or displaced, and a felony if the emergency response results in the serious bodily injury or death of another person.[16] Somewhat more trivially, in some states it is a crime just to knowingly make a false report - or knowingly cause a false report to be made - of an emergency to emergency services.[16] In Colorado it is a crime to knowingly cause "a false alarm of fire" to be transmitted to "any...government agency which deals with emergencies involving danger to life or property."[16] This crime could plausibly be made out where, for instance, in response to the false shout, an innocent bystander calls emergency services to report the fire, and this is found to have been such a foreseeable response to the shouts that the shouter is deemed to have caused the false report to be made.
Whether those laws actually survive the Brandenburg test is untested, from my understanding. But given that potential first amendment violations are held to strict scrutiny, I question whether the government could actually pass the imminent lawless action test even had someone did it knowing it would cause a panic, and would need to try with some other offense.
As I said, I haven't watched Legal Eagles full video, but I don't like that ever since that video came out everyone on the internet tries to correct anyone who uses the phrase even in it's colloquial sense. Maybe the source video covers the nuance (I wouldn't know), but the folks on the internet "umm acshually"ing everyone seldom do.
I don't see why prompt injection to delete files on someone else's machine would be any different.
> I don't see why prompt injection to delete files on someone else's machine would be any different.
The difference: they chose to download and execute your prompt without examining it, vs you injecting it into their system.
In the second case, I've provided instructions on how to destroy the data, said "don't do this", and then someone has done it anyway. _They_ have destroyed their data, and it's now up to them to justify why that's my fault.
If we want to get into legal territory about it, which I'm sure we're both woefully underqualified to comment on... the CFAA is all worded around "intentionally accesses a protected computer...". How exactly do you show intent to access a protected computer here? The developer never took any sort of positive step to access _any_ specific computer. The best I could see would be negligence where a reasonable person would have to have known someone would run this on a protected computer, but that still feels like something a good lawyer would find a way out of.
If his non-wrong acts could be criminally prosecuted like that, this case - intentional damage! - is even riskier.
Whether it was via prompt injection or SQL injection is irrelevant. Whether you agree with his politics or not is irrelevant. All that matters is he wasn't authorized to delete code from your system, and he abused the level of access granted to him to do that anyhow.
Yet, hopefully we can agree that sql injections are illegal.
You are authorized to do what the user agreed to, no more. Further the agreement must be reasonable. Exploiting the victims system to intentionally cause harm isn't reasonable.
F-secure once included a clause to use their wifi that you "assign their first born child to us for the duration of eternity." It was funny, but not legally enforceable and would have offered them no legal shelter if they'd gone out on a kidnapping spree that night.
If we're slicing on technicalities, there's a lot of ways to decide. "PROSECUTE THEM!" seems like an extremely hostile one when the website and readme and release notes said "don't do this" already. The agent ignored those things? Is that the author's fault?
Say I lay a log on a road which you can clearly see and avoid but choose to drive over and crash your car, that’s prompt injection.
One is way worse than the other.
The prosecution wouldn't even blink if you pointed this out.
Unless the perpetrator intended for that to be the effect.
Have you heard about mens rea?
It turns random logging into laying logs onto a road intending to harm someone with the foreknowledge that they will harm the target and as a consequence any other people traveling on that road.
Terrorism charges and straight to gitmo.
Is writing a prompt in your blog any different from laying a hunting trap on your own property and catching someone's pet? Are you liable for the pet wandering on your property?
Start laying hazards in the middle of the road and see how quickly the police introduce you to things like “reckless endangerment” and “involuntary manslaughter”. The general social contract is that you don’t take actions with the intent of causing harm to others regardless of whether the victim could have avoided the harm had they taken different actions.
When it comes to responsibility, usually we consider a person intentionally doing something that they reasonably believe will have some consequence as responsible for that consequence. Especially when the primary reason they took the action was to generate the consequence. Excuces of the form "Technically i didn't do it, i just knowingly did something for the explicit purpose of triggering some downstream consequence" generally do not fly.
Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead.
I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an unexploitable manner, but it will usually result in ties, and lost time and money to rake, and theoretically any attempt to exploit another player, leaves you exploitable to another player. The classical example is rock paper scissors, unexploitable strategy is to play randomly with p=1/3 for each choice, however if one really wishes to win more often than their opponent, they have to guess, and if in that guessing they choose an option with 100% certainty, they become exploitable to someone choosing another option with 100% certainty.
In effect the very act of attempting to extract value from free software, is the very act that leaves one vulnerable to being extracted value from.
I do not think that someone's status as a contributor to open source mediates their safety from supply chain attacks. Big companies that donate gobs of money get hit, and so do small operators who have contributed nothing are just trying out a hobby project.
If you pay for software, your supply chain risk is reduced, if you don't pay for software, your risk is increased.
But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense.
Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link. I don't think there's anything there. I will grant that you are at slightly less risk from software you know well and contribute to directly, but that's only of any help for very low level stuff that doesn't have many dependencies.
I'd look beyond software as it seems more of a general economic or political matter.
Here's what I found:
Theory that there's an incentive to pay above market price: https://en.wikipedia.org/wiki/Efficiency_wage
Namely that the 'market price' for some software might be 0$ or 0+tips/donations, but paying in excess of that would be an efficiency wage.
https://en.wikipedia.org/wiki/Principal-agent_problem
Academic term for the conflict of interest, which would be applied to the difference of interests between an OS dev and its downstream users.
https://en.wikipedia.org/wiki/Multiple_principal_problem
This would not apply to something as extreme as a supply chain incident, but if an OS library has multiple users, it can't serve all of them equally well. If one consumer throws a big donation, of course they will serve them better, potentially at the expense of others.
Experiment on corruption and wages for public officials: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2039238
Self explanatory.
https://en.wikipedia.org/wiki/Shapiro%E2%80%93Stiglitz_theor...
Source of the efficiency wage concept, more detailed and mathematical explanation. Shirking would be the term for a worker not working or working against their employer.
There is mention to the fact that not all contracts are perfect, and there's always some implicit terms, governed by reputation at least (like, say, the quality of the work will be good, or you won't plant a worm in the code). Also it is mentioned that the costs of monitoring perfect contracts would be too high (for example defining a system of story points and ensuring that the story points contributed are above a certain level, or auditing code and code changes to ensure there are no worms in the code), so an efficiency wage provides an incentive for the worker to self-police, potentially removing the cost of auditing.
Just think about what the contrary would imply, if paying workers didn't increase security, then a free worker writing code would be as secure as a paid worker writing code. In order to introduce security you'd have to introduce an external auditor, which can be free or paid. An auditor cannot introduce vulnerabilities, but they can shirk and get paid without doing the work, but you claim that a system with a free codewriter and a free auditor is more secure than a system with a paid codewriter and a paid auditor? I contend that a system without auditor and a paid codewriter is more secure. And it is even more secure if the worker is paid in excess of perfect market wages, potentially only based on incentives, but also due to the fact that their payment is conditional on job execution, depending on the contract type, if they are negligent on their job, you can recover at least their payment in court, and possibly additional damages.
Finally, I couldn't find sources on the discussions around the ends of 18th century when mandatory salaries for public officials where defined in constitutions of countries of the Americas, but it was my understanding that in most democracies public officials MUST be compensated, it is often justified that otherwise only wealthy classes would have access to such positions so this increases representation, but it's also possible that resistance to bribery and corruption is another reason most governments converge on this decision. The experiment cited supporting this correlation.
I'll add even more from anecdotal experience, sometimes prices can be too cheap, and this will dissuade buyers of the item, because we have an internal model of what cost structures are like, a price too cheap suggests underfunding or cheap materials. I can personally recount many examples where a sale was lost not because the price was too high, but because it was too low. And 0 is a number, which again might be too low. In this case the rule is not about security strictly, but about sale price being a signal of the investment in the quality of the contracted asset. Naturally the cost invested into the product cannot be lower than the price, excluding price dumping (which arguably are not deals you want to take), or lead lossing, so it's a necessary conclusion that low price signal low investment and thus quality, not only of the visible immediate product, but of less visible qualities like future commitment to the product, and less visible present qualities like hidden backdoors.
I grant that if you act as a significant source of funding to a given entity, that entity is less likely to hurt you, if it knows what your interests are. Of course, this is completely impractical for hobbyist users of open source software.