Extracting AI models from mobile apps
altayakkus.substack.com
altayakkus.substack.com
An analogous situation is seeing a blog that purports to "show you code", and the code returns an object, and commenting "This is cool, but doesn't show you how to turn a function return value into a human readable format" More noise, than signal.
The techniques in the article are trivially understood to also apply to discovering the input tokenization format, and Netron shows you the types of inputs and outputs.
Thanks for the article OP, really fascinating.
Key here is, a binary model is just a bag-of-floats with primitively typed inputs and outputs.
It's ~impossible to write up more than what's here because either:
A) you understand reverse engineering and model basics, and thus the current content is clear you'd use Frida to figure out how the arguments are passed to TensorFlow
or
B) you don't understand this is a binary reverse engineering problem, even when shown Frida. If more content was provided, you'd see it as specific to a particular problem. Which it has to be. You'd also need a walkthrough by hand about batching, tokenization, so on and so forth, too much for a write up, and it'd be too confusing to follow for another model.
TL;Dr a request for more content is asking for a reverse engineering article to give you a full education on modal inference
I don't understand what you mean: I have no clue about anything related to reverse engineering, but I ported the mistral tokenizer to Rust and also wrote a basic CPU Llama training and inference implementation in Rust, so I definitely wouldn't need an intro to model inference…
Except you just did - or at least you wrote an outline for it, which is 80% of the value already.
E.g. there are two common spelling of bizarre which are popular on Gboard : bizzare and bizarre.
Can something similar help in model encryption?
they also claim unsuppervisioned users typing away is better than tagged training data, which explain the wild grammar suggestions on the top comment. guess the age of quantity over quality is finally peaking.
in the end it's the same as grammarly but without any verification of the interested data, and calling the collection of user data "federation"
they have made the system so general that it can handle any quirk users throw at it.
So e.g.: Apple CoreML issues a Public Key, the model is encrypted with that Public Key, and somewhere in a trusted computing environment the model is decrypted using a private key, and then inferred.
They should of course use multiple keypairs etc. but in the end this is just another obstacle in your way. When you own the device, root it or even gain JTAG access to it, you can access and control everything.
And matrix-multiplication is a computationally expensive process, in which I guess they won't add some sort of encryption technique for each and every cycle.
You usually inference those on GPUs anyway, and they usually have some kind of hardware DRM support for video already.
The way hardware DRM works is that you pass some encrypted content to the GPU and get a blob containing the content key from somewhere, encrypted in a way that only this GPU can decrypt. This way, even if the OS is fully compromised, it never sees the decrypted content.
Look at the bootloader, can you open a console?
If not, can you desolder the flash and read the key?
If not, can you access the bootloader when the flash is not detected anymore?
...
Can you solder off the capacitors and glitch the power line, to do a [Voltage Fault Injection](https://www.synacktiv.com/en/publications/how-to-voltage-fau...)?
Can you solder a shunt resistor to the power line, observe the fluctuations and do [Power analysis](https://en.wikipedia.org/wiki/Power_analysis)?
There are a lot of doors and every time someone closes them a window remains tilted.
Try getting a private key off a TPM. There have been novel attacks, but they are few and far between.
Try getting a key from Apple's trusted enclave (or whatever buzz-word they call it).
But is the TPM doing matrix-mulitiplication at 1.3 Petaflops?
Or are you just sending the encrypted file to the TPM, getting the unencrypted file back from it, which I can intercept, be it on SPI or by gaining higher privileges on the core itself? Just like with this app but down lower?
Whatever core executes the multiplications will be vulnerable by some way or the other, for an motivated attacker which has the proper resources. This is true for every hardware device, but the attack vector of someone jailbreaking a Nintendo Switch by using a electron microscope and a ion-beam miller is neglectable.
If you are that paranoid about AI models being stolen, they are worth it, so some attacker will have enough motivation to power through.
Stealing the private key out of a GPU which allows you steal a lot of valuable AI models is break-once-break-everywhere.
Apple trusted enclave is also just a TPM with other branding, or maybe a HSM dunno.
However, if NVidia or similar was serious about securing these models, I'd be pretty sure they could integrate the crypto in hardware multipliers / etc such that the model doesn't need to be decrypted anywhere in memory.
But at this point there isn't much value in deploying models to the edge. Particularly the type of models they would really want to protect as they are too large.
The types of models deployed to edge devices (like the Apple ones) are generally quite small and frankly not too difficult (computationally) to reimplement.
My general writing style is directed mainly towards my non-technical colleagues, which I wish to inspire to learn about computers.
This is no novelty, by far, it is a pretty standard use-case of Frida. But I think many people, even software developers, don't grasp the concept of "what runs on your device is yours, you just dont have it yet".
Especially in mobile apps, many devs get sloppy on their mobile APIs because you can't just open the developer tools.
The community around Frida is a a) a bit small and b) a bit unorganized/shadowy. You cannot find that many resources, atleast I have not found them.
I would suggest you to use Objection, explore an app, enumerate the classes with android hooking list classes or android hooking search classes, then dynamically watch and unwatch them. That is the quickest way to start, when you start developing your own scripts you can always check out code at https://codeshare.frida.re/.
For everything else join the Frida Telegram chat, most knowledge sits there, I am also there feel free to reach out to @altayakkus
Oh and btw, I would start with Android, even though iOS is fun too, and I would really really suggest getting a rooted phone/emulator. For the Android Studio Emulator you can use rootAVD (GitHub), just install Magisk Frida. Installing the Frida gadget into APKs is a mess which you wont miss when you go root
(and a bunch of people seem to be interested in the "IP" note, but I took as, just trying to not get run into legal trouble for advertising "here's how you can 'steal' models!")
It's supposed to be "free-IDA" and the work put in by the developers and maintainers is truly phenomenal.
EDIT: This isn't really an attack imo. If you are going to take "secrets" and shove it into a mobile app, they can't really be considered secret. I suppose it's a tradeoff - if you want to do this kind of thing client-side - the secret sauce isn't so secret.
Is it ironic or missing a /s? I can't really tell here.
If you use one of those AI model as a basis for your AI model the real danger could be that the owners of the originating data are going after you at some point as well.
If you actually expected anything to be open about OpenAI's products, please get in touch, I have an incredible business opportunity for you in the form of a bridge in New York.
So if a model is copyrighted, you should still be able to use it if you generate a different one based on it. I.e. copyright laundry. I assume this would be similar to how fonts work. You can copyright a font file, but not the actual shapes. So if you re-encode the shapes with different points, that's legal.
But, I don't think a model can be copyrighted. Isn't it the case that something created mechanically can't be copyrighted? It has to be authored by a person.
I find it weird that so many hackers go out of their way to approve of the legal claims of Big AI before it's even settled, instead of undermining Big AI. Isn't the hacker ethos all about decentralization?
1. the current, unproven-in-court legal understanding, 2. standard disclaimer to cover OP's ass 3. tongue-in-cheek reference to the prevalent argument that training AI on data, and then offering it via AI is being a parasite on that original data
Prevalent or not, phrased this way it's clear how nonsense it is. The data isn't hurt or destroyed in the process of being trained on, nor does the process deprive the data owners from their data or opportunity to monetize it the way they ordinarily would.
The right terms here are "learning from", "taking inspiration from", not "being a parasite".
(Now, feeling entitled to rent because someone invented something useful and your work accidentally turned out to be useful, infinitesimally, in making it happen - now that is wanting to be a parasite on society.)
Getting back to "learning from", I think the issue is not the learning part, but the recreation part. AI can churn content to orders of magnitude higher than before, even in the age of Fiverr and other tools-opportunities. This changes the dynamics of the interaction, because previously, it took someone tens of hours to create something, now it takes AI minutes. That is not participating in the same playing field, it's absolutely dominating it, completely changing it. That is something to have feelings about, especially if one's livelihood is also impacted. Data is not destroyed, and neither is its ownership, but people don't usually want the exact thing, they are content with a good enough thing, and this takes away a lot of power from the artists, whose work is the lifeblood of artistic AI in the first place.
So I don't think it's as nonsense as you state it. But I do understand that it's not cut and dry the other way around either. Gatekeeping culture is definitely not a humane thing to do. Culture comes and goes, intermingles, inspires and changes all the time, and people take from it and add to it all the time. Preserving copyright perfectly would neuter it, and slant the landscape even more towards the already powerful.
The simple fact that models are released under license, which may or may not be free, imply that it is intellectual property. You can't license something that is not intellectual property.
It is a standard disclaimer, if you disagree, talk to your lawyer. The legal situation of AI models is such a mess that I am not even sure that a non-specialist professional will be of great help, let alone random people on the internet.
I understand that its not very clear if the neural net and its weights & biases are considered as IP, I personally think that if some OpenAI employee just leaks GPT-4o it isn't magically public domain and everyone can just use it. I think lawmakers would start to sue AWS if they just re-host ChatGPT. Not that I endorse it, but especially in IP and in law in general "judge law" ("Richterrecht" in german) is prevalent, and laws are not a DSL with a few ifs and whiles.
But it is also a "cover my ass" notice as others said, I live in Germany and our law regarding "hacking" is quite ancient.
I.e. the first one concerns you, the other is none of your business.
Simply showing up to court wearing a tshirt that says "what she said" probably wouldn't fly, but I like to imagine that any arguments made by them about their copyrights would be equally true of my copyrights.
At this point I'm mostly wondering if "you ripped me off first" is a viable legal defense to copyright battles where it's unclear if either party is distributing the works of the other. One thing is for sure though, if I were to do this as an individual, the discovery process would be much more expensive for them than me.
(1) That they are actually protected by copyright in the first place, or
(2) That the particular act described does not fall into an exception to copyright like fair use, exactly as many model creators assert that the exact same act done with the materials models are trained on does, rendering the restrictions of the license offered moot for that purpose.
An example for legal reference might be convolution reverb. Basically it's a way to record what a fancy reverb machines does (using copyrighted complex math algorithms) and cheaply recreate the reverb on my computer. It seems like companies can do this as long as they distribute protected reverbs separately from the commercial application. So Liquidsonics (https://www.liquidsonics.com/software/) sells reverb software but puts for free download the 'protected' convolution reverbs specifically the Bricasti ones in dispute (https://www.liquidsonics.com/fusion-ir/reverberate-3/)
Also, while a SQL server can be copyright protected, a SQL database is not given copyright protection/ownership to the SQL server software creators by extension of that.
The author only mentions APK for Android, but what about iOS IPA? Is there an alternative method for handling that archive?
But the Objective C code is actually compiled, and decompilation is a lot harder than with the JVM languages on Android.
My next article will be about CoreML on iOS, doing the same exact thing :)
Can't wait - thanks for writing it up!
As for the copyright treatment? As far as I know it’s a bit up in the air at the moment. I suspect that the major frontier vendors would mostly contend that training data is fair use but weights are copyrighted. But that’s because they’re bad people.
But there is a group of people, growing daily in influence, who utterly reject such principles as either worthy or useful. This group of people is defined by the ego necessary to conclude that when the stakes are this high, the decisions should be made by them, that the ends justify the means on arbitrary antisocial behavior (c.f. the behavior of their scrapers) as long as this quasi-religious orgasm of singularity is steered by the firm hand that is willing and able to see it through.
That doesn’t distress me: L Ron Hubbard has that.
It distresses me that HN as a community refuses to stand up to these people.
Basically its just a synthetic loop of using a previously developed SOTA (was) model like GPT-4 to train your model.
This can produce models with seemingly similar performance at a smaller size, but to some extent, less bits will be less good.
I'm not quite sure I understand the firebase feature btw. From the docs, it's pretty much file storage with a dedicated API? I suppose you can use those models for inference in the cloud, but still, the storage API seems redundant.
This works, just like ChatGPT works, but has the downside of 1. You have to pay the computing for every inference 2. Your users can't access it offline 3. Your users will have to use a lot of data from their mobile network operator. 4. Your inference will be slower
And since SeeingAI infers the model every second, your and your customers bill will be huge.
In any case, yeah you can not download the model to the device at all, but then you have to deal with the other angle - making sure the endpoint isn't abused.
Maybe a hybrid approach would work - infer just part of the model (layers?) on the cloud, and then carry on the inference on the device? I'm not familiar with how AI models look like and work like exactly, but I feel like hiding even a tiny portion of the model would make it not usable in practice
For models which use a lot of input nodes, a lot of "hidden layers" and in the end just perform a softmax this may get infeasible because of the amount of data you would have to transfer.
You may have inspired a second article :)
The file is only 7.7kb, so it couldn't contain many weights anyways.
Unless all LLM are a ruthless parody of human intelligence, which they may be, the legal issues will continue.
most other places use fair dealing which is more restrictive https://en.m.wikipedia.org/wiki/Fair_dealing
- Addenda -
For the interested parties, the law states the following [0].
Notwithstanding the provisions of sections 17 U.S.C. § 106 and 17 U.S.C. § 106A, the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include:
1. the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
2. the nature of the copyrighted work;
3. the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and
4. the effect of the use upon the potential market for or value of the copyrighted work.
So, if you say that these factors can be flexed depending on the defendant, and can be just waved away to protect the wealthy, then it becomes something else, but given these factors, and how damaging this "fair use" is, I can certainly say that training AI models with copyrighted corpus is not fair use in any way.
Of course at the end of the day, IANAL & IANAJ. However, my moral compass directly bars use of copyrighted corpus in publicly accessible, for profit models which undermine many people of their livelihoods.
From my perspective, people can whitewash AI training as they see fit to sleep sound at night, but this doesn't change anything from my PoV.
[0]: https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors
> In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include:
> * the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
> * the nature of the copyrighted work;
> * the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and
> * the effect of the use upon the potential market for or value of the copyrighted work.
Another weird carve-out for copyright law in the US: parody. Honestly, I don't know if other jurisdictions allow parody in the same protected manner.
Germany: https://www.gesetze-im-internet.de/urhg/__51a.html (Though this explicit carve-out is a recent development, though generally speaking parodies were allowed even under the previous version of the law.)
Nope :-), just a normal citizen, but sometimes I am curious enough to look up a law, plus sometimes I need to refer to/look up some law in my day job as a civil engineer, too.
When you need to do that, it's not too hard to stumble upon the existence of that page through some web searches, plus the German Wikipedia often links to that page, too (as well as to some alternative platforms run by private entities, which sometimes provide some added value, e.g. buzer.de provides change history since 2006, too, other pages link relevant court decisions, etc. etc. – but gesetze-im-internet.de is the official page run by the federal government itself).
The nature of copyright and plagiarism boils down to paraphrasing, and so long as LLMs sufficiently paraphrase the content it's an open question whether it's copyright infringement and requires new law/precedent.
So the fact they are earning money is a red herring unless they are reproducing the exact same content without paraphrasing (with exception to commentary). E.g. they can quote part of a work while commenting on it.
Where they have gotten into trouble with e.g. NYT afaik is when the LLM reproduced a whole article word for word. I think they have all tried hard to prevent the LLM from ever doing that to avoid that legal risk.
How many books can you read, understand and memorize in T time, and how many books an AI can ingest in the T time?
If we're down to paraphrasing, watch this video [1], and think again.
Many models, given that you ask the correct questions, reproduce their training set with great accuracy, and this is only prevented with monkey patching, IIUC.
So, it's still a big mess, even if we don't add copyrighted corpus to the mix. Oh, BTW, datasets like "The Stack" are not clean as they claim. I have seen at least two non-permissively licensed code repositories inside that dataset.
I am curious about the video, but am not compelled to spend 24 min watching it when you haven't summarized its thesis for me. The title of the video makes it seem adjacent at best to the points I was making. (Some automated flagging system =/= actual law)
as an example: saying “i really like james holden’s inheritors album for the rough and dissonant sounds” isn’t covered by copyright.
if i reproduced it verbatim using my mouth, or created a derived work which is noticeably similar to the original, that’s a different question though.
in your example, a derivative work example could be akin to only quoting from the book for the audience and modifying a word of each quote.
“derived” works are always a grey area, especially around generative machine learning right now.
At the start of the tape, there was a copyright notice forbidding the VHS tape from being played at, amongst other places, schools.
Copyright rules are a strange thing.
If I can hire an employee who draws on knowledge they learned from copyrighted textbooks, why can't I hire an AI which draws on knowledge it learned from copyrighted textbooks? What makes that argument "wacky" in your eyes?
You're either going to get: it's a technological, infinitely scalable process, and the training data should be considered what it is, which is intellectual property that should be being licensed before being used.
...or... It actually is the same as human learning, and it's time we started loading these things up with other baggage to be attached to persons if we're going to accept it's possible for a machine to learn like a human.
There isn't a reasonable middle ground due to the magnitude of social disruption a chattel quasi-human technological human replacement would cause.
exactly how the new work is created is important when it comes to derivative works.
does it use a copy of the original work to create it, or a vague idea/memory of the original work’s composition?
when i make music it’s usually vague memories. i’d argue that LLMs have an encoded representation of the original work in their weights (along with all the other stuff).
but that’s the legal grey area bit. is the “mush” of model weights an encoded representation of works, or vague memories?
not sure how i feel about being reduced down to that as a human being.
Besides, “black box” just means that your internal mental life and cognitive mechanism is opaque to me. It’s not like I’m calling you a p-zombie.
> The point of copyright is to protect an exclusive right to copy, not the right to produce original works influenced by previous works.
> If an LLM produces original works that are influenced by the training data, that is not a violation of copyright.
This isn’t how LLM’s work though. Samples are just that, literal samples that are copied from one work to another verbatim. LLM’s use training data to construct a predictive model of which tokens follow each other. You probably could get an LLM to use samples deliberately if you wanted to, but this isn’t how they typically work.
Regardless, at that point you’re just evaluating the claim of copyright infringement based on the nature of the work itself, which is exactly what I’m advocating, versus presuming that all LLM output is necessarily copyright infringement if any copyrighted material was used in training.
Can you help me to understand the term "chattel" as you used it? I never heard the term before I read your post, and I needed to Google for it: <<
(in general use) a personal possession.
(in law) an item of property other than freehold land, including tangible goods ( chattels personal ) and leasehold interests ( chattels real ). >>
Namely, a magic, technologically reproducible box that can be applied almost as effectively as a human hireling, but isn't a human hireling is near infinitely more desirable in a capitalist system since the blackbox is chattel, the hired human is not. The chattel has no natural rights, no claim to self sovereignty, and is an asset that is legally extant by virtue of the fact it is owned by the owner.
Chattel that are flexible enough to replace the legal burdens incurred by hiring a human to do the same job, will naturally be converged upon due to the capitalistic optimation function of minimizing unit input cost for output over dollars and potential dollars as expressed through legal exposure.
Imagine you had two human like populations. One made of plastic that aren't considered humans but property. I.e. are chattel. Then you have a bunch of people with all the baggage that comes with it.
Hiring people/employing people is hard. Particularly in the U.S. and other jurisdictions where a great deal of responsibility for actually implementing regulations/ taxation/immigration and such is tacked onto being an employer/being able to hire.
As the gap between the capability of the chattel population closes in on the human population, the more economic and workload sense it makes for the system to improve the chattel population under our current optimization strategy, (given no pre-emptive work to cut off externality dumping). Humans are messy and complicated to work with. Often unpredictable. Chattel are easy to account for; especially when combined with "technical restraints". You have to fundamentally engage in negotiation with another human being to get them on board with working for you. You buy the chattel, and that's that. The chattel has no grounds to refuse service. Socially speaking, we don't even recognize it's outputs as carrying any social weight, or resistance as anything but malfunctions.
Economics is the science around using access to resources as a means to get other people to work with you. Being chattel means you can cut out entirely all that complexity. You are resource. Not people.
Unironically, we need to have an answer to whether or not we are going to consider a sufficiently complex function imitator as something that requires a classification above "chattel" or controls around how we apply it in order to not self-destruct the economic equilibria in which we purport to exist. Because all it takes is removing or sufficiently obstructing the flow of value down from individuals who accrete the most of these wunder-chattel to render things so top heavy, most of the constraints/invariants of our socioeconomic systems as we know them become invalidated.
That does not bode well for anyone.
Unlike a person, an large language model is product built by a company and sold by a company. While I am not a lawyer, I believe much of the copyright arguments around LLM training revolve around the idea that copyrighted content should be licensed by the company training the LLM. In much the same way that people are not allowed to scrape the content of the New York Time website and then pass it off as their own content, so should OpenAI be barred from scraping the New York Times website to train ChatGPT and then sell the service without providing some dollars back to the New York Times.
The other is a person learning from a copyrighted textbook in the legally protected manner, and whom and use the textbook was written for.
"Can you elaborate on how it's not comparable?"
The process of individual people interacting with their culture is a vastly different process than that used to train large language models. In what ways to you think these processes have anything in common?
"It seems obvious to me that it is -- they both learn and then create -- so what's the difference?"
This doesn't seem obvious to me (obviously)! Maybe you can argue that an LLM "learns" during training, but that ceases once training is complete. For sure, there are work-arounds that meet certain goals (RAG, fine-tuning); maybe your already vague definition of "learning" could be stretched to include these? Still, comparing this to how people learn is pretty far-fetched. AFAICT, there's no literature supporting the view that there's any commonality here; if you have some I would be very interested to read it. :-)
Do they both create? I suspect not; an LLM is parroting back data from it's training set. We've seen many studies showing that tested LLMs perform poorly on novel problem sets. This article was posted just this week:
https://news.ycombinator.com/item?id=42565606
The court is still out on the copyright issue, for the perspective of US law we'll have to wait on this one. Still, it's clear that an LLM can't "create" in any meaningful way.
And so on and so forth. How is hiring an employee at all similar to subscribing to an OpenAI ChatGPT plan? Wacky indeed!
But if they're learning from the same kinds of materials, and producing the same kind of output, then obviously the comparison can be made. And your idea that LLM's don't create seems obviously false.
So I have to conclude the two seem comparable, and someone would have to show why different legal principles around copyright ought to apply, when it's a simple question of input/output. Why should it matter if it's a human or algorithm doing the processing, from a copyright perspective? Nothing "wacky" about the question at all.
Strangely like the situation itself.
The question is just looked to how can we guarantee a model is influenced rather than memorising an input?
And then is a human who is influenced simply relying on a faulty or less than perfect memory?
LLMs are not humans. They’re essentially a probabilistic compression algorithm (encode data into model weights/decode with prompt to retrieve data).
Depending on how much music you've listened to, you very well may have "downloaded terabytes" of it into your brain. Your argument is specious.
i listen to it on apple music.
i pay money to apple for this.
some of that money that i pay to apple goes to the rights holders of that music for the copying and performance of their work through my speakers.
that’s a pretty big difference to how most LLMs are trained right there! i actually pay original creators some money.
-
i am a human being. you cannot reduce me down to some easy information theory.
an LLM is a tool. an algorithm. with the same random seed etc etc it will get the same results. it is not human.
you put me in the same room as yesterday i’ll behave completely differently.
-
i have listened to way more than terabytes of music in my life. doesn’t mean i have the ability to regurgitate any of it verbatim though. i’m crap at that stuff.
LLMs seem to be really good at it though.
There is little mystery to how large language models function and it's clear that their output is parroting back portions of their training data, the quality of output degrades greatly when novel input is provided. Is your argument that people fundamentally function in the same way? That would be a bold and novel assertion!
If this were true, then you would be able to identify the specific work being "parroted" and you'd have a case for copyright infringement regardless of whether it was produced by an LLM at all. This isn't how LLMs work though. For instance, if an LLM's training data includes the complete works of a given author and then you prompt the LLM to write a story in the style of that author, it will actually write an original story instead of reproducing one of the stories in its training corpus. It won't be particularly good but it will be an original work.
It also isn't obvious whether or not, or to what degree, LLM training works differently from human learning. You yourself acknowledged that there are "many open questions" about how human learning works, so how can you be so confident that it's fundamentally different? It doesn't matter anyway because you can still apply the exact same standards to LLM output to judge whether it infringes copyright that you would to something that was produced by a human being.
Although with the 'good new everyone, we built the torment nexus' trajectory of AI my guess is at this point AI companies would just incorporate actual human brains instead of digital storage if that was the requirement.
And before you knee-jerk "it's a compression algo!", I invite you to archive all your data with an LLMs "compression algo".
Would Einstein would have been possible without Newton?
But as things are, the megacorps are training their LLMs on the commons while asserting "intellectual property" rights on the resulting weights. So, fuck them, and cheers to those who try to do something about this state of affairs.
And is technically copyright infringement outside fair use exceptions.
Neither am I, yet, I am still capable of reproducing copyrighted works to a level that most would describe as illegal.
> And before you knee-jerk "it's a compression algo!"
It's literally a fundamental part of the technology so I can't see how you call it a "knee jerk." It's lossy compression, the same way a JPEG might be, and simply recompressing your picture to a lower resolution does not at all obviate your copyright.
> I invite you to archive all your data with an LLMs "compression algo".
As long as we agree it is _my data_ and not yours.
Compression yes, but this is co-mingling as well. The entire corpus is compressed together, which identifies common patterns, and in the model they are essentially now overlapping.
The original document is represented statistically in the final model, but you’ve lost the ability to extract it closely. Instead you gain the ability to generate something statistically similar to a large number of original documents that are related or are structurally similar.
I’m just commenting, not disputing any argument about fair use.
Is that really true. Is the law settled in this area. Is it the same everywhere or does it vary from jurisdiction to jurisdiction.
Paper: https://arxiv.org/pdf/2204.03738
Code: https://github.com/microsoft/banknote-net Training data: https://raw.githubusercontent.com/microsoft/banknote-net/ref...
model: https://github.com/microsoft/banknote-net/blob/main/models/b...
Kinda easier to download it straight from github.
Its licenced under MIT and CDLA-Permissive-2.0 licenses.
But lets not let that get in the way of hating on AI shall we?
Can you please edit this kind of thing out of your HN comments? (This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.)
It leads to a downward spiral, as one can see in the progression to https://news.ycombinator.com/item?id=42604422 and https://news.ycombinator.com/item?id=42604728. That's what we're trying to avoid here.
Your post is informative and would be just fine without the last sentence (well, plus the snarky first two words).
If the last sentence were explicit rather than implied, for instance
This article seems to be serving the growing prejudice against AI
Is that better? It is still likely to be controversial and the accuracy debatable, but it is at least sincere and could be the start of a reasonable conversation, provided the responders behave accordingly.
I would like people to talk about controversial things here if they do so in a considerate manner.
I'd also like to personally acknowledge how much work you do to defuse situations on HN. You represent an excellent example of how to behave. Even when the people you are talking to assume bad faith you hold your composure.
I would also like to point out that this is a fine tuned classifier vision model based on mobilenetv2 and not an LLM.
The same method/strategy applies to closed tools and models too, although you should probably be careful if you've handed over a credit card for a decryption key to a service and try this ;)
I was rather interested in the process of instrumenting of TF to make this "attack" scalable to other apps.
That’s not true, is it? It would be a copyright violation to distribute an extracted model, but you can do what you want with it yourself.
Additionally, the debate around the sources companies use to train their models remains unresolved, raising ethical and legal questions about data ownership and consent.
If that is the law, it is a defect that we need to fix. Laws do not come down from heaven in the form of commandments. We, humans, write laws. If there is a defect in the laws, we should fix it.
If this is the law, time shifting and format shifting is unlawful as well which to me is unacceptable.
Disclaimer: As usual, I anal.
I don't think YouTube intended regular uploads to have DRM, if only because they support Creative Commons metadata on uploads, and Creative Commons specifically forbids the use of technical protection measures on CC-licensed content[0]. On a less moralistic note, applying encryption to all YouTube videos would be prohibitively expensive because DRM vendors charge $$$ for the tech.
But the RIAA wants DRM because, well, they don't want people taking what they have rightfully stolen. So YouTube engineered a weak form of URL obfuscation that would only stop very basic scrapers[1]. DMCA 1201 doesn't care about encryption or obfuscation, though. What it does care about is if something was intended to stop copying, and if so, if the defendant's product was designed to defeat that thing.
There's an interesting wrinkle in DMCA 1201 in that merely being able to defeat DRM does not make something illegal. Defeating DRM has to be the tool's only function[2], or you have to advertise the tool as being able to defeat DRM[3], in order to actually violate DMCA 1201. DRM vendors usually resort to encryption, because it makes the circumvention tools specialized enough that they have no other purpose and thus fall afoul of DMCA 1201. But there's nothing stopping you from using really basic schemes (ROT-13 your DVDs!) and still getting to sue for 1201.
Going back to the AI ripping question, this blog post is probably not in and of itself a circumvention tool[4], but anyone implementing it is very much making circumvention tools, which are illegal to distribute. Circumvention itself is also illegal, but only when there's an underlying copyright infringement. i.e. you can't just encrypt something that's public domain or uncopyrightable and sue anyone who decrypts it.
So the next question is: is AI copyrightable? And can you sue for 1201 circumvention for something that is fundamentally composed of someone else's copyrighted work that you don't own and haven't licensed?
[0] Additionally, there is a very large repository of CC-BY music from Kevin MacLeod that is used all over YouTube that would have to be removed or relicensed if the RIAA were to prevail on this case.
I have no idea if Kevin actually intends to enforce the no-DRM clause in this way, though. Kevin actually has a fairly loose interpretation of CC-BY. For example, nobody attributes his music correctly, either the way the license requires, or with Kevin's (legally insufficient) recommended attribution strings. He does sell commercial (non-attribution) licenses but I've yet to hear of any enforcement actions from him.
[1] To be clear, without DRM encryption, any video can be ripped by hooking standard HTML5 video APIs using an extension.
[2] Things with "limited commercial purposes" beyond breaking DRM may also be construed as circumvention tools under DMCA 1201.
[3] My favorite example: someone tried selling a VGA-to-composite adapter as a way to copy movies off Netflix. That is illegal under DMCA 1201.
[4] To be clear, this is NOT settled law, this is "get sued and find out if the Supreme Court likes you that day" law.
> Circumventing a copy-prevention system without a valid exemption is a crime, even if you don't make unlawful copies.
Yep, this is the DMCA section 1201. Late '90s law in the US.
> Copyright covers the right to make copies, not the right to distribute
This is where I got confused. Copyright covers four rights: copying, distribution, creation of derivative works, and public performance. So I'm not sure what you were getting at with the copy/distribute dichotomy.
But here's a question I'm curious about: Can DMCA apply to a copy-protection mechanism that's being applied to non-copyrightable work? Based on my reading of https://www.copyright.gov/dmca/:
> First, it prohibits circumventing technological protection measures (or TPMs) used by copyright owners to control access to their works.
That's not the letter of the law, but an overview, but it does seem to suggest you can't bring a DMCA 1201 claim against someone circumventing copy-protection for uncopyrightable works.
> Whether or not model weights are copyrightable remains an open question.
And this is where the interaction with the wording of 1201 gets interesting, in my (non-professional) opinion!
> No person shall circumvent a technological measure that effectively controls access to a work protected under this title.
The inclusion of “work protected under this title” makes it clear in the law, though I doubt a judge would rule otherwise without that line. (Otherwise, I’d wonder if I could claim damages that Google et al. are violating the technological measures I’ve put in place to protect the specificity of my interests, because it wouldn’t matter that such is not protected by copyright law.)
Also not an attorney, for what it’s worth.
> (A) to “circumvent a technological measure” means to descramble a scrambled work, to decrypt an encrypted work, or otherwise to avoid, bypass, remove, deactivate, or impair a technological measure, without the authority of the copyright owner
In this case there is no copyright owner.
But then again, knowing the number is a far cry from using that number to circumvent DRM
> Outside the Internet and the mass media, the key has appeared in or on T-shirts, poetry, songs and music videos, illustrations and other graphic artworks, tattoos and body art, and comic strips.
Using the encryption key to decrypt the data on a DVD is illegal “circumvention” per DMCA 1201, if it’s done without authorization from the copyright owner of the data on the DVD. If it were really illegal to simply publish the key on a website, then printing it on clothing that they sold instead would not be a viable loophole.
I’m glad it is still referred to as a controversy that they were issuing cease and desist letters for publishing information when the actual crime they had in mind, which was not alleged in the letters, is using the information to decrypt a DVD.
https://en.m.wikipedia.org/wiki/Chamberlain_Group,_Inc._v._S...
so this is where it all goes in several years, if i were the gov.
Can you copyright a set of coefficients for a formula? In the sense of a JPEG it would be considered that the image being reproduced is the thing that has the copyright. Being the first to run the calculations that produces a compressed version of that data should not grant you any special rights to that compressed form.
An AI model is just a form of that writ large. When the models generalize and create new content, it seems hard to see how that either the output or the model that generated it could be considered someone's property.
People possess models, I'm not sure if they own them.
There are however billions of dollars at play here and enough money can buy you whichever legal opinion you want.
If companies train on data they don't own and expect to own their model weights, that's hypocritical.
Model weights shouldn't be copyrightable if the training data was pilfered.
But this hasn't been tested because models are locked away in data centers as trade secrets. There's no opportunity to observe or copy them outside of using their outputs as synthetic data.
On that subject, training on model outputs should be fair use, and an area we should use legislation to defend access to (similar to web scraping provisions).
Perhaps 'strongly suggestive' isn't enough.
https://harvardlawreview.org/blog/2024/04/nyt-v-openai-the-t...
https://www.publishersweekly.com/pw/by-topic/industry-news/p...
Given that everything -- including this comment -- is copyrighted unless it is (1) old or (2) deliberately put into the public domain, this is almost certainly true.
> User Content Transmitted Through the Site: With respect to the content or other materials you upload through the Site or share with other users or recipients (collectively, “User Content”), you represent and warrant that you own all right, title and interest in and to such User Content, including, without limitation, all copyrights and rights of publicity contained therein. By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies a nonexclusive, worldwide, royalty free, fully paid up, transferable, sublicensable, perpetual, irrevocable license to copy, display, upload, perform, distribute, store, modify and otherwise use your User Content for any Y Combinator-related purpose in any form, medium or technology now known or later developed. However, please review the Privacy Policy located here for more information on how we treat information included in applications submitted to us.
> You acknowledge and agree that any questions, comments, suggestions, ideas, feedback or other information about the Site (“Submissions”) provided by you to Y Combinator are non-confidential and Y Combinator will be entitled to the unrestricted use and dissemination of these Submissions for any purpose, without acknowledgment or compensation to you.
Many people seem to think that no licence = public domain, but it's still under strong copyright protection. This is the point of things like the Unlicense license.
Yes you could, unless you agreed to forum terms that said otherwise, fair use aside. Its the same in most jurisdictions
No, there have been lawsuits, and the data centers have not been fair game because whether or not the models were trained on copyright-protected works is not generally in dispute. Discovery only applies to evidence relevant to facts in dispute.
Its not hypocritical to follow a line of legal analysis whoch holds that copying material in the course of training AI on it is outside the scope of copyright protection (as, e.g., fair use in the US), but that the model weights resulting from the training are protected by copyright.
It maybe wrong, and it may be convenient for the interests of the firms involved, but it is not self-inconsistent in the way required for it to be hypocrisy.
Educated human beings are not protected by copyright, hence neither should trained AI models. Conversely, if a copyrightable work is produced based on work which itself is copyrighted, the resulting work needs the consent of the original authors of the prior work.
AI models can't have their ©ake and eat it.
(Such a model/statistical-summary, along with a dictionary, could be used to generate nonsensical texts which have similar patterns in terms of just word lengths.)
Should the resulting work be protected by copyright? I’m not entirely sure…
I guess one thing is, the specific numbers I obtain by doing this are not a consequence of any creative decision making on my part, which I think in some jurisdictions (I don’t remember which) plays a role in whether a work is copyrightable (I will use “copyrightable” as an abbreviation for “protected by copyright”. I don’t mean to imply a requirement that someone specifically registers for copyright.). (Iirc this makes it so phone books are copyrightable in some jurisdictions but not others?)
The particular choice of statistical analysis does seem like it may involve creative decision making, but that would just be about like, what analysis I describe, and how the numbers I publish are to be interpreted, not what the numbers are? (Analogous to the source code of an ML model, not the parameters.)
Here is another question: suppose there is a method of producing a data artifact which would be genuinely (and economically) useful, and which does not rely on taking in any copyrighted input, but requires a large (expensive) amount of compute to produce, and which also uses a lot of randomness so that the result would be different each time it was done (but suppose also that there isn’t much point doing it multiple times at the same scale, as having two of this kind of data artifact wouldn’t be much more valuable than having one).
Should such data artifacts be protected by copyright or something like it?
Well, if copyright requires creative human decision making, then they wouldn’t be.
It seems like it would make sense to want it to be economically incentivized to create such data artifacts of higher sizes (to a point of course. Only as much as is justified by the value that is produced by them being available.) .
If such data artifacts can always be distributed without restriction, then ones that are publicly available would be public goods, and I guess only ones that are trade secrets would be private goods? It seems to me like having some mechanism to incentivize their creation and being-eventually-freely-distributed would be beneficial?
But maybe copyright isn’t the best way to do that? Idk.
This has already been settled hasn't it? Don't companies have to introduce 'flaws' in order for data sets to be 'protected'? Just compiled lists of facts can't be protected. Which is why things like election result companies having to rely on NDAs and not copyright protections to protect their services on election night.
No, flaws are generally introduced to make it easier to detect copies; if multiple flawless reference works covering the same data (road maps of the same region, for instance) exist, each is copyrightable without flaws to the extent any would be with flaws, but you can't prove that someone else copied yours without permission if copying any of the others would give the same result. With flaws, gou can attribute the source that was copied more easily, but this isn't about being legally protected but about the practicality of enforcing that protection.
Exactly. It would be patents, not copyright.
The selection and structuring of the training set may involve sufficient creativity to be copyrightable (as demonstrated by the existence of “compilation copyrights”), even if it is largely or even entirely composed of existing works, the statistical analysis part doesn't have to be the source of the creativity.
No one training (foundation) models makes that fair use argument by analogy, they make arguments that addresses the specific statutory and case law criteria for fair use (abd frequently focus on the transformative character of the use); its true that the analogy to a learning human argument is frequently made in internet fora by AI enthusiasts who aren't the people training models on vaat scraped datasets. That argument is bunk for a number of reasons, but most critically the fact that a human learning from material isn’t fair use, because a human brain isn’t treated as a fixed medium, so learning in a human brain isn’t legally a copy or derivative work that would violate copyright without the fair use exception, so it's not a use to which fair use analysis even applies, so you can't argue anything is fair use by analogy to that. But its moot to any argument for hypocrisy by the big model makers, because they aren’t using that argument to start with.
To substitute either party with a computer system and assume that the existing law still makes sense may be assuming too much.
In my unusually well-informed on copyright but not a lawyer opinion, without any new legislation on the subject, I suspect that the most likely scenario for intellectual property rights surrounding AI is that using other people's works for training probably falls under fair use, since it's extremely transformative (an AI that makes text and a textual work are very different things) and it's extremely difficult to argue that the AI, as it exists today, directly impacts the value of the original work.
The list of what traing data to use is probably protected by copyright if hand-picked, otherwise just whatever web-crawler they wrote to gather it.
The AI models, as in, the inference and training applications are protected by copyright, like any other application.
The architecture of a particular AI model can be protected by patents.
The weights, as the result of an automated process, are probably not protected by copyright.
Object code is the result of an automated process and is covered by the copyright on the source code.
Compilations are covered by copyright separate from that of the individual works, and it is arguable that a training set would be covered by a compilation copyright, and the result of applying an automated training processs to it would remain covered by that copyright.
The architecture and the weights in a model are the secret process used to make a commercially valuable output. It makes the most sense to treat them as a trade secret, in a court of law.
IANAL, but I have serious doubts about the applicability of current copyright law to existing AI models. I imagine the courts will decide the same.
There is an argument also that LLMs are derivative works of the training data, which I'm somewhat sympathetic to, though clearly there's a difference and lots of ambiguity about which contributions to which weights correspond to any particular source work.
Again IANAL, and this is my opinion based on reading the law & precedents. Consult a real copyright attorney for real advice.
- Models are processes/concepts, thus not copyrightable, but are subject to trade secret law, contract and license restrictions, patents, etc.
- Concrete implementations may be copyrighted like any code.
- Parameters are "facts", thus not copyrightable, but are similarly subject to trade secret and contract law.
IANAL, not legal advice, yadda yadda yadda.
AI models are generally regarded as a company’s asset (like a customer database would also be), and rightly so given the cost required to generate one. But that’s a different matter entirely to copyright.
(OTOH, whether models, as the output of a mechanical process, are subject to copyright is a matter of some debate. The firms training models tend to treat the models as if they were protected by copyright but also tend to treat the source works as if copying for the purpose of training AI were within a copyright exception; why each of those positions is in their interest is obvious, but neither is well-established.)
Laundering IP. FTFY.
If weights and biases contained in "AI models" are prorietary, then for one model owner to detect infingement by another model owner, it may be necessary to download and extract.