2,337 karma · joined July 21, 2016
Until this is decided, likely by a long series of trials in high courts, or passage of law through the appropriate legislative bodies (and then likely after many challenges to it) will GDPR and CCPA be able to apply to these language models.
So no this is not currently a violation of any law.
I'd love to see a video exploring the rack and showing the first time setup.
What I really want is for my synchronous and asynchronous code to be a lot more transparent and not force my hand depending on whether I'm calling from a synchronous or asynchronous context. I know this is a really difficult ask because of the lack of knowledge of what runtime to use when calling an async task in the context of a synchronous call...
Its the coloring problem I ultimately hate fighting against. There are definitely times that I care about it, long running blocking synchronous calls and getting a handle on a Future are really the only ones I can think of and they tend to be the exception not the norm.
I think this one needs to go back to the drawing board.
What I can recommend is to take the coffee tasting approach. Don't try and test and evaluate individual responses, instead lock the seed used in generation, and use the same prompt for two different runs. Change one variable and do a relative comparison of the two outputs. The variables probably worth testing for you off the top of my head:
* Choice of models and/or tunes
* System prompts
* Temperature of the model against your queries
* Threshold for similarity for document inclusions (you only want relevant documents from your RAG, set it too low and you'll get some extra distractions, too high and useful information might be left out of the context).
If you setup a system to track the comparisons either automatically or by hand that just indicates which side of the change worked better for your use case, and test that same change for a bunch of different prompts you should be able to tally up whether the control or change was more preferred.
Keep those data points! The data points are your bench log and can be invaluable later on for anything you do with the system to see what changed in aggregate, what had the most outsized impact, etc and can guide you to build useful tooling for testing or finding existing solutions out there.
Without details of what specific data a language model was trained on, there is no way to differentiate it having retrieved that name from a review of the book that was used with permission from the author (which is their own copyright and fair use on its own) or from the books itself (which would be copyright infringement).
If those records don't exist, the only mechanism you have left is to try and get the language model to spit out sufficient verbatim text from the source material to cross the fair use threshold. This doesn't work in the case of "obvious extrapolation" either which is a whole other defense that could be used depending on what that body produced was (If knowing only that the main character is named Harry, who is a wizard going to a wizarding school in the British countryside is required to produce a couple of close looking paragraphs, you need a much closer match with the original text over those two paragraphs for it to be infringing).
Let's ignore ongoing feature creep and development in projects for now (I'll address this in a second). As the core software becomes functional, gains users, has bugs fixed, the threat model of the software needs to change and grow. Generally software in this state gets progressively more stable and in that stability you can see the tiny ripples of very small edge cases.
For software where security is paramount and has an uncountable number of users protecting uncountable bytes of data like SSH, the tiny ripples of extremely esoteric attacks become a reasonable thing to address as a feature improvement. Do I need this feature for my personal threat model or even the threat model of my company? No probably not... But it doesn't hurt to have it either.
This feature raises the cost of attacking the software, and may dissuade different attacker from even trying. Most of practical security is more about making attacks expensive enough that they attacker will have to spend more than the value of what they would obtain if they were able to break into it and this change helps keep that high watermark well above those thresholds.
For software that does have ongoing feature creep this process is a lot slower but does seem to happen for opensource projects specifically that can survive their infancy sufficiently long enough to become foundational components like SSH and Curl. I've never seen this level stability from closed source software.
So what does this actually protect against? Who will this benefit? Mostly people in censored countries and companies. This removes the last piece of information that can be used to block HTTPS traffic based on the site your visiting without being a party to the exchange.
I still think DoH is hot garbage and the way it has been implemented across browsers is an atrocity. It's actively harmful to security even if the spirit is in the right place. I've got no complaints about ECH.
The easiest thing to do here, is to simply not associate those sessions with a particular user. Even if your user accounts are tied to specific PII for essential purposes of your app. As long as the tracking data is not connected to that identifier, does not log any PII data on it own, and is not shared with third parties you do not need consent.
One quick edit: Be careful with collecting errors, its easy for backtraces to include application specific data including any PII you might have which will tie that session back to a specific user and becomes a violation.
Just don't collect PII beyond was is absolutely essential for your application, and don't share it with third parties. Bam you don't have to get consent. Knowing what classifies as PII is still a hard problem because its full of so many conditionals. Email is not PII unless you have some part of their name for example and it counts if your company receives an email from that person that includes their name in the From field.
All the cookie banners out there are designed to make people weary of them into just accepting the previous practices. It's malicious compliance.
I'm not sure how experienced you are in the field but there are kind of two levels of fine-tuning, full fine-tuning (update all the weights of the model, usually requires 2-3x the memory required for inference). This allows you to change and update the knowledge contained inside the model.
If the model has sufficient understanding already of the content of the task and you want to change how it responds, such as to a specific output format, "personality" or "flavor" of output, or to have it already know the kind of task its performing without including those details in the prompt I would go with parameter efficient fine-tuning.
If you're looking to do a one-off train for a model, you might be able to get away with doing it in something like this: https://github.com/oobabooga/text-generation-webui Very easy to use project but it really doesn't allow for the kind of metrics, analysis, or professional grade hosting you'll want.
vLLM can help with the hosting and is really solid once you have the models fine-tuned we tried that at first but its core architecture simply wouldn't work for what we were trying to do which is why we went fully in-house.
Once you get into a lot of fine-tuning, you're probably going to want to do it directly in pytorch or the equivalent for your language of choice. A good resource for seeing how people do this is actually opensource models published on hugging face. Look for some LoRA models, or fine tunes similar to what you'd like. A lot of people publish their training code and datasets on GitHub which can be very useful references.
Right now I'd recommend llama2 as a base model for most general language model based tasks if you don't cross their commercial use threshold (which is very very generous).
Hope this helps!
The Meta AI integration is also a massive turn off to me. To each their own, but I want to keep the smart bits mostly in the smart phone. The assistants so far have been anything but assistive on my devices and usually both get in the way (since for a while everyone was really pushing you to use them, only Google seems to have not gotten that particular message) and are privacy nightmares in their own right.
(For clarification the users preference in my original post, is about interactive users preferring to see a stream of tokens coming in rather than waiting for the entire request to complete and having it show up all at once. The performance of that sets the expectation for the time of non-interactive responses.)
As for performance, I'm generally seeing 40-50 tokens/sec per model on a Tesla family Nvidia GPU but I keep multiple models loaded and active at a time so that estimate is probably a bit low for overall throughput (I also realized that our monitoring doesn't have any cumulative GPU token rate metrics just now thanks to this question hahah).
Interesting anecdote others may be interested in... I'm rate limiting the output from our streaming API to 8 tokens/sec to artificially smooth out front-end requests. Interactive users will wait and even prefer seeing the stream of the response, and non-interactive users tend to base their performance expectations on the what the streaming API does. It's kind of sneaky but I'm also artificially slowing down those API requests.
The other side of the coin, the other option we shouldn't really rely on but we can, is entirely blocked by the way these vendors operate. Android has had its fragmentation issues because the vendors aren't using the stock Android, and aren't upstreaming the drivers and tweaks needed to actually run Android on their hardware. There is usually an economic incentive to not do so (if a custom ROM can be made for their phone, they won't get the ad and deal revenue from the garbage they've preloaded into the phone).
Some do this better than others and there are custom ROMs out there that allow you to update, but custom ROMs are a horrible solution for most customers. It's really only accessible to technical people that have the time and energy to put into it and maintain it directly.
The same thing is true for most IoT devices, the drivers aren't upstreamed, the source is closed, third parties can't fix the software or use it outside the designed ecosystem and the manufacturer doesn't have an economic incentive to give you any time after you've purchased their product.
A lot of manufacturers of devices are mostly white labeling reference designs with a small modification for an extra sensor or other relatively small benign change to the electrical schematic and they'll correspondingly use the reference software for that design which hasn't been updated in 20 years since the reference design was created. Those designs usually have the same problem, the manufacturer of the chip had some devs hand edit a "stable" version of the Linux kernel outside of any source control or public repository and shipped a binary with custom drivers (probably not up to the normal quality expected of code merged into the kernel either).
I think that last one is really the sticking point. The fractured Android ecosystem isn't great but its fracturing issues have been much better in the past 6-8 years than it was up till about Android 8 (based on my recollections) so is kind of an outdated example at this point.
If manufacturers of custom ICs kept reference designs up to date, or upstream their drivers into mainline Linux that becomes the economically cheapest, lowest effort, and widest impact change that could make meaningful security improvements to all devices produced. I have zero idea how to turn that into something that can be legislated though.
Beyond that the big win over performance even is customization. The most secure code, and the fastest code, is the code that isn't there at all. Do you really need your entire system to support LDAP authentication? Maybe... What about your local email daemon? Do you need that? Because cron does and since your mail daemon also has MySQL support built in, installing cron gets you the MySQL libraries.
I don't use it anymore because of the overhead, but there are a lot of performance and security benefits to be had there.
Additionally, I strongly disagree with your assertion. Adding any kind of scanning system like that is technically infeasible unless you sacrifice some measure of security. The government and many nation states have been trying to argue against encryption of various levels (even less than end-to-end encryption) and the ultimate outcome is it being rejected because all of those measures _meaningfully weaken the security of the system_.
If it was necessary for some reason... Running a language model to keep something like this is sync over long term training and iteration would likely be more expensive than a developer's time AND block the researcher in a verification loop on the output which still probably needs to be checked by the developer (they could be the same person which will just deepen the frustration they experience).
The use of a lot of garbage accounts in this thread and lack of model details also looks pretty shady...
> Junior minister Stephen Parkinson appeared to concede ground to the tech companies' arguments on Wednesday, saying in parliament's upper chamber that the Ofcom communications regulator would only require them to scan content where "technically feasible".
What this means is any chat system that isn't using end-to-end encryption is subject to these laws. Anything using end-to-end encryption is considered unfeasible to scan.
I don't know what improvements to the current process actually look like but it needs to be able to account for effectively fraud and apathy on both side of the equation.