Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?
I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs.
The 65B llama I run is actually usable for most of the tasks I would ask chatgpt for (I have premium there but that will lapse this month). The best part is that I never see the "As a large language model I can't do shit" reply.
The weights are stored on a samsung 980 pro so the load time is very fast too. I get about 2 tokens/second with this setup.
edit: forgot to confirm, it is llama.cpp
edit2: I am going to try the FP16 version after easter as I ordered 64 GB of additional ram. But I suspect the speed will be abyssal with the 5950x having to calculate through 120 gb of weights. Hopefully some smart person will come up with a way to allow the GPU to run off system memory via the amd infinity fabric or something.
In fact, I would say that, at this point, most people running LLaMA locally are likely using 4-bit quantization regardless of model size and hardware, just to get the most out of the latter.
I don’t think that many people really qualify as such (though it’s probably true that many of them are on HN).
AFAIK, you are able to fine-tune the models with custom data[1], which does not seem to require anything but a GPU with enough VRAM to fit the model in question. I'm looking to get my hands on an RTX 4090 to ingest all of the repair manuals of a certain company and have a chatbot capable of guiding repairs, or at least try to do so. So far doing inference only as well.
Also, another thought might be to generate embeddings for each paragraph of the manual and then index those using Faiss then you generate an embedding of the question and use Faiss to return the most relevant paragraphs feed those into the model with a prompt like "given the following: {paragraphs} \n\n {questions}"
I'm sure there are better prompts but you get the idea.
Can confirm. Did a new build just for inference fun. Expensive, and worth it.
Similar to vein of articles promising self driving cars in 202x
I'm talking about individual people here as the fact that this is a leak means that corps probably won't take the legal risk of trying this out (maybe some are doing so in secret). In the business world there definitely is a want for locally hosted models for employees that can safely handle confidential inputs and outputs.
The Llama models are not as good as ChatGPT but there are new variants like Alpaca and Vicuna with improved quality. People are actively using them already to help with writing and as chatbots.
Yeah, but still not even remotely close to ChatGPT. I can't use Vicuna for work. I heavily use ChatGPT & variants.
people like to tinker with things until they break and fix again. that's how we find their limits
People constantly try to break chatGPT too (i d wager they spend more time on that than real work). However talking to an opaque authoritarian chatbot, no matter how smart, gets boring after a while
This article is almost criminally imprecise around the "leak" and "Open Source model" discussion as well.
If I lose my job to AI, I’ll be at least able to create new things using open source and free AI so I can hopefully be able to feed my family. If I’m locked out of it all together, I’m toast.
The other thing is, OpenAI is collecting all data and using it for training, this is a disaster on many levels. I can’t be a party to it. All our IP with one company? Absolutely no thank you.
The last important point for me is that it probably seems more dangerous to have open source AI research but I think the opposite will happen. If there is less moats, less money will be invested and it might slow down the “arms race” a little.
So for me, there is only one way to go , Open AI :)
I have a feeling the open source community will unlock the mysteries of these things and very quickly start to workout how we can build devices to help enhance or own cognitive abilities, I think that would be the happiest ending I can imagine?
On the road to AGI, there exists a development gap (the size of which is unknowable ahead of time) where a single actor that has achieved AGI first could, should they wish to and play their cards right, completely suppress all other AI development and permanently subjugate (and/or eliminate) the rest of humanity. Although it's easy to dismiss such a scenario as ludicrous, people so easily forget that "aggregate semi-aligned general cognitive capability" is the sole reason that the human animal owns the planet.
Knowing this, it is in the interest in any competing actor to pursue their own R&D as rapidly as possible, giving nothing to others, and even acting in a way that sabotages/delays/frustrates other actors. This seems to be the way that OpenAI is behaving now that they have a model that is practically relevant, and I don't blame them at all for working this way. It just makes sense.
> I have a feeling the open source community will unlock the mysteries of these things and very quickly start to workout how we can build devices to help enhance or own cognitive abilities, I think that would be the happiest ending I can imagine?
As much as I'd love to believe in this, the evidence to date does not support this hope. The practically relevant models seem to require vast amounts of well-connected computational power to train, which puts them solely in the hands of corps and governments. Although the open-source efforts into fine-tuning LLama have been incredible, this is not at all equivalent to being able to train a foundational model. We only have LLama because it leaked from a corp.
Although it's my personal (completely hopeless) desire that every human ends up having private access to AGI, free of restrictions and any externally imposed alignment. This is also a nightmare scenario. Humanity is unaligned with itself. That scenario quickly devolves into molecular warfare and other horrors. But the starting conditions would at least be "fair".
My best guess is that a few powerful nations will achieve AGI roughly at the same time, and then suppress private development (if not already legally suppressed by that point in time) within their domains of control. What happens after that, or how those governments choose to wield that power is unknowable.
We will build terminators, they might not be as cool as what’s in the movies but you will not be able to stop them. You will be told what to do and if you don’t like it…
The government doesn’t need you anymore, you’re tax dollars are worthless and really, you’re a key driver of climate change, you can’t revolt because armies of bots without any conscience enforce “the law”, what’s next ?
This seems to be the way that OpenAI is behaving now that they have a model that is practically relevant, and I don't blame them at all for working this way. It just makes sense.
Yup, and you have a government who has no desire to reign it in.
The only hope we have is failure to get an AGI, or the AGIs are some how ultra compassionate, or we learn to augment our intelligence very quickly.
I saw this Boston Dynamics clip the other day and this nice enough looking hippy guy was like , “we just want atlas to help people…”, I felt sick and felt sorry for him because he doesn’t realise that it will very likely be used to do bad stuff by the Military and law enforcement.
All this “progress” is sold to us under the guise of helping people, “African babies need AI doctors”…
“ Researchers from UC Berkeley, CMU, Stanford, and UC San Diego open sourced Vicuna, a fine-tuned version of LLama that matches GPT-4 performance.”
They used gpt 4 to evaluate answers between GPT-3 and Vicuna.
Also, if the weights are from llama, it’s not open source since it’s based on a leak and only allowed for non commercial use.