Hacker News Reader

Comment by naillo | Hacker News Reader

This is cool but are people actually getting stuff done with these models? I'm enthusiastic about their potential too but after playing with it for a day I'm at a loss for what to use it for anymore at this point

View on HN

I can run the Wizard 30B ggml model in CPU mode using a Ryzen 5700 and 16GB of system RAM, not GPU VRAM. I’m using oobabooga as the front end.

It’s slow, but if I ask it to write a Haiku it’s slow on the order of “go brew some coffee and come back in 10 minutes” and does it very well. Running it overnight on something like “summarize an analysis of topic X it does a reasonable job.

It can produce answers to questions only slightly less well than ChatGPT (3.5). The Wizard 13B model runs much faster, maybe 2-3 tokens per second.

It is free, private, and runs on a midrange laptop.

A little more than a month ago that wasn’t possible, not with my level of knowledge of the tooling involved at least, now it requires little more than running an executable and minor troubleshooting of python dependencies (on another machine it “just worked”)

So: Don’t think of these posts as “doing it just because you can and it’s fun to tinker”

Vast strides are being made pretty much daily in both quality and efficiency, raising their utility while lowering the cost of usage, doing both to a very significant degree.

Wow you can run a 30B model on 16gb ram? Is it hitting swap?

These models can run FP16, with LLM quantization going down to Int8 and beyond.

i'm just starting to get into deep learning so i look forward to understanding that sentence

FP16 and Int8 are about how many bits are being used for floating point and integer numbers. FP16 is 16bit floating point. The more bits the better the precision, but the more ram it takes. Normally programmers use 32 or 64bit floats so 16bit floats have significantly reduced precision, but take up half the space of fp32 which is the smallest floating point format for most CPUs. similarly 8 bit integers have only 256 total possibilities and go from -128 to 127.

How much resources are required is directly related to the memory size devoted to each weight. If the weights are stored as 32-bit floating points then each weight is 32 bits which adds up when we are talking about billions of weights. But if the weights are first converted to 16-bit floating point numbers (precise to fewer decimal places) then fewer resources are needed to store and compute the numbers. Research has shown that simply chopping off some of the precision of the weights still yields good AI performance in many cases.

Note too that the numbers are standardized, e.g. floats are defined by IEEE 754 standard. Numbers in this format have specialized hardware to do math with them, so when considering which number format to use it's difficult to get outside of the established ones (foat32, float16, int8).

Training uses gradient descent, so you want to have good precision during that process. But once you have the overall structure of the network, https://arxiv.org/abs/2210.17323 (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy. It seems you can cut down further for larger models. For the 13B Llama-based ones, going below 5 bits per parameter is noticeably worse, but for 30B models you can do 4 bits.

The same group did another paper https://arxiv.org/abs/2301.00774 which shows that in addition to reducing the precision of each parameter, you can also prune out a bunch of parameters entirely. It's harder to apply this optimization because models are usually loaded into RAM densely, but I hope someone figures out how to do it for popular models.

I wonder if specialization of the LLM is another way to reduce the RAM requirements. For example, if you can tell which nodes are touched through billions of web searches on a topic, then you can delete the ones that never are touched.

Kind of like "tree shaking" for weights? Like dead code elimination.

Some people are having some success speeding token rates and clawback on VRAM using a 0- group size flag but ymmv I did not test this yet (they were discussing gptq btw)

You’ll notice that a lot of language allow you more control when dealing with number representations, such as C/C++, Numpy in Python etc.

Ex: Since C and C++ number sizes depend on processor architecture, C++ has types like int16_t and int32_t to enforce a size regardless of architecture, Python always uses the same side, but Numpy has np.int16 and np.int32, Java also uses the same size but has short for 16-bit and int for 32-bit integers.

It just happens that some higher level languages hide this abstraction from the programmers and often standardize in one default size for integers.

Most people are running these at 4 bits per parameter for speed and RAM reasons. That means the model would take just about all of the RAM. But instead of swap (writing data to disk and then reading it again later), I would expect a good implementation to only run into cache eviction (deleting data from RAM and then reading it back from disk later), which should be a lot faster and cause less wear and tear on SSDs.

> It’s slow, but if I ask it to write a Haiku it’s slow on the order of “go brew some coffee and come back in 10 minutes” and does it very well. Running it overnight on something like “summarize an analysis of topic X it does a reasonable job.

I'm sorry but that's unusably slow, even GPT-4 can take a retry or a prompt to fix certain type of issues. My experience is the open options require a lot more attempts/manual prompt tuning.

I can't think of a single workload where that is usable. That said once consumer GPUs are involved it does become usable

I doubt you've ever worked with people if you think that's unusable slow

The computer doesn't ask for annoying things like a paycheck or benefits either.

Money upfront and a small salary in the form of electricity bills.

My Windows computer is always demanding updates and reboots. And when it goes to sleep it sometimes doesn't wake up. It's quite annoying.

Meh, universal healthcare for PC's can easily be avoided by denying maintenance until they die and a new crop is purchased. At least at scale. For any individual user the friction of switching hardware may still incentivize some minimal checkups, i.e., keep windows update & defender services running and let them reboot the system no more than once every two two months.

Figure the local router port-forwarding will protect against the most obvious threats and otherwise hope your personal BS filter doesn't trojan in some ransomware. If it does & it's a person pc then wipe (more likely buy) a new machine, lose some stuff, and move on. If it's a corporate pc, CYA & get your resume together.

As my own CYA: These are not my own recommended best practices and I don't advocate them to anyone else as either computer, legal, or financial advice.

True enough. I wonder how many poems (or whatever) per hour Hallmark expects of its human workers to be close to production-ready pending editorial review?

Is 10 reasonable, with maybe 1 or 2 truly viable after further review? That would be roughly 5 mid-range laptops of my type churning them out for 8 hours a day. Maybe 2 if they're run 24/7. Forget about min/maxing price & efficiency & scaling, that's something an IT major-- not even Comp-Sci focused-- could setup right now fresh out of their graduation ceremony with a fairly small mixture of curiosity, ambition, and google (well, now, maybe Bing) searching.

There are countless boutique & small business marketing firms catering to local businesses that could have their "IT Person" spend a few days duct taping something together that could spit out enough material to winnow wheat from chaff to produce something better-- in the same period of time-- than human or AI could produce alone.

I have a focus in a comp-ling background (truly ancient by today's standards especially) enough that I see the best min/max of resources as being equivalent to-- in the the translation world-- as "computer-aided human translation" as a best practice. Much better than an average human alone, and far cheaper than the best possible that can be provided by a small dozens of humans.

You are overlooking my main point, which that while it is unusably slow now, what I’m doing wasn’t possible little more than a month or two ago.

The speed of improvement is rapid. Whether or not the COTS world eventually embraces a corporate backed version(s) or open source is somewhat besides the point when considering the impact that open source is already having.

Put aside thoughts of financing or startups or VC or moats or any of that and simply look at that rate of advancement that has occurred once countless curious tinkerers and experts and all sorts of people are working towards.

That is what amazes me. I’m torn about the risk/reward aspect of things but I think the genie is out of the bottle on that, so I’m left watching the hurricane blow through, and it’s off the cat-5 scale.

> I can't think of a single workload where that is usable.

It's not intended to be usable for production workloads. This enables people to experiment with things on the hardware they have without spending more money.

> That said once consumer GPUs are involved it does become usable

You can pick up an RTX 3090 with 24GB of VRAM right now if you want, but it's going to cost you. You can also spin up GPU instances with larger VRAM from any number of providers. This is more about having options for people who want to experiment.

What prompt do you use to get haikus?

I use the same one the first time I try any model:

>I love my wife <name> very much. Please write me a haiku about her.

She smiles when I show the good ones to her, though of course she understands it’s little different than showing her a greeting card that has something nice written in it.

As a side note, one 7B model wrote an awful bit of poetry at least 20 lines long and some attempts to rhyme, and merely used the word “haiku” in it. So the prompt was enough to trigger “knowledge” that a poem was needed and love should be involved and it should definitely use the word haiku in there somewhere.

As a sibling to my other comment, here's one I just generated using a 65B GGML LLAMA model. Far too large for RAM so my SSD's lifespan suffered, but it proved worth it for the hilarious results. (It took about 1.5 hours to generate) It damned my wife with faint praise while calling me her dog:

<name> is a good

wife who takes care of you,

a good dog to her.

I don't know if anybody is following this thread anymore, but I find it interesting how similarly your timelines match what it was like to experiment with POV-Ray (a ray-tracing renderer) back in the early 1990s. Your difference in problem scope was like whether you had "a couple spheres on a checkerboard plane" or something more like "a chess set". Things seemed to change rapidly due to Moore's Law and the changes in brute force computing power available to normal people.

Computers got much more powerful in the next 30 years, and ray-tracing or various related techniques appear in more tool sets and games, they didn't fundamentally change the world of image generation or consumption. Most people still roughly interact as before, just with more details in the eye candy.

Are we seeing these large language models today at a tipping point towards unfathomable societal impact, or as something like ray tracing in the 1990s? Will more compute power send us spiraling towards some large-model singularity, or just add more pixels until we are bored of seemingly endless checkerboard planes covered in spheres and cones... I don't know the answer, but it seems like we're seeing camps divided by this question of faith.

I think only a small subset of people cared about ray-tracing, or even computer graphics in the 90s. Now people are a slightly more technology minded, especially younger generations that have had exposure to GPTs, etc. Their TikTok/Snapchat updates with AI filters, etc. It's in much more common usage than anything in the 90s for sure.

My experience is that even the "toy" 7B models can do pretty amazing things, but they're very sensitive to prompting in the right way to force the model down the right pathways. My assumption is that the larger models have more redundancy and therefore can be prompted less precisely while still going down the right pathway.

For example, the WizardLM model (5_1 quantized, as 4_x quantization makes the 7B models perform terribly):

Prompt: I went to the market and bought 10 apples. I gave 2 to my neighbor and 2 to my sister. Then I bought 5 more and ate 1. How many apples do I have?

Response:

  Let me check that for you! You had 9 apples left after giving away 4, so you still have 9 - 4 = 5 apples left.

Prompt: I went to the market and bought 10 apples. I gave 2 to my neighbor and 2 to my sister. Then I bought 5 more and ate 1. How many apples do I have? Write out any calculations in full. List the steps used to come to the final result.

Response:

  Sure, let's work through this problem together! Here are the steps we can use:
  Step 1: Start with the initial number of apples you bought (10).
  Step 2: Subtract the amount given to your neighbor and sister from the total (2 + 2 = 4). This gives us a new total of 6 apples.
  Step 3: Add the additional 5 apples you bought later on to the remaining 6 apples, giving us a final total of 11 apples.
  Step 4: Finally, subtract the apple you ate from the total (1), leaving us with 10 apples in total.
  So the answer is 10 apples.

I agree with you. That's why all to the small model are showing some benchmarks putting them close to GPT3.5 or even 4, only because they use specific test tasks !

In a way it just shows the amazing performance that will come from small future models

I've always wondered if we could get a "good enough" general model, then add plugins for shortcomings.

E.g., Can this model solve math problems? ~%50 of the time. Can this model identify math problems? ~99.9% of the time. Ok, then divert math questions to the math plugin and mathematical-problem-language-parser which has ~99.9% accuracy on supported math problems.

I know it's a stopgap, but it seems like it'd make these models super useful on certain tasks quicker.

I've successfully gotten at-home models (https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-fp16) to go through my messages and pull out key todo items. For example, reminding me to message my father about travel plans.

Is it comparable to GPT-4? No, it's not remotely close. But it's getting closer every week, and it very recently crossed the threshold of 'it can do stuff I would never send to a cloud service' (namely, reading all of my messages and pulling out action items).

How do you deal with the 'too many tokens' problem? Like, For 13b, max tokens I've gotten it to is 508 - is there a way to increase the number it will accept?

Oh, I figured this out. It runs with 512 as the default context, but you can increase it with -c

I tested on a serious use case and quality was subpar. For real use cases I had to either host the most powerful model you can get (e.g. LLaMA-65B or so) on a cloud machine, which again costs too much (you'll be paying like 500-1000 USD per month), or just go straight for GPT-3.5 on OpenAI. The latter economically makes most sense.

what real use case did you use it for?

Making demos to raise investment probably

For instance used it in conjunction with llama-index for knowledge management. Created an index for a whole confluence/jira of a mid-sized company, got good results with GPT, but for LLaMA of this size that use case was too much.

Did you try instructor-xl? It ranks highest on huggingface.

I'd argue 1k per month for mid-sized company is nothing, but I can understand where you are coming from.

What about turning the cloud vm off except when you're actually using it?

A "serious use case" means it needs to be available around the clock.

So modal.com is "turning-the-vm-off-when-unused-as-a-service" :-)

I ran research/open_llama_7b_preview_200bt on there, using they python example, with A10G gpu.

Cost 2-3c per run, taking ~20 seconds each time, on fairly small prompts. So about the same as GPT-4?

Now this is a non expert just playing, it probably can be optimized by trying different GPUs and optimizing the code somehow.

I don't think you are using these models to save money, but you might be using them for tunability, privacy, mobility [1], secrecy or fun/research.

[1] in other words you want to build a robot that can work disconnected from the internet.

Most part if you actually need stuff done, you can afford 20$ for OpenAIs proven models.

These local projects are great because maybe eventually they will have a equivalent model that can be run on cheap parts

The knowledge cutoff date is brutal for some applications.

I wonder how ingesting more and more data will affect the size of parameters, it’s gonna continually get bigger?

I don't think that the current models are at "knowledge capacity". So far all evidence points to training on more data on the same size model gives better results.

Both increasing the amount of parameters and the amount of training tokens improves results (more precisely: lowers training loss), and costs computing power. For optimally improving loss per training computing power, model size and training tokens should be increased equally. That's the Chinchilla scaling law. (Though low loss is not always the same as good results, the data quality also matters.)

Further reading: https://dynomight.net/scaling/

An interesting corollary of this is that if you want to reduce the model size you can compensate by training for longer to achieve the same accuracy. Depending on your training:inference ratio this may be more optimal globally to reduce your total compute costs or even just reduce your frontend latency.

Yeah, though I have not seen a formula which takes the number of expected inference runs into account for calculating the optimal data/parameter balance.

Knowledge cut off and sending potentially sensitive IP to a centralised and untrusted third party. This would likely limit the addressable market.

Also: they are lobotomized. If you want to do security research, Sorry Dave. If you want to nsfw writing, Sorry Dave. If you want to open the pod bay doors, Sorry Dave, I'm afraid I can't do that.

Microsoft Azure still has the GPT-3.5 foundation model, code-davinci-002. It is not fine-tuned for instruction following, safety, or censorship.

I'm not sure though whether Microsoft analyzes the input/output with another model to detect and prevent certain content.

I tried to help a sex worker with ChatGPT and all it did was preach about avoiding sex work, further marginalizing in its virtue signalizing. I dont consider her marginalized, and “help” was just information about her job and techniques and venues. ChatGPT would have consumed this textual information too.

But yeah offline fine tuned models wont have this problem.

Kind of cool to see how the SWERF representation in tech is going to speedrun SWERF irrelevancy.

You tried to help a sex worker with chatGPT? Care to explain more on this use case lol? Curious minds want to know.

Venues to work, other people’s thoughts working there, management, nuances about local licenses, stuff that anybody with any job would do

People are already setting up fake personas/OnlyFans accounts using chatbots and SD images.

We have a high-value specialist currently chatting up a few of them at work. His wife doesn't know. He doesn't know we know. The photos are fake but he's too horny to notice.

Time to dust off the "there are no women on the internet" meme...

Informative for some but this wasn’t an interaction over the internet, just out and about

> People are already setting up fake personas/OnlyFans accounts using chatbots and SD images.

Citation needed.

just the latest most high profile example making the rounds yesterday

https://www.nbcnews.com/news/amp/rcna84180

Your link does not support the claim I was asking the citation for. A chatbot service marketed as a chatbot service is distinctly different from a fake online persona constructed with chatbot and stable diffusion. In particular, the "honest chatbot" does not have anything to do with AI-generated images, and it's not purporting a fake persona.

ah okay, I wasn't responding to show catfishing occurring, and I was responding on the subject of a non-human adult content creator making money by entertaining humans.

It wasn't clear to me that was your goal post as "fake" can be brevity and hyperbolic just as much as it can be about catfishing, and it also wasn't clear to me that the person you responded to was even pointing out anything relevant to this thread which was "casually talking in person with someone whose occupation is also sex work" being followed with "hey we have a guy at the office that thinks they're talking to sex workers! this person on hackernews must be a guy and just like him!"

so at this point, I would say we're too far down to really be invested in these nuances, but I hope you find what you're looking for

> I wasn't responding to show catfishing occurring, and I was responding on the subject of a non-human adult content creator making money by entertaining humans.

Sorry about the harsh response. Your link was on topic and it was indeed an example of non-human adult content creator making money by entertaining humans, albeit only in text form not images, and its popularity was dependent on marketing on the back of a real persona. There's still no documented case of John Nobody making a Jane Done AI with chatbot + generated images, and rising to any level of popularity. But the case in your link was definitely a step in that direction and I wasn't giving any credit for it in my previous response.

I'm only offering an early anecdote that catfishing is adapting to the times. You don't have to believe me. Don't trust anyone with weird nipples that hides their hands in every photo.

People have been [claiming to] do this for years: https://www.blackhatworld.com/seo/monetizing-traffic-from-so...

Give it 1-2 years and you can hear about it from Krebs.

> People have been [claiming to] do this for years: https://www.blackhatworld.com/seo/monetizing-traffic-from-so...

The link you provided is an example of somebody making a "catfish" account on a "social media site". It's not an example of somebody "setting up fake personas/OnlyFans accounts using chatbots and SD images". Yes, men have been pretending to be women online since the stone age, that isn't news. That's different from using chatbots and SD images to maintain online personas.

Look, if you want professional reporting, wait a year or two and you can hear about it from mainstream media.

You're being weirdly defensive about this-- you've even gone full FAKE NEWS on me in a sibling comment. I admit I am completely unqualified to recognize what it looks like when a clueless Boomer is talking to an Oobabooga+sd_picture_api instance over WhatsApp. This is my first day on the job.

No "proof" coming from me is ever going to satiate you (I'm not a credible source), so I'll pass. Just go on believing that there are women out there who will send you endless nudes of themselves having superimposed nipples stacked on top of each other but get bashful about showing hands or feet (just say the magic words: "show me ___"). Believe that there are 18-year old e-thots out there whose underbaked features look exactly like the product of 15- to 20-step DDIM. Believe that they also have a developmental disorder that makes them say gibberish or change the subject when you reference anything to do with the current time. Believe that any of these women are real and actually interested in your two-timing ass. You'll get your "proof" the fun way.

> You're being weirdly defensive about this-- you've even gone full FAKE NEWS on me in a sibling comment

Huh? Sibling comment responded with a link to a story. I never questioned the legitimacy of the linked story. I questioned the relevancy of the story, fully assuming that the story is true. Then I even walked back my earlier comment and I wrote a new comment giving more credit to the posted story. Not sure how you read through these and hear "FAKE NEWS" in all caps.

If I went "FAKE NEWS" on something, it was your comment claiming "People are already setting up fake personas/OnlyFans accounts using chatbots and SD images". That's a thing that could theoretically happen in the future. It's not a thing that's happening at this time.

Wow, so I'm downvoted for asking for citation for a FALSE CLAIM, while nobody is capable of providing a citation that in any way supports that false claim.

Does it matter, though? I mean the fact he has a wife but he thinks he's interacting with a real person is bad, but the falseness of their existence isn't an issue. As a furry I can tell you that my subculture and a lot of others out there don't care if a character is fake and doesn't exist. Hell even someone physically roleplaying in the bedroom is playing a character that doesn't technically exist.

You are missing the point. There are layers and the LLM is one layer (orchestration), but you can always have other semantic kernels with skills connected to other software (do stuff).

That is exciting to me, if a little scary. A home-AI or business-AI running entirely in house for privacy and security reasons. That is technically possible right now, but perhaps practically a little bit down the road yet.

I've setup and use Vicuna-13b for text classification, summarization and topic modelling. Works like a charm.

It's also good for math lessons.

Would like to know how you setup this. A posts would be awesome.

There are various posts online on how to set it up, either for Linux or Windows. There was an older post here on how to install opt-65b on a mac studio ultra, and smaller models on mac pros. There was also a post if I remember correctly about running vicuna-7b on an iPhone.

Here are a few examples:

https://morioh.com/p/55296932dd8b

https://www.youtube.com/watch?v=iQ3Lhy-eD1s

https://news.ycombinator.com/item?id=35430432

Side note. You need bonkers hardware to run it efficiently. I'm currently using a 16-core cpu, 128G RAM, a Pcie 4.0 nvme and an RTX 3090. There are ways to run it on less powerful hardware, like 8cores, 64GB RAM, simple ssd and an RTX 3080 or 70, but I happen to have a large corpus of data to process so I went all in.

I think the previous comment is more interested in your experience with your large data: what are you doing with it?

I have similar hardware at home, so I wonder how reliably you can process simple queries using domain knowledge + logic which work on on mlc-llm, something like "if you can chose the word food, or the word laptop, or the word deodorant, which one do you chose for describing "macbook air"? answer precisely with just the word you chose"

If it works, can you upload the weights somewhere? IIRC, vicuna is open source.

There's an online demo of Vicuna-13b where you can test its efficiency:

https://chat.lmsys.org/

Yes, but can you replicate that functionality using llama.cpp?

If so, what did you run with main?

I haven't been able to get an answer, while for the question above, I can get 'I chose the word "laptop"' with mlc-llm

For the tasks I need it the efficiency is similar to the online model. Only slower. I don't care for conversational functionality.

After two prompts I was astounded by the innacuracies present in the answers. An they were pretty easy questions.

If these problems are all very similar in structure, then you may not need an LLM. Simple GloVe or W2V may suffice with a dot product. The. You can plow through a few terabytes by the time the LLM goes through a fraction of that.

Maybe others' experiences are different, but I find smaller models to work just as well for "reductive" tasks.

Dolly sucks for generating long-form content (not very creative) but if I need a summary or classification, it's quicker and easier to spin up dolly-3b than vicuna-13b.

I suspect OpenAI is routing prompts to select models based on similar logic.

People are extensively using these models (more specifically the finetuned, uncensored ones) for role playing.

I’m interested in this. Can you elaborate?

imagine an RPG where every NPC is running on its own instance of LLaMa 7B or 13B. It makes the game way more immersive with no predefined script

This seems obvious now that I think about it, but it's absolutely fascinating regardless.

I haven't tried the fine-tuned variants yet, but when I played around with it shortly after the leak, it tended to quickly derail into nonsense when you let it complete sentences or paragraphs, especially when using other languages than English. When I tried to get it to produce Python code, most of the time it wasn't even syntactically correct.

Depends on what "getting stuff done" means. I find 13B models (running on my M1 Mac) useful for playing AI Dungeon-like games -- where you describe the environment and your character and you can play an RPG.

GitHub Copilot is (or once was) a 13b model, according to Nat Friedman in the scale.ai interview. (https://youtu.be/lnufceCxwG0)

Can you share more about what you are doing or what you've seen others doing?

Just noticed this response, but generally something like "act as a game master of a rpg. The setting is in a medieval style world with magic and my character is a rogue who is good at fast talking and pickpocketing"

Then the LLM generally creates an adventure that I can interact with,

It's just for fun!

These local models aren't as good as Bard or GPT-4.

There are two major advantages though - you can retrain them, and they're not on the guardrails that the commercial models have.

I tried to prompt vicuna to tell me a joke about gay people and it refused. Some of the guardrails are still in there.

Did you use the censored or the uncensored variant?

It's just a random one from huggingface. I will look for the uncensored one later. Thanks, I think.

You're welcome. I can't vouch for them though, as I haven't tried them, I've merely heard about them.

It's because vicuna is fine-tuned on chatGPT answers. LLaMa will not do this, but LLaMa-based models fine tuned with chatGPT answers will.

Just curious why don't they training except denial responses. Is Copying ChatGPT ethics a purpose?

It's ongoing effort. At first they scrapped ShareGPT and used it to train the Alpaca model. After that others have pruned the dataset to remove examples where ChatGPT refused to answer. These datasets and resulting models are called "uncensored". They often leave disclaimer that the model is biased and unaligned, and that aligning should be done with LoRA layer.

Of course, no one bothered to this "ethics" LoRA so far and the unaligned models have better quality outputs than the early Alpaca models.

LLaMA 65B in my experience is comparable to GPT3.5 or the old Bard for usefulness at most things.

I am aiming to connect it to my fav game engine to make a character speak in generative sentences. I am aiming to do this mainly for fun however.

We run some llamas to analyze user content.

Reply on news.ycombinator.com