Many options for running Mistral models in your terminal using LLM

This actually worked as advertised.

Other than that it's a great project - very easy to get started and has a solid API implementation. I've got it running on both a Win 10 + WSL2 docker and on a Mac M1.

If you want to run most models, get 64GB. This just gives you some more room to work with.

If you want to run anything, get 128GB or more. Unquantized 70b? Check. Goliath 120b? Check.

Note that high end consumer gpus end at 24GB VRAM. I have one 7900xtx for running llms, and the best it can reliably run is 4-bit quantized 34b models, anything larger is partially in regular ram.

16GB is not enough.

32GB is enough to run quantized Mixtral, which is the current best openly licensed model.

... but who knows what will emerge in the next 12 months?

I have 64GB and I'm regretting not shelling out for more.

Frustratingly you still have WAY more options for running interesting models on a Linux or Windows NVIDIA device, but I like Mac for a bunch of other reasons.

In the article, Simon mentions the Q6_K.gguf model, which is about 40GB. A Mac Studio can handle this, but any of these models are going to be a tight fit or impossible on a Mac laptop without swapping to disk. Maybe NVME is fast enough that swapping isn't too terrible.

In my experience, the Mixtral models work pretty well on llama.cpp on my Linux workstation with a 10GB GPU, and offloading the rest to CPU.

It is impressive how fast the smaller models are improving. Still, a safe rule of thumb is the more RAM the better.

Also, really question how much you need to run these models locally. If you just want to play around with these models, it's probably far more cost effective to rent something in the cloud.

memory bandwidth is the key to model speed, and memory size is what enable you to use larger model (quantization let you push thing further, to a point) so one thing to note is that on the M3 pro/max only the top end model gets the full bandwidh, while the m1/m2 pro enjoy full bandwidth from a smaller memory size. this may be important if you value speed above model size or vice versa. M2 Pro, M2 Max get approximately 200 GB/s and 400 GB/s, but things are more complicated for m3: M3 Pro gets 150mb/s, and M3 max gets 300mb/s at 36gb and 400mb/s at 48gb

few more things to note:

it's absolutely fine to go and play around with llm but even with a llm monster machine there's nothing wrong in starting with smaller models and learning how to squeeze the maximum amount of work out of them. the learning do transfer to larger model. this may or may not be important if at some point you'll want to monetize or deploy to production what you learned. while the mac itself is a good investment for personal use, once you move to servers, cost skyrockets with model size, because of supply constraints on 40gb+ memory gpus. if you are dependent to a 70b parameter model, you'll have a hard time to make a cost effective solution. if it's stricly to playing around, you can disregard this concern

even if you're playing around, a 70b is going to run at 7 tokens / second, which is fine for a local chat, but if you are writing a program and need inference in bulk, it's fairly slow.

another thing of note is that while the field is still undecided on which size and architecture is good enough, the moltitude of small fish experimenting with tuning and mix of instructions are largely experimenting on smaller models. currently my favorite is openhermes-2.5-mistral-7b-16k, but it's not an indication that mistrals are strictly better than llama2, more an indication that experimenting with 7b is more cost effective for third parties without access to gpu than experimenting with 13b, and so you'll find 13b model kinda stagnating, with many of them trained in a period where people didn't really know the best parameters for finetuning and are so to say a bit behind the curve. a few tuners are working at 70b models, but these seems to be pivoting to mixtrals and the likes, which will cause a similar stagnation on the top end, that is, until llama3 or the next mixtral size drops, then, who knows

What I can draw from reading of that thread is that you can buy a Desktop Rig with 200GB memory bandwidth (comparable to m3 pro and max) and a lot of expansion capability (256GB RAM). You should find out if that's still good enough for your local use case for token per second or training.

Then just use SSH/XTerm(and possibly ngrok) to login with good speed from anywhere into your rig with a light M2 ?

- finetuning, as in restarting the models checkpoint and relearning it on the previous + new data

- adding extra neurons (e.g. LoRA adapters) at certain places and restarting learning

Oh, in classic machine learning there's also the "bagging/boosting classifiers" option, but I have no knowledge if that can be applied to a ANN.

It's possible to introduce new information by fine-tuning a new model on top of the existing model, but it's debatable how effective that is for introducing new information - most fine-tuning success stories I've seen focus on teaching a model how to perform new kinds of task as opposed to teaching it new "facts".

If you want a model to have access to updated information, the best way to do that is still via Retrieval Augmented Generation. That's the fancy name for the trick where you give the model the ability to run searches for information relevant to the user's questions and then invisibly paste that content into the prompt - effectively what Bing, Bard and ChatGPT do when they run searches against their attached search engines.

For specifically knowledge you want it to be able to recall (like knowledge base articles or blog posts) vector database embeddings are best.

For knowledge you want it to operationalize, like being able to program in a new language the last resort is finetuning but this is not easy, requires massive amounts of high quality data, and is not generally effective for things which do not have a large amount of data to fine tune on (tens of thousands of pages worth of content).

These open weights models can be retrained. Start with a foundational model like Llama2 or something and expose it to more recent training data that includes whatever updated information you want it to have access to. This is relatively expensive, but allows for big changes to the model.

If you have some relatively small subset of new information you want to bring in, you could build a Lora. Then either run your model with the Lora, or fold the Lora into your base model. This is relatively cheap, but fairly narrow in terms of your updates.

In the long run, it might be that Retrieval Augmented Generation (RAG) is the way to go. Here, your embeddings go into a vector database, and the model reads from there. Then you just need to update the database for the model to have access to new information.

This LLM stuff is new enough that anything like best practices are still being worked out. The optimal way to bring in new information could be a variant of one of the methods I mentioned above, or some combination of all three, or something else altogether.

https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-fo...

Here's a good example from previous HN comments: https://news.ycombinator.com/item?id=38482347

Most GPT plugins for code editors will also work for local models since you can have OpenAI API stubs running locally.

Clearly it is nowhere near GPT-4 capacity, but if you ask simple boilerplate things ("write a class with the following methods", then "write unit tests for it") it will mostly work. Even if it doesn't, you can manually fix it, and it still can save you some time.

Always review code generated by LLM, regardless if it comes from GPT-4!

* how do i use library X to do task Y (excellent for quickly getting up to speed with new libraries).

* actual example from a few days ago: "the most common CI/CD systems and how to identify them" - chatgpt correctly gave me the environment variable names for github actions, gitlab ci, travis, circleci, jenkins and a one or two others. theoretically it saved me having to go through the docs of 7 different systems looking for the right information, which i still did to make sure the data was legit. just confirming the info was still a lot less work as i already knew what to look for.

* how do i create a certain style with css framework XYZ

* is there an algorithm for solving the following problem ...?

* alternative phrasings or synonyms if i can't find the right words myself.

* cooking recipe suggestions ("i have ingredients a, b and c, give me a stew recipe")

* pop culture questions ("why did the fremen settle on a hostile planet like arrakis in the first place?") i'm too lazy to research myself or ask on reddit

* sometimes my (non-natively english speaking) coworkers produce engrish i just can't parse. asking gpt to correct or explain the sentence often yields surprisingly good results.

usually i double check the results but that's still less work than doing all the work by myself from the beginning. recently i also let it write and style html forms which works quite well.

so for me, they're a welcome productivity boost.

Llama.cpp was the project that popularized running LLMs on the CPU due to its very efficient implementation. Ollama is a frontend to it.

Installing WSL (on Windows) is similarly straightforward nowadays. In your search bar, lookup the Microsoft Store, open the app, search for Ubuntu, install it, run it, follow the one-liner for installing Ollama.

https://ollama.ai/library/mistral

If they add that it will work out of the box: https://llm.datasette.io/en/stable/other-models.html#openai-...

Otherwise someone would need to write a plugin for it, which would probably be pretty simple - I imagine it would look a bit like the llm-mistral plugin but adapted for the Ollama API design: https://github.com/simonw/llm-mistral/blob/main/llm_mistral....