brucethemoose2

My concern is that Elon has shut the door with a sloppy lawsuit.

If some other entity (Government agency? Another company?) Comes around to pull this string, its going to set a bad precedent.

- A: I didn't know this, awesome.

- B: Yes, uploading via a web page would be nice too. I have seen other apps do this.

Is there an equivalent of this with an http front end, so the client doesn't need to install anything?

I have used such apps before, but it seems they have gone unmaintained and don't work anymore.

Yi 9B has been released with quite impressive looking benchmarks. While probably not a SOTA coding model, it looks like a strong competitor to leading edge "small" models Mistral 0.2 and Solar.

Meanwhile, Yi-34B-200K has been updated:

> In the "Needle-in-a-Haystack" test, the Yi-34B-200K's performance is improved by 10.5%, rising from 89.3% to an impressive 99.8%. We continue to pretrain the model on 5B tokens long-context data mixture and demonstrate a near-all-green performance.

I am particularly excited for this, as Yi 34B 200K finetunes are already my favorite non coding models, even including 70Bs. And its just about perfect for a 24GB consumer card (which I can cram about 75K onto without too much compromise).

Apple could dramatically improve performance if they just tasked one Metal engineer on llama.cpp. Like, just to finish up flash attention and quantum KV cache, and optimize the Metal kernels.

I wouldn't be surprised if they could double performance.

I know Apple is pushing MLX, and MLC-LLM is fast too, but in practice most Mac users (I think) are using llama.cpp based stacks.

AI is unique in that the incentive is to write engaging, compelling, perhaps wonderful ostensibly truthful content that is total nonsense.

Enthusiasts in a subject are often wrong, but usually the goal is at least to convey something useful about the topic to the reader.

The spammer/content farmer doesn't care about that. Users trying to farm attention on social media don't care about that. A manipulator on a certain topic doesn't care about that. They just want SEO and eyeballs. But usually the intersection between high quality topical writers and people with these motivations is very low.

...That is not the case any more.

On the contrary, I think open human discourse in platforms that aren't really a lure for spammers or trolls (like code repos or really niche topics) is more important than ever.

Generic social media is going to be a hellscape, yes, but I think weird niches could still thrive if Discord wasn't walling them off.

One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and they are finicky, especially at smaller sizes.

Running one as a background desktop assistant is whole different animal than calling a Microsoft API.

I passed over the 7900 XTX because its not worth the trouble over a 3090.

That calculus totally changes with 48GB. I would put up with A LOT of trouble for that.

Its insane that a 2020 gaming GPU is such a niche for deep learning. Thats ancient history in the world of graphics, and its only that way because the market is straight up anticompetitive (and AMD is too dense to break out of it with a 48GB consumer card).

How about higher capacity GDDR6X?

48GB consumer cards (or 96GB pro cards) would sell like hotcakes if AMD/Intel dare to break the artificial VRAM segmentation status quo.

Asus uses custom firmware for more performance consistency, but yeah the silicon is the same AFAIK.

I would not trust AMD CPUs either. All modern desktop CPUs on most motherboards, especially those at the high end, will clock themselves to ridiculous voltages/power levels just to squeeze out another 5% performance. Intel is particularly egregious this generation, but the motherboard makers who further boost them by default are also to blame.

Thats how you get electromigration.

I would suggest AMD's X3D-only CPUs instead (which stick to very reasonable clocks/voltages), but I guess there's a risk of the stacked die failing as well.

Nah. Reddit will escape the enshittification cycle, just like... Uh...

In hindsight I think the post was overly negative, even though I am no Gemma fan. People are downvoting just because of that.

Now, what's a little odd to me is that there was so much interest positive interest from HNers? People submit really good new models/finetunes all the time, but there's usually no interest in them, at least not unless I bring them up in tangentially related comment sections.

exl2 is Nvidia/AMD only.

But GGUF Mixtral should fit in 32GB... just not with the full 32K context. Long context is very memory intense in llama.cpp, at least until they fully implement flash attention and a quantized cache.

On 33B/34B models I get 35 tokens/sec, way faster than I can read streaming in. At huge contexts (like 30K-74K), prompt processing takes forever and token generation is slower, but its still faster than I can read.

Miqu 70B is slow (less than 10 tok/sec, I think) because I have to split it with llama.cpp. I only use it for short context questions where I need a bit more intelligence.

And for reference, this is a SFF desktop! It's no Macbook, but still small enough (10L and flat) for me to fly with in carry on.

Yi 34 200K finetunes (like Tess 1.5),Deepseek Code 33B and Miqu 70B definitely outpace ChatGPT-3.5, at least for me.

They don't have the augmentations of being a service, but generally they are smarter, have a bigger context and (perhaps most importantly) are truly unbound.

I am on a single 3090 desktop, for reference. Admittedly, this is much more expensive now than it was a few months ago, with the insane prices used 3090s are going for now.

Counterpoints:

- Local models are pretty easy to de-censor, if thats what you mean.

- ...Yeah, it should not be labeled as a 7B. Its sort of 7B class.

- The repo mentions they use the llama-cpp-python server

- 1M context brute forced across TPUs is insanely expensive, I can see why Google reigned it in.

But overall your message is not wrong. Google is hyping Gemma a ton when its... Well, not very remarkable. And they could have certainly made something niche and interesting, like a long context 8.5B model, a specialized model, a vastly more multilingual model, something to differentiate it from Mistral 7B 0.2

Practically, speaking, I OOM'd running Gemma on a 3090 using a config that had VRAM to spare for Mistral 7B. It kinda surprised me at first, until I realized why.

What are HNers looking for in this article? The architectural differences, or how to run/finetune it?

I just loaded it in vllm with default settings.

I can't share the eval, but it's pretty simple: it asks a question about some data, and is restricted to only answer yes/no (based on the output logits and suggested in the prompt). It's called with 0 temperature and only 1 output token, so sampling shouldn't be an issue.

The scale of the cloning is quite fascinating.

As I said, the entire world needs to shift how they percieve digital media. People are used to seeing a photo/video and thinking of it as representing something real... that's not the case. Now it's a fantasy, with the degree depending on the source.

OK that sounds lovely. We are but a poor startup, but I am hoping we can afford to reach out to y'all soon. Maybe even share workflows/stuff that works for us. If some ML (unsloth? lorax? sglang?) work on a Mi300 with a little tinkering, that would be spectacular.

We can't quite afford an 8x bare metal instance, but I am pinning this for later.

Think ya'll will ever offer single MI300X instances?

Some of these are pretty good. The AITemplate Stable diffusion demo, for instance, is a nice hidden gem (though I'm not sure if it works on consumer hardware these days).

It's complicated, but basically because most are llama architecture. Meta all but set the standard for open source llms when they released llama1, and anyone trying to deviate from it has run into trouble because the models don't work with the hyper optimized llama runtumes.

Also, there's a lot of magic going on behind the scenes with configs stored in gguf/huggingface format models, and the libraries that use them. There are different tokenizers, but they mostly follow the same standards.

I eval'd it with vllm.

One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab.

Many Chinese llms have a similar "default sampling" issue.

But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.

At the risk of being off topic: I am jealous of your handle.

And yes, this was an amazing landing and proof of the tech.

And cache! The talk of AI hardware is now "how do we fit these darn things inside SRAM?"