Hacker News Reader

Comment by saberience | Hacker News Reader

I think the author doesn’t realize that 99.9% of people with an m1 Mac don’t care at all about running local ml workloads. The people that do care about running local ml workloads have either a workstation with an NVidia card, access to virtual machines on the cloud with NVidia processors or some Apple computer with an M2 Max or similar processor.

Put simply, this isn’t a big deal at all. Most people are not ml power users and most people are using their Mac for web apps via a browser, local office apps, and something like zoom or slack.

Also, as an aside, Siri is mostly useless and I’ve never met anyone who uses it regularly. So whether it needs ml processing or not is irrelevant.

View on HN

Counterargument: most people are not running directly the ML workloads, agreed - however Apple implements them as features of the OS (like the new GPT powered text predictions). So it WILL affect the regular users over time as new features are not made available for M1 users.

The only reason you want to use bfloat16 is to train "compressed" models. Best inference performance tends to be at 4 bith quantization (and many people are trying to make it go down, even to 1 bit, and with sparsity, actually below that)

So this just doesn't matter.

Ironically, memory speed is what matters. And that's exactly the point where the M1 actually beats even the M3.

Thanks. I was going to rebut your point but I re-read the article and while one of the headings seem to rebut it, the actual content does not contradict anything, so I'll agree with your argument :)

“ Siri is mostly useless and I’ve never met anyone who uses it regularly. So whether it needs ml processing or not is irrelevant.”

You should get out more. I use Siri daily to switch lights, set reminders, turn on timers, and play music. It works well within limited domains like that. I don’t try to use it as an answerbot. That is a much bigger task and it is not suited for that.

Certainly Siri (and Alex and Google Assistant) is not very good right now for more complex tasks. That’s why Apple’s announcement that they will be adding LLM-style AI in the next OS interesting as it could make Siri more reliable and useful for more people.

I have an M2 Pro with 32Gb RAM. I keep ollama running in the background and use it either on the terminal for a chat interface, or as a code assistant on VS Code.

It works pretty great. Of course, larger models take longer to respond or don't work at all but that hasn't been an issue for my use-case.

Do you mind sharing how you use ollama in VS Code?

Hey, sure. It's nothing fancy really. I've been using 2 extensions: Continue.dev [0] and Llama Coder [1].

The former for a chat-like interface where you can ask questions about the code, request it to write a function, tests, refactor, etc. The second for an inline, more Intellisense-like completion.

I'm on a Mac, so installed ollama from Homebrew, then `homebrew services enable ollama` to have it always in the background.

Configuring the extensions to use it is well documented, I imagine you won't have any trouble but let me know if you get stuck.

[0] https://continue.dev/

[1] https://marketplace.visualstudio.com/items?itemName=ex3ndr.l...

> I think the author doesn’t realize that 99.9% of people with an m1 Mac don’t care at all about running local ml workloads.

> (…)

> Put simply, this isn’t a big deal at all.

The author isn’t making a consumer argument, but a statement of fact. Which is perfectly in line with this blog, its author explores technical details of Macs but doesn’t necessarily make a value judgment on if something is good or bad, it’s about explaining how something works. The typical readers of this blog are the ones which will care about these details.

Most won't know they're running ML workloads. All they know is that cool new Photoshop feature doesn't work as well on their computer as their coworkers'.

But this has always been the case with older computers simply due to the inevitably faster processors coming out every year, and it hasn’t changed now. M1 processors were released more than 3 years ago.

At anytime in the last 30 years of computing, if you have a three year old computer it won’t run the latest version of Photoshop, Maya, whatever videogame, as well as the newest machines.

This was the case when I had a 16Mhz Mac and I saw people buying computers with 66Mhz computers and it’s still the case now.

Technology moves forward and new stuff will start to run slow on older machines, it’s not revelatory news.

I have a M1 Ultra Mac Studio with 128GB of RAM and the ability to run LLMs like LLaMa2 locally thanks to the unified memory architecture was a big factor. Just because you don't have a use for it doesn't mean everyone else is as blinkered as you.

As soon as the M3 Ultra lands, I am getting one, preferably with 256GB RAM.

Where is my M3 Ultra MacBook Pro? I’m okay with reduced battery if I can have 256GB of ram and the processing power.

Have you seen the size of the heatsink on the Mac Studio Ultra? It's there for a reason...

It's a 60W part, so it would probably be _just_ about doable (old MBPs had up to about 75W worth of stuff to cool between CPU and GPU, though they definitely struggled a bit and the battery life on the dGPU ones was never amazing).

>Siri is mostly useless

I would normally agree with you, but both my daughter and I had experiences with it this past week where we asked it to provide information while we were driving and it actually did - correctly even. We both commented that it felt like something had changed or improved

Yeah, why can’t you just log in remotely?

Reply on news.ycombinator.com