I want to be able to run a large model AND other apps at the same time.
To be fair, I don't think people mucking around with large LLMs is the primary target market for the iMacs.
The sort of people who muck around with LLMs almost certainly already have a monitor, keyboard and mouse. And so are more likely to pick up a Mac Studio which will no doubt be coming soon with M4 Ultra.
https://cerebras.ai/blog/introducing-cerebras-inference-ai-a...
- CPU RAM
- GPU RAM
- GPU SRAM
The grandposter was talking about moving layers between CPU and GPU RAM, while Cerebras store models entirely in SRAM.
CPU <-> GPU is _much_ slower than GPU RAM <-> GPU SRAM: the former will never be competitive with keeping the weights entirely on accelerator. Comparing the bandwidth numbers given in that article with the bandwidth of PCI-E should demonstrate the problem.
Larger models are split between multiple accelerators, but even then they avoid going through the CPU as much as possible and instead use a direct interconnect (NVLink, etc). The weights are uploaded ahead of time to ensure they don't need to be reloaded during active inference.
The considerations for training are different, but the general principle of avoiding CPU traffic still holds.
Mac hardware offers up to 128 GB shared RAM.
If you want a Mac anyway, sure. But if you don't care, this seems like a very, very specific Venn diagram.
You're correct that it's only good for inference, but most people running local LLMs only do inference.
Although price-conscious LLM muckers are most likely to pick any Apple-hardware. You can easily build rigs that are twice as powerful for half the price, assuming we compare desktops.
I have a ten year old iMac at the moment that would make an amazing second monitor... but it doesn't quite have the features I need to use it like that. An HDMI input would be great.
There is a reason that is difficult to do: the licensing terms for HDMI require manufacturers to put DRM in place to make it impossible (or at least not straightforward) to record from an HDMI source. It's nearly impossible to meet that requirement on a computer.
Edit: nevermind, I see you addressed this in another comment.
In the specific case of the iMac:
- USB-C video outputs are all either DisplayPort[0]. HDMI Forum was promoting an HDMI USB-C altmode but it was never implemented. USB-C HDMI adapters are all DisplayPort adapters that additionally tunnel DVI traffic (because HDMI 1.x is actually single-link DVI with a funny connector and extra bits).
- Apple used to support Target Display Mode for video input until the Retina iMac a decade ago. They dropped Target Display Mode because their 5K display panels were using eight lanes of DisplayPort, a configuration that doesn't exist[1] outside of Apple's custom display controllers. Every other 5K display (and even some 4K ones) were instead using multiple cables, with some extra (poorly-supported) metadata to tell the graphics card to treat both halves of the display as one.
Why Apple hasn't brought TDM back now that we have enough bandwidth for 4.5K over standard cables is beyond me.
[0] Or MyDP, aka "the funny Nintendo Switch altmode"
[1] Think of it like a "32-lane PCIe port". It's technically possible but no host or device silicon supports a link that wide.
https://www.makeuseof.com/hdcp-vs-hdmi-whats-the-difference/
Not tunnel, convert: these are active adapters that take in a DisplayPort signal from the computer and re-encode it as a DVI/HDMI output signal. The computer doesn't need to know or care that there's HDMI happening downstream, because it's still just outputting DisplayPort signal. It's not like Thunderbolt where DisplayPort traffic is tunneled inside a different signalling mode that the host machine supports.
This is why USB-C to HDMI adapters are unidirectional (at least, all the mainstream commodity ones). You'd need a different converter chip to do the conversion in the other direction.
I’m contemplating doing this myself at some point but my maxed out 2019 iMac upgraded to 128GB ram and extra SSD is still plenty fast for me, actually feels subjectively quicker than my M2 Pro MacBook Pro with significantly less ram feel. I was a bit surprised as I had read all the hype of the responsiveness of the Apple M-machines.
Instead, the 2018 iMac is incredibly slow, and can be thrown away.
The screen of an iMac (at least the 5K 27" one) is the part that still has value, so its annoying when you can't use it on new hardware.
Perhaps check out the Asus PA27JCV: 27" at 5120x2880.
They were announced 'all the way' back in June, and I've heard rumours of October 31 being the ship date, so perhaps check back next week.
In the US, B&H is taking pre-orders, so presumably it can't be that far off:
* https://www.bhphotovideo.com/c/product/1850479-REG/asus_pa27...
The speakers and mic in the Apple display are nice, but if you're just concerned about the display itself save yourself the bucks and stay with the 4k.
I've had a few of these iMacs and would like another. When I buy a new one (every ~5 years) I hand down the old one to a family member (the last one is ~10 years old now, and stuck at macOS 12(?)).
I like the form factor as it is convenient.
I'm typing this on 2019 Retina 5K and am hoping Apple will bring back that form factor (there have been rumours of a 32", but that's a bit too big IMHO).
As it stands, it looks like the Asus PA27JCV has similar specs, and so I may end up with that and an Mac mini.
At some point we need to stop putting these things into landfills or even recycling them. I've been using the same PC case for at least 10 years ship of Theseus style and the same monitors for 5.
I appreciate apples recycling stance but even better is a reuse stance. Even just stripping back all the aluminum, melting it all down, re casting it and then remachining it has a significant cost.
For the last 10-20 years apple has been pretty good about reusing case designs for a few generations before doing some kind of redesign. Seems silly that I can't swap out a motherboard for a m2 macbook for an m3 macbook. (maybe this would also stop them from fucking soldering the storage to the motherboard since an upgrade that wipes my whole machine is utter bullshit)
Not everything you see is designed to be useful for you specifically
What do you mean? This was just announced, no? Are there any sales/pre-order figures already?
Also, how would you even go about analyzing the counterfactual of whether the number of people who would buy this spec are "plenty" compared to the number who would have bought a spec with more RAM had it been available?
Someone who is looking to muck about with LLMs isn't looking to pay for a 4k screen that cannot be separated from a non-upgradeable PC.
If you're in the Apple ecosystem, you're going to want eithet a Mac Mini Pro or a Mac Studio, depending on where the RAM configurations on the Mac Mini Pro tops out. .
You're using a M2 Max, not a M2. Two (big) steps up the chip product stack from what's used in the iMac.
The “Apple Intelligence” branding is nothing more than a selling point. Sure it might run some very small LLM workloads , but don’t expect much especially in the locked down hell known as Apple ecosystem.
This is not meant for heavy workloads.
Larger models that don’t fit are only moderately better, while much slower. I’d want higher memory bandwidth over more memory.
The next five years will push for better models that are smaller. Smaller is faster and more useful.
I feel like we are in the brute force phase of model development and that it will pass.
Running 405B param model in 16bit on my laptop would be neat, but I’d stop after the novelty wore off.
The comparison is apples to oranges.
Sheesh
I can run 70B at the moment... but not if I also want Firefox and VS Code at the same time.
your specific use-case is probably irrelevant to apple.
> I want to be able to run a large model AND other apps at the same time.
you answered yourself... go for a machine with 96 or 128gb ram.
btw for the money you'd be spending (6-7 k$) you might as well rent or buy a dedicated box with 128, 256gb or more ram and all the gpus you need.
I bet the M4 Max goes to 256. Oh the envy I will have.
iMacs are, generally speaking, targeted for home users, normies.
I mean, I’m livid Ferrari don’t make a cheap commuter for my family and 9 dogs. So I shop elsewhere.
Obviously there will always be use cases at each far end and in the middle they won’t meet unless that have 50,000 SKUs.
It’s fine, just spend your money elsewhere. Complaining about a trillion dollar company is not going to change anything or make you happy, so why do it?
#OneOfUs #OneOfUs
- iMac today, USB accessories
- Mac Mini on Tuesday, likely debut M4 Pro
- MacBooks on Wednesday, debut M4 Max
Now in the “I wish” category (zero percent chance):
- Thursday would be MacStudio with HBM memory for Max and Ultra
- Friday would be macPro with 1 to 4 Ultra in NUMA configuration.
Now I don’t believe those last two.
But I do want to see the X-ray of the max chip to see if it has the UltraFusion part that allowed for combining two chips. It was missing from the M3 Max (and maybe all future odd numbered max chips). If it returns, then we know an Ultra is on the way for sure.
https://news.ycombinator.com/item?id=35079
> pg: That has to be the comeback of all time.
Same original article:
https://news.ycombinator.com/item?id=35103
Tarsnap is functionally unchanged as a start-up and Dropbox is $2.5bn in revenue
That is a poor example of the thing you’re trying to mock. That exchange is an example of what one should do, it does not deserve derision. dang explained it well:
I'm actually a little hurt that you would treat my comment with such derision :(
Awards can definitely be derisive.
Yes, they make better laptops from a hardware perspective. No, M series chips are not actually competitive with Nvidia hardware on anything except cheap (and relatively slow) inference of big models.
The fact that windows laptops are considered DOA despite the insane amount of inertia behind Nvidia GPUs is just sad. I want a world where Dell/Lenovo can actually convince folks like you or other "elites" to use their shit. The XPS should be a better laptop than the macbook pro. Yet, I have to watch as the technical elites fawn over a company which continues to sell a starting level un-upgradable laptop with 8gb of ram and a gimped "SSD" in 2024 (this was criminal to do back in 2018) all because other companies can't make a good keyboard or touchpad.
I have a similar old-man yelling at cloud tier rant about the slow death of X86...
Until then it’s meaningless to even consider windows (laptop)
Except its.. not. I'm a longtime windows/linux user who recently switched to mac, and my mac somehow manages to get better performance out of the same specs while lasting 2x longer on battery. In my price range, there is no competition on price:performance:battery.
> all because other companies can't make a good keyboard or touchpad.
The whole point of a laptop is to have integrated input and display. Why wouldn't you expect the laptops with the best displays and input devices to be the most popular?
it does sound like envy. I hate macos post 10.8ish myself, but the hardware is pretty solid. my 12 year old Mac book air would be enough for my work although I have to use a windows laptop issued by work. I don't want to dox myself, but I'd say my work is of international interest even though it's quite niche.
It's very common in networking, sysadmin/devops, and web development in the UK, for instance. I go to a reasonable number of industry events and MacBooks are definitely the majority at these.
Or screen, sound, battery life, cooling, weight...
Are MBPs the best possible configuration out there? Of course not. But they're just much better _mobile devices_ than anything else available.
Integrated memory is important, since CPU <-> GPU bandwidth is often a limiting factor
until you can show me small nvidia laptop with 128gb of vram and 20 hours of battery life, i'll keep using my macbook.
As absurd as it is I just built a PC with a 3090 at both home and work for training and inference then carry around a MBP for everything else.
Apple's new hardware pales in comparison to the relative abilities of what features existed on a 27-inch iMac with Retina 5K Display 10 years ago.
All of the power of modern computers come primarily from their GPUs, and Apple's aren't very good, and have been chronically underpowered compared to the competition for years now.
I'm considering relegating Apple devices to being good for just design work.
Edit: I agree with Der_Einzige's sentiments. It's beyond time for us to move past Apple hardware.
Can you share where you’re finding inadequate performance? Genuinely curious to know
The MacBook Pros are in a class of their own but I suspect that most who buy the higher end ones do not actually use the power all that much (they would prefer a real workstation otherwise).
When I used to work at an Apple Premium Service Provider, all the clients who had the high-end MacBook Pro were the executives and people like that who made little use of the power. Every worker, even in media agencies had a desktop workstation (iMac or Mac Pro).
The MacBook Pro craze is mainly a Silicon Valley startup thing.
And yes, Apple computers are mostly useful for design/media work, if you have more use cases you are very likely to hit a roadblock, which is sad for the price.