Hacker News Reader

> M4 with a terabyte of RAM will run GPT-4 level models locally.

No it won't. Apart from the fact that (if the article is correct), M4 will support maximum of 512 GB (and I don't want to know how many thousands of dollars you will have to spend on it given how much Apple wants you to pay for a difference between 8 GB and 16 GB MacBooks...), GPT-4-level models that you could run locally don't exist and it would be a challenge in itself to create them. The closest, Claude 3 Opus, as far as I can tell can't be run locally, and it is in nobody's interest to change this status quo (less capable models = run locally, powerful ones = pay us).

I do hope this changes one day, but it is not directly related to M4 (or M5/M6).

Apple can create them. They have the resources and can hire or acquire the expertise. And 512 GB should be more than enough for GPT-4 performance. We have invented a whole lot of techniques to reduce RAM requirements since GPT-4 was trained. Of course, GPT-5 and GPT-6 will come out and be better, but even just GPT-4 level performance will be enough for useful applications. And we will undoubtedly continue to discover ways to get better performance with less RAM. We've barely started optimizing these things.

I'm personally looking forward to it. One of the biggest advantages with running locally is cost. Yes, the upfront cost to getting your own hardware is much higher, but I'd much rather pay for something once then worry about paying a monthly bill for using someone else's infra that they may at any time decide to pull out of service or hike the price for.

The "no data sent to servers" point I agree is super important for work things. I pay for and use Github Copilot for personal things but I'm not allowed to use it at work for obvious reasons. If we could remove the privacy concerns and run these models on local hardware (that people actually have) then it'd be game changing.

Reading some of the other anti-AI comments is a bit odd. I guess it's going to depend on what you do for work, but as someone who writes software not using Copilot is like comparing myself from a 0.25x developer to a 2x developer. I mean that when I say it, but I get that you do need to be able to identify wrong code faster than you can run it.

I don't really want any of those things. Predictability and control is what I want from a computer, not to be an ersatz companion that back-seat-drives everything I do.

Like many, I am not overly bearish "on ai" - I am bearish on an "ai-focused laptop in late 2024." I think what you say is roughly true and maybe in six months I'll be wishing my portable computer ran AI faster, but I don't feel that way now. As you say, the full client-side experiences are still in development and, I would wager, more than six months away. Other attempts at "local AI" from less skilled hardware designers have not impressed me and, though I'm eager to see what Apple comes up with, I see no reason to get excited yet.

I agree that the most compelling applications are more than six months away. But I hope that a computer I buy now will last a lot longer than that. I don't upgrade my computer every two years anymore like in the 90s, and I don't want to go back to that. Three or four years from now I think you'll be feeling pretty left out if your computer can't run much AI.

It's relative though. I mean, how excited for an M4 without LLM capabilities are you though? Or an iPhone 16 that's basically the same as the 15 or the 14 with better specs?

I agree with what you are saying, but I think we will have to wait two or three years for the very nice scenario you laid out for us.

Earlier this year I popped for a M2 Pro 32G. I am amazed at what even that can run, but I have to use Ollama to run individual models for general text processing and NLP, one for medical advice, one vision model, one large uncensored model to get an understanding of general utility of censored vs. uncensored models (I wish all models I used were uncensored and trained just on English. I have only used non-English language support for a handful of interactions).

I also want Apple to develop very strong support for cloud models for my Apple Watch. With the excitement and disappointment for the new AI alternative pin computers, etc., I say we already have something useful in the Apple Watch so just keep improving that. I like to go about my day phone-free and the Apple Watch makes that work out OK for me.

The Gartner Hype Cycle has proven itself over and over again.

https://www.gartner.com/en/research/methodologies/gartner-hy...

We have not reached the "Peak of Inflated Expectations" yet in AI. Contrast that to Blockchain and Cryptocurrency, where we are definitely in the "Trough of Disillusionment".

> Contrast that to Blockchain and Cryptocurrency, where we are definitely in the "Trough of Disillusionment".

Bitcoin has been hovering around its all-time high of around $74,000 and is still expected to continue to increase, especially now that several ETFs have been approved.

Bitcoin =/= blockchain as a technology. It is the exception that proves the rule.

I'm pretty certain the median price among all crypto tokens introduced in the last 10 years is 0.

I really, really hope that the future you describe doesn’t come to pass. It sounds like hell.

There is an aspect of personal bias for sure: I don’t want my profession or the countless related fields to be devalued, I dislike the imprecision of LLMs, and they spark none of the joy in me that programming does.

But there’s something beyond that I find difficult to put into words. There is a tremendous inelegance to it all, in the same way a brute force solution feels fundamentally unsatisfying.

Most software today only uses a tiny fraction of the capabilities of the hardware, because we optimise for how fast we can implement marketable (and often hostile) features. LLMs and co strike me as dialling that up to 11; why improve anything when we can just throw more hardware at the stuff that sucks?

I like browsing the web god damn it, and I don’t need some corporate facsimile of a human to ask me why I don’t look happy.

> we optimise for how fast we can implement marketable (and often hostile) features.

I actually think local LMs are the antidote for this. Every UI I use these days is optimized by designers for attractiveness and hordes of PMs for business KPIs, not engineers for useful functionality. With local LMs I can take back control over the UI I actually use. The LM can deal with whatever trash UI redesign is being forced on me, and present a much simpler UI that matches my specific needs.

For example, it can summarize the news and I never have to visit a news site and be assaulted by dark patterns. It can collect family photos from Facebook and show me a few without random TikTok videos inserted between them. It can trawl Amazon and Target and Walmart to find a shortlist of products for me to pick from, filtering the trash. It can watch that 10 minute long YouTube video past the sponsored segment to find the nugget of information I need to fix my dishwasher or whatever.

We're a long way from this now. But I see a path there, and a world where users have control over the UIs they use is a better world.

We very much differ in optimism I think. I expect model-driven UIs to end up the same as software and services today: you can use open, non-hostile options if you go out of your way, but most people use the offerings from the big corporations and if you want a friction-free experience you have to as well.

I would love to use a third-party Instagram app that only shows posts from accounts I follow, but I can’t because Facebook has locked that down. I can just as easily see a future where I can summarise my feed, but only if I use the Metabot or whatever which is just implementing some different dark pattern.

Although that does bring me back somewhat to my initial point in that it all feels like brute forcing our way out of problems instead of solving them. We could have a Facebook that shows a nice summary of your family photos, we could have a shopping search engine that doesn’t include the trash, we could have short and to the point videos that aren’t optimising for the engagement flavour of the month.

We don’t because there are too many financial incentives pushing back and too many established players that can burn piles of money for much longer than you, and I just do not see how those same facets won’t just be translated for models.

> I would love to use a third-party Instagram app that only shows posts from accounts I follow, but I can’t because Facebook has locked that down

The point is that this is a local model. You don't have to use Meta's AI because Meta can't stop you from running a model on your own computer to browse Instagram for you and filter it however you like.

Personally I think that, if "model-driven browsing" (for lack of a better term) does take off, we'll see mitigations against it. Not unlike how we currently see mitigations (of mixed effectiveness) against ad-blockers and using things like Puppeteer as scraping tools.

I'm sorry to be so pessimistic about it, and I genuinely do hope we see some positives out of this stuff. But I'm becoming increasingly disillusioned with what so much of the software industry has become, and seeing how much of the AI space is led or backed by the tech giants, I can't help but assume the worst.

I would like something like Perplexity Pro that I could run locally. I have little pieces in the LangChain/LlamaIndex book I wrote and more little pieces in my new open source project [1]. I have become addicted to Perplexity Pro, it makes the web fun again for me, but I would like to have open source code I could run locally with similar functionality (that would be difficult because Perplexity Pro is a brilliant product).

[1] https://universalworker.org/

> You will be able to converse directly with your computer as if it was a person.

No thanks, I like that my computer only speaks an incredibly specific language which I can use to tell it exactly what to do, and know it’s not going to do anything else. If I wanted a PA, I would hire one.

Perhaps you aren't the target market for this.

I want a PA at all times, but cannot afford one, but I can afford to spend a few thousand on a laptop.

I agree. I just still don’t believe in Apple’s marketing in regards to AI. Past M chips were way slower than consumer Nvidia RTX chips. Their only advantage was the available memory. I’m speaking as a member of the Apple cult.

I can't imagine anything facile and under-stimulating than composing music by prompting a computer.

Honestly I agree that the current user interface of Udio and Suno is pretty lame. But it shouldn't take much imagination to see that it won't always be this way. Now that the core capability of generating music is there, a better UI will come. It will be interactive, it will be fun, it will be powerful and sophisticated. It's only a matter of time.

> A M4 with a terabyte of RAM will run GPT-4 level models locally.

Perhaps, but a 1TB RAM upgrade would probably cost about as much as a new car. This is Apple we're talking about.

The original 128kb Mac was about $7,300 (today's dollars).

Apple's Lisa was about $30,000 (today's dollars)

I don't know what the relevance of prices from 40 years ago is, but today they want +$800 just to add +64GB RAM to a machine.

Today, we get a much better deal.

it wasn't long ago that 128GB was a gobsmacking number for a home user

I look forward to checking the "+$1000" box for a terabyte of on-chip RAM

For context, if you get the beefiest MacBook Pro right now you'd pay an extra $1000 to upgrade from 48GB to 128GB RAM. For lower end machines you pay more for RAM, $200 per each 8GB you add on top of the 8GB baseline.

But that's if you want Apple to do that for you. What if you buy the memory separately and put it in yourself? I'm considering doing that for my own iMac, and I saw that a 32 GB memory unit is about $65, so 4 of those $260. For laptops it's probably different, but surely it's going to be less than $1000.

You can't anymore, none of the Apple Silicon Macs have upgradable memory. If you have an older Intel Mac then it might be possible depending on the exact model.

Can't add RAM to any Mac (except the Pro, I think). You have to buy it from Apple with all the RAM it'll ever have. Which is why they overprice the upgrades so much

You can't add RAM to the M series Mac Pro. It's just a Mac Studio with PCI-e slots these days.

Maybe that M4 was the real electric Apple car project.

This seems to suggest that you can install it on a 2019 Mac Pro at least: https://support.apple.com/en-us/102742.

$4,859.99 for 8 128GB LR-DIMMS on amazon: https://www.amazon.com/8x128GB-DDR4-3200-PC4-25600-NEMIX-RAM...

The new Mac Pro (with the M2 Ultra) uses unified memory, so I guess no more third party upgrades? Looking here:

https://everymac.com/actual-maximum-mac-ram/actual-maximum-m...

The new Mac Pro is limited to 192GB while the previous Mac Pro can go up to 1.5TB.

I am guessing they will be working to bring down those prices given how much of an advantage large RAM is for AI. Maybe they could even bring back RAM slots. Although I guess that wouldn't make sense given that bandwidth is also important and HBM doesn't go in slots.

Or they will just include 1TB of RAM when the M10 is ready? I know, I can dream...

Seriously, though, RAM should be getting cheaper/larger over time.

I recently needed a computer to run fusion 360. Options were Mac vs Windows.

I could do a SFF PC or a Mac mini. After looking at the paltry memory/storage Apple chose to give with the mini, I ended up going with an AMD system with 96GB of ram and 4TB of SSD storage for less than a 16GB Mac mini.

> terabyte of RAM

LOL. Have you seem the pricing of Mac memory?

Probably it's like their storage. LTT determined that Apple storage is worth more than it's weight in gold (at least at the Apple prices).

perhaps we should change the saying to worth its weight in Apple ram.

At Mac Pro pricing (128 GB = $1600) that's $12,800

Of course, I can imagine that there will be a significant discount even just considering the price of high end branded memory outside Apple.

Still, Apple has showed no sign of abandoning 8GB for regular people. Even if they switch by the end of the year on their high model, they have a serious handicap in their own installed base.

> Of course, I can imagine that there will be a significant discount even just considering the price of high end branded memory outside Apple.

Apple wanted $3,000 for 160GB of memory for my Mac Pro (i.e. going from a base 32 to 192GB).

OWC sells the same memory (same spec, same manufacturer) now for $620, though I believe I paid about $1,000.

They also wanted another $3,000 for an 8TB SSD. Similarly, I was able to buy a 4xM.2 PCI card, and populate it with 4 x 2TB SSDs for under $1,500. Furthermore my PCI setup is FASTER than the Apple drive (7GBps versus 5.5).

Waiting for the EU to force PCs to ship with at least 16GB of RAM and then Apple to immediately announce how they were planning this whole time to double the starting RAM on their base machines then turn it into a 2h presentation about innovation, sustainability and thetics.