Comment by conjecTech | Hacker News Reader

Comment by conjecTech | Hacker News Reader

I've noticed that despite Apple's stated on-device aspirations, they've put very little incremental die space toward the neural engine in the most recent generations. I think there's a number of possible explanations for the disconnect. Curious if anyone with a better perspective knows which are true.

1) They think the existing hardware is powerful enough.

2) The utilization of the ANE doesn't justify increased resources.

3) They plan to re-generalize AI compute through things like vector operations.

4) (the most pessimistic) They are saving large increases for future releases when they need to force upgrades.

I could see the math for on-device just not working for things like massive LLMs. The amount of silicon you'd need to make it possible would be large, and the frequency of use low. The same silicon for one person's phone could likely support dozens if it were in a datacenter instead.

There wasn't a "killer app" for on-die AI accelerators until roughly this March. Until about nine months ago, the only widely-discussed ideas were re-aiming pupils to make your video call feel more natural, and applying background filters.

So today's on-die AI accelerators were, as far as I can tell, a cautious bet. It turns out caution was warranted, because LLM's need large amounts of memory bandwidth and capacity far more than they need a specialized neural compute engine.

Killer app would be Siri that's not a complete disgrace and embarrassment.

I'm not clear on what precisely the Neural Engine can do, but Apple has put plenty of resources into improving their GPUs, which can definitely do compute well (see eg, MacWhisper).

Yeah, it could be that they plan to do #3 through use of their chunky GPUs. There is definitely a lot of focus on the whole APU concept. From look at some of the dies and what projects like ggml have been able to do through mixed computation, I'm guessing the neural engine is ~3-5x the throughput/area vs GPU for the kinds of operations its doing.

Trying to map neural nets onto graphics seems reasonable, but personally I would bet on neuralizing graphics to be the better strategy long term.

I wonder if there will be a separate product line, e.g. iPhoneAI or aIphone, that will have the necessary hardware upgrades with corresponding jump in price, to not have such a cost hit across their entire lineup when the cloud model will do fine for most people.

Reply on news.ycombinator.com