Comment by adolph | Hacker News Reader

Comment by adolph | Hacker News Reader

Apple's been making hay with "neural engines" in their hardware for quite a while. It is curious why Intel hasn't made a bigger splash with its efforts, which go back a ways and have included (semi?) cancelled things like the RealSense from aquisitions like Movidius (whose chips also formed the core of early products like the Google AIY Vision Kit; AIY still being marketed as a deployment target for Coral).

I guess that: 1. Apple has been able to capitalize on their constrained stack of HW/SW. 2. Intel has struggled with being the unpopular dish at a well optioned buffet.

On-device ML inferencing is going to be an interesting space.

Your iPhone, iPad, Mac, and Apple TV make use of a specialized neural processing unit called Apple Neural Engine (ANE) that's way faster and more energy efficient than the CPU or GPU.

https://www.makeuseof.com/what-is-a-neural-engine-how-does-i...

This year at WWDC 2022, Apple is making available an open-source reference PyTorch implementation of the Transformer architecture . . . specifically optimized for the Apple Neural Engine (ANE), the energy-efficient and high-throughput engine for ML inference on Apple silicon.

https://machinelearning.apple.com/research/neural-engine-tra...

Intel RealSense Technology, formerly known as Intel Perceptual Computing, was a product range of depth and tracking technologies designed to give machines and devices depth perception capabilities. . . In August 2021 Intel announced it was "winding down" its RealSense computer vision division to focus on its core businesses.

https://en.wikipedia.org/wiki/Intel_RealSense

The AIY Maker Kit . . . Using a Raspberry Pi and a few accessories, our tutorials show you how to build a portable device that can identify objects, people, and body poses with a camera, and recognize voice commands or other sounds with a microphone. . . [using] TensorFlow Lite models. . . .

https://aiyprojects.withgoogle.com/maker/

Very true. Apple is in a prime position to make strides in language model development on their hardware. An Apple-designed LLM that runs locally would be amazing. Especially if they could integrate it with iOS or macOS.

While I’m very excited for an LLM powered Siri that has access to my calendar, email, and messages, it’s worth noting that llama.cpp is using metal for acceleration and the M2 Mac Studio is one of the best values out there for running very large models like llama2 70B derivatives or Falcon 180B. The shared ram/vram allows you to load huge models and it’s got decent tokens/sec performance.

In the last few weeks, the initial prompt evaluation step got much faster with llama.cpp so everything that depends on it like the python library and lmstudio are faster as well. I’m very happy with my purchase at this point and it seems to keep getting better.

I'm curious about this memory things with the Macs. I was looking at Airs with M2, and the memory is only 8G. Is that the limit, if I'm say... running containers and applications?

Where can I learn more?

I don't understand how one can ship a computer today with only 8G of mem. My phone has more than that. Are developers able to work with this amount of mem, or are they forking out $3k + for a Pro with 24G?

I went with a Framework because I got good hardware for way less money

My previous MBP had 32 GB of ram and it was constantly swapping. Now I have 64, and it's holding up. I guess it really depends on your work environment, but I need Docker, IntelliJ, plus a bunch of other stuff, and it quickly adds up.

I don't know how anyone can use a laptop with only 8GB of ram. Maybe if all they do on it is Facebook and Youtube ¯\_(ツ)_/¯.

> Apple is in a prime position to make strides in language model development on their hardware.

Ok, I'll take the bait and disagree with your agreement.

Apple has a good thing going for now. However, their tightly coupled HW/OS and insular corporate culture limit their reach. LLM is a good example of this. They may have their own cooking in some secret room. Or not. What is the value=(reward/risk) of making something like an on-device LLM that Apple might do? Maybe Apple will acquire you or maybe you get Sherlocked.

As an example, Siri was an acquisition and neat trick, but it's barely improved since 2011 (I'm still a basically satisfied many x daily user). Although Apple has changed lots since 2011, it hasn't demonstrated an ability to execute something like meaningful integration and improvement of an LLM. Therefor I would not agree that "Apple is in a prime position to make strides in language model development on their hardware."

Intel hasn't been able to read industry trends for decades now.

Reply on news.ycombinator.com