Hacker News Reader

Comment by fxtentacle | Hacker News Reader

Surely not. I'm working on a PC with 64GB of main RAM and 12GB of dedicated GPU RAM and I already have to do tricks to be able to squeeze state of the art models into my limited memory.

For comparison, NVIDIA just upgraded from 40GB to 80GB.

View on HN

Unified memory on Apple silicon means the gpu/neural engine has as much ram as the mac. A future mbp with 64 gb ram will be able to fit a lot of models. A whole ML laptop for the price of one gpu.

Agree, if Apple ever goes back to offering 64+ GB RAM Notebooks, they will become a viable choice again.

Contrary to the way you and a lot of people have been spinning things recently, Apple has not discontinued the 16" Macbook Pro, and you can buy an Apple laptop today in all their form factors with just as much memory as you could a few weeks ago. The M1 machines are only replacing models that only went up to 16 GB already.

I thought ML models were tiny, small enough to run on mobile devices.

I believe you have confused training and inference.

Training usually uses large amounts of data to get your system to recognize a pattern. It generally uses huge memory and compute and generates a model.

Inference will use that generated model to recognize the pattern in new data. It uses significantly fewer resources and can run either very fast or on a smaller system.

I believe the speed of inference may be affected by the resources available during training, where more speed/memory for training can produce better models.

I think it's parent commenter who is confused, complains how hard it is to "squeeze state of the art models into my limited memory", which sounds like inference.

Yeah when you've a trained model you can copy it even to mobile devices.

Very, very much depends on the model.

This would depend on the model

Most ML models can be shrunk at the cost of some accuracy, but some models are extremely large.

Reply on news.ycombinator.com