Hacker News Reader

Comment by deeeeplearning | Hacker News Reader

Does anyone doing serious commercial grade ML actually train models on their macbook?

When prototyping or developing new model code, yes. If you’re building production ML systems, there’s plenty code to write and test before deploying to prod infrastructure.

To be clear, you would work with a very small data sample or synthetic data as your objective isn’t to train a model for production use.

Edit: clarity and grammar

Yeah, exactly this. I mostly am using cloud VMs for training, but sometimes just need to be able to mess around on my local machine for my deployment pipeline. I don't really care if it takes ~10 seconds to do inference on a handful of images, I really just need to tinker with the interfaces to ensure everything is hooked up correctly.

Surely not. I'm working on a PC with 64GB of main RAM and 12GB of dedicated GPU RAM and I already have to do tricks to be able to squeeze state of the art models into my limited memory.

For comparison, NVIDIA just upgraded from 40GB to 80GB.

Unified memory on Apple silicon means the gpu/neural engine has as much ram as the mac. A future mbp with 64 gb ram will be able to fit a lot of models. A whole ML laptop for the price of one gpu.

Agree, if Apple ever goes back to offering 64+ GB RAM Notebooks, they will become a viable choice again.

Contrary to the way you and a lot of people have been spinning things recently, Apple has not discontinued the 16" Macbook Pro, and you can buy an Apple laptop today in all their form factors with just as much memory as you could a few weeks ago. The M1 machines are only replacing models that only went up to 16 GB already.

I thought ML models were tiny, small enough to run on mobile devices.

Yeah when you've a trained model you can copy it even to mobile devices.

This would depend on the model

Very, very much depends on the model.

Most ML models can be shrunk at the cost of some accuracy, but some models are extremely large.

I believe you have confused training and inference.

Training usually uses large amounts of data to get your system to recognize a pattern. It generally uses huge memory and compute and generates a model.

Inference will use that generated model to recognize the pattern in new data. It uses significantly fewer resources and can run either very fast or on a smaller system.

I believe the speed of inference may be affected by the resources available during training, where more speed/memory for training can produce better models.

I think it's parent commenter who is confused, complains how hard it is to "squeeze state of the art models into my limited memory", which sounds like inference.

Not yet, but the fact this post exists, that Apple have worked to accelerate TensorFlow, that the Apple Silicon specifically has hardware for these workloads, and they're specifically highlighting the performance on the Mac Pro leads me to think people might be soon.

Let's wait and see what Apple Silicon has in store for the iMac Pro and Mac Pro.

If you think "commercial grade ML" means only big models, that's the wrong way to think already.

You don't have to throw the biggest model you can at any problem you have.

Since this is referencing Tensorflow performance that implies Deep Learning models which are anything but small so I'm not sure I see your point.

Probably very few if any. I think the main use cases would be convenience for playing around with things locally and students.

I use it to make simple models for apps, because that is what I have.

On device ML is better for privacy.

Reply on news.ycombinator.com