Anyone training multiple ML networks on a single machine?
I have a Mac M1 Max that I'm training 3 TensorFlow networks at once and there seems to be plenty of juice left.
Curious if there are any downsides to doing so.
Curious if there are any downsides to doing so.
Is training set 'sufficiently large' relative to intended use of a trained Tensorflow network?
Are they all using/modifying the same training data input/output concurrently?
Is Max just a terminal server for each of the tensorflow networks?
Each network has two sets of inputs, and I'm running about 480,000 data rows per channel, with about 23 features and 10 targets (these are not LLMs).
Each network is training on the same set of data, with slightly different architectures.
I'm using Keras Tuner to help find the best architecture for each.
They are being trained directly on the Max (not sure if I understood that question).
According to [0], apple doesn't support multiple-gpu's :( Multi-gpu setup (say one gpu per addon card) could allow for each tensorflow network to be run on it's own 'hardware box'/gpu card (and/or distribute/farm out the training to other hardware), In which case, the typical 'max hardware' just becomes a tty user interface layer to various gpu plug-in cards / processes distributed/farmed out.
[0] : https://stackoverflow.com/questions/75236134/training-with-t...
distributed tensorflow:
Scaling TensorFlow Models on kubernetes : https://iamondemand.com/blog/scaling-tensorflow-models-on-ku...
Introduction to distributed TensorFlow on Kubernets : https://techblog.cisco.com/blog/tensorflow-on-k8s
https://medium.com/utility-machine-learning/a-guide-to-distr...
https://medium.com/utility-machine-learning/a-guide-to-distr...
There isn't any other hardware involved other than this one machine; I imagine there was some throttling involved, but I didn't notice it, really, but I think it's because my model isn't that complex.
-------
When I decided to try to do this, I had about 40% GPU usage while training one epoch, and the CPU load was around 35%.
When I added a parallel run, I was getting around 25% GPU load on the other process, and the CPU load went up about 10%.
The third run, I was getting about 9% GPU load on the process and the CPU load went up by another 5%.
-------
This isn't something that I plan on doing continuously either, but I was partly debugging some things and figured I could squeeze in some extra runs.
-------
Thanks for the links to the distributed TensorFlow. I may eventually get around to doing that but at the moment I'm merely trying to validate an idea that I have at minimal cost, and happen to have this machine around practically doing nothing else.