If you can make sure all the resources are namespaced well I’d actually start moving some local workloads to our GKE cluster. Very nice work!
Thanks for the feedback! All resources are namespaced right now, except there is an issue the notebook plugin where namespaces are indeed broken: https://github.com/substratusai/substratus/issues/193 Will get that fixed very soon.
This issue is now fixed in v0.9.0 (check "kubectl notebook --version").
This is really cool, especially with enterprises moving to using LLMs in production. These K8s tools make a lot of sense.
Thanks! Kubernetes provides some solid building blocks for training LLMs. I believe that what is missing is a little tooling to link together those primitives into a streamlined ML DevX.
This is awesome, been needing something like this for a while for our workflows.