claytonjy

I just implemented something like this in Github Actions for a single python application which includes 2 containers and a model repository, deployed to Kubernetes via Argo from the in-repo Helm chart.

The same semantic version string is in pyproject.toml and values.yaml. These are enforced to be the same via pre-commit hook. Environment-specific versions are in env-specific values files like values-prod.yaml.

I have a build workflow which triggers on PRs that change certain files, computes either a pre- or post-release tag by appending the git hash to the server tag, and pushes the images and model repo using that tag. This will re-trigger as the PR is updated.

Users deploy these pre/post-releases in a few clicks with a manual github action which updates the branch and tag in our dev cluster Argo.

The PR can’t be merged unless the version is bumped.

When the PR is merged, a “promote” workflow runs which re-tags the latest image and model repo with the release version (pre-release without git hash suffix). This is fast, which means there’s no risk of image backoff loop from Argo trying to sync before artifacts are available.

Most PRs don’t touch the version in values-prod.yaml, so merges to main auto-deploy to staging while prod requires another PR updating only values-prod.yaml.

Folks complain a bit about manually bumping the version, and needing 2 PRs to take a change to prod, but it’s pretty smooth otherwise!

do we know roughly how common use of this or similar technologies are in various markets?

I was under the impression this was quite common, though I have no evidence and would love to be shown otherwise!

Looks interesting, but I'm having a hard time understanding what this does that isn't covered by the ruff rules checking for uses of private functions or classes.

It seems you can define an extra level of who's allowed to import what...but why would I want to add that on top rather than using standard conventions? Why would I want to let foo import baz, but not let bar import baz?

Is there any reason to still use Rye, now? looks like this release adds all the things I would have missed from Rye, but I don't think I use all of Rye's features

That's wild, I usually offer candidates closer to 15 minutes (out of an hour interview), and have insisted upon the same when interviewing. Been doing both for a decade, mostly startups.

I used OSMRouter maybe 7 or 8 years ago to process a couple billion routes and it was about as simple as GP described. Just need Docker and an index file. the US one was massive so I kept needing a bigger VM just to load it, but once I did I was able to make HTTP calls over the network to get what I needed. Took a few days to get a working setup but only a few hours to rip through billions of requests, and I was making them synchronously with R; could have been much faster if I was smarter then.

What installation issues have you had with Numpy lately?

Python packaging is a mess, but compared to issues with Torch or Nvidia stuff, Numpy has been a cakewalk whether using pip, conda, poetry, rye, etc.

My first thought was CI providers, where we often have to specify a "service" to allow `docker build`.

I don't know much about the internals there; would this bug allow me to do bad stuff on shared CI runners?

as sibling mentioned the 188GB is for a pair. The memory bump is from enabling the 6th block of memory that is otherwise disabled on H100s. I assume an "NVL box" is still 8 total GPUs, so more like

    8x H100 NVL (94GB, 800W)

the AMD box has a lot more GPU memory

Slurm is definitely still dominant, but OpenAI has been using k8s for training for many years now¹, and there are various ways to run slurm on top of Kubernetes, including the recent SUNK from coreweave²

at my company we use slurm "directly" for static compute we rent or own (i.e. not in a public cloud), but are considering using Kubernetes because that's how we run the rest of the company, and we'd rather invest more effort into being better at k8s than becoming good slurm admins.

¹: https://openai.com/index/scaling-kubernetes-to-2500-nodes/

²: https://www.coreweave.com/blog/sunk-slurm-on-kubernetes-impl...

What did you move to instead?

Would love to hear more about the scale issues you saw. How many workflows or actions was too many? which components started breaking down, what were their failure modes?

That's fair, but they stopped saying it about CV models in 2012. We've been saying this about foundational forecasting models since...2019 at least, probably earlier. But it is a harder problem!

if I knew it was good, why would I tell you that?

And like all deep learning forecasting models thus far, it makes for a nice paper but is not worth anyone using for a real problem. Much slower than the classical methods it fails to beat.

An issue we've had when looking into some of these is that, they provide a layer of software abstraction we're not looking for. I don't want to use some providers bespoke library to wrap my model code; I just want to use NVIDIA Triton, either by providing an image or by providing a model repo. I only want the inference provider to handle the hardware.

I understand that's exactly what those provider _don't_ want, because it means they can't lock us in. But particularly when comparing an inference provider to GCP, where we already run everything in Triton on GKE, I don't want to rewrite my code just to see what their hardware layer is like.

Another complication is we often run multiple tightly-integrated models for a single application, where having them on the same GPU is critical. This is tricky or impossible in some inference-provider-frameworks.

There's too many options for running the latest LLM, and far fewer for running a bespoke set of fine-tuned models on GPUs.

The middlemen you're talking about do two things: buy lots of reserved compute on the hyperscalers, and then pit the hyperscalers against eachother to get better pricing.

If you're reserving thousands of GPUs from the same hyperscaler, even if they're the only cloud you run on, you're not paying the price shown in the calculator. If you have other suppliers, you'll get an even better deal. Then you resell that reserved compute as on-demand compute, somewhere between your costs and what your customers would pay a hyperscaler directly.

What are folk's experiences with alternative cloud GPUs for inference?

If you're doing a lot of model training, buying GPUs or long-term reservations of GPUs is a no-brainer. But when it comes to inference, latency matters and it gets trickier talking between e.g. your AWS infra and your GPUs somewhere else.

It seems lots of providers can give you enough to get by doing inference in a company's earliest stages. But what if I need hundreds or thousands of A100s during peak usage? Is anyone doing this successfully with a non-hyperscaler?

I'm somewhat new to helm, after a year I feel like I'm pretty good at it, but struggle to teach other ML folks what I've learned the hard way over the last year. Some of that is helm, plenty of it is Kubernetes. Most of our charts are ingress, service, deployment, service account, scaled object (keda), but there's plenty of small variation even across ML applications, plus the different needs of other teams.

Any good resources you can point me to on how Kustomize might be easier for us?

Has anyone successfully implemented Backstage at their company? How'd it go?

We're looking at it for an internal platform serving ~50 engineers today, spanning mobile, backend, enterprise integrations, and ML teams. The hope is it'll make it easier to spin up new things "the right way" without e.g. ML PhDs having to learn Helm and K8s, plus other goodies like nice docs.

to reduce the chances of GitHub taking it down, perhaps? not sure it'll help though

And ruff has a bunch of the pyupgrade rules included, which has made it easy for me to start catching things like using List instead of list in py3.10+.

Yes, this is a big part of what makes this approach tolerable. In VSCode, the on-hover tooltip includes a link the ruff rule page with this information, so there's no searching needed.

I've come to rely on that so much, it's really annoying when the error is from mypy, whose tooltips do not have such links.

Patiently waiting for Astral's mypy-killer!

This is the first I've heard of Tuta. How does it compare to the more popular nerd-focused Gmail alternatives like fastmail, proton, or hey?

the person you're responding to is not the OP

CoreWeave owns and runs their own infra, they aren't a middleman.

There are middlemen that can actually get you better prices and availability than you can get yourself, because they buy or rent in volume, and can pit the hyper scalers against each other to get better prices.

If you want even a handful of A100s on GCP you need to buy reserved instances, but lots of resellers will rent them to you hourly, and they have RIs underneath.

have them read blog posts, docs, and client code from Temporal

Yup, looks like a hopeful competitor to Temporal but too scared to name them directly.

if you're using poetry, how does pyenv-virtualenv fit in?

pyenv manages python versions, not virtual environments. venv won't help you grab python versions you don't have, and will happily install your requirements into a venv with the wrong python version.