bfirsh

Replicate | Berkeley, CA + Remote | https://replicate.com/

Replicate makes it easy to run machine learning models in the cloud. We've got a ton of open source generative AI models you can run off the shelf, and ultimately we want to become Vercel for hosting custom machine learning models at scale.

We're an experienced team from Spotify, Docker, GitHub, Heroku, and various other places. We're backed by some top investors.

Hiring for:

- ML engineer – somebody who knows how to optimize and deploy ML models

- Infrastructure engineer – somebody to help us deploy them at scale

... and more. More details: https://replicate.com/about#join-us

Email us: jobs@replicate.com

Replicate | Berkeley, CA + Remote | https://replicate.com/

Replicate makes it easy to run machine learning models in the cloud. We've got a ton of open source generative AI models you can run off the shelf, and ultimately we want to become Vercel for hosting custom machine learning models at scale.

We're an experienced team from Spotify, Docker, GitHub, Heroku, and various other places. We're backed by some top investors.

Hiring for:

- Product engineer – somebody who is an expert at building developer tools - ML infrastructure – somebody who knows how to deploy ML models and GPUs at scale - Model builder – somebody to help us implement models (could be part-time if you're doing a PhD or something)

More details: https://replicate.com/about#join-us

Email us: jobs@replicate.com

If you want to run it, this is the main model on Replicate: https://replicate.com/deforum/deforum_stable_diffusion

It's by deforum. Here are links to their Discord, GitHub, and Colab: https://deforum.github.io/

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1]

These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems.

For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker.

I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...)

[0] https://tvm.apache.org/ [1] https://onnx.ai/ [2] https://github.com/replicate/cog

Boom - nice. Here's a fork with that: https://github.com/bfirsh/stable-diffusion/tree/lstein

Requirements are "requirements-mac.txt" which'll need subbing in the guide.

We're testing this out with a few people in Discord before shipping to the blog post.

There's a little "." before "venv/bin/activate" that's easy to miss. I'll update it to "source" to make it more obvious.

Correct. We've been seeing 8GB is super slow, >=16GB is fast. We'll add that to the prerequisites.

Running lstein's fork with these requirements[0] but seeing this output[1]. Same steps as original guide otherwise.

Anyone got any ideas?

[0] https://github.com/bfirsh/stable-diffusion/blob/392cda328a69...

[1] https://gist.github.com/bfirsh/594c50fd9b2e6b173e31de753a842...

Unfortunately this can't run in Docker because Docker for Mac can't access the M1 GPU. (Several layers of virtualization and emulation!)

Nice. We'll get this guide updated for this fork. Everything's moving so fast it's hard to keep track!

We struggled to get Conda working reliably for people, which it looks like lstein's fork recommends. I'll see if we can get it working with plain pip.

It isn't required for some (most?) users. Weirdly sometimes pip is picking up the wheel for `onnx`, sometimes it isn't, and we can't figure out why.

Any Python packaging experts know what's going on? all macOS 12, arm64, Python 3.10. Can't think it wouldn't resolve the wheel.

But yes, good idea to move up. I'll stick it next to the `pip install`.

You can hack on it, modify it, integrate it with other code, etc!

Are you running macOS >=12.3?

Unfortunately this can't run in Docker because Docker for Mac can't access the M1 GPU. (Several layers of virtualization and emulation!)

You can run it on Replicate here: https://replicate.com/google-research/frame-interpolation

You can run it on Replicate too! https://replicate.com/google-research/frame-interpolation

Here's an example of doing img2img: https://replicate.com/stability-ai/stable-diffusion?predicti...

You can also run it on Replicate without a queue: https://replicate.com/stability-ai/stable-diffusion (set “init_image”)

Major contributor to the Docker Python SDK and attempted contributor to a better Docker Go SDK here.

The Go SDK is very low level, and a lot of the high level stuff happens inside the client code in a way that is hard to extract. There was an effort to abstract that in a way that could be reused but it was never prioritized so got stuck on the backburner. I attempted to create a separate SDK that the Docker client didn't build upon, but, if I recall correctly, there wasn't much appetite to do that because then we would have to maintain two codebases and it wouldn't automatically get better when we fixed bugs/improved things in the Docker client.

Here was the prototype: https://github.com/bfirsh/docker-sdk-go

As others say here, Go code does tend to be more verbose, but the Docker Go SDK is very low level and could be higher level. It doesn't have a `Run()` function for example, and there is no technical reason why it couldn't.

You need to put in the herokudns.com address that the CNAME is pointing at – e.g. stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com in our case.

This got us back online. Thank you so much.

Looks like just DNS for the CNAME is broken.

    $ dig @1.1.1.1 stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com A
    ...
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 1232
    ; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 53 20 66 6f 75 6e 64 20 66 6f 72 20 68 65 72 6f 6b 75 64 6e 73 2e 63 6f 6d 2e ("..no SEP matching the DS found for herokudns.com.")
    ;; QUESTION SECTION:
    ;stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com. IN A

I wonder if there is any way to get out an IP address of the Heroku router we were assigned to that we can use in place of the CNAME.

Might be in the logs somewhere, or in Cloudflare somewhere?

You can run it with Cog: https://github.com/replicate/cog

    cog predict r8.im/stability-ai/stable-diffusion -i prompt="cow flying in space"

Or you can run the Docker image directly. More details under "run on your own computer" here: https://replicate.com/stability-ai/stable-diffusion

We have an API for Stable Diffusion on Replicate: https://replicate.com/stability-ai/stable-diffusion

You can do this with Cog! Once you've written cog.yaml, you can run arbitrary commands inside the environment which has CUDA installed correctly:

  $ cog run python train.py

There is a fair bit of overlap.

Cog is optimized for getting a deep learning model inside a Docker image. We found that ML researchers struggled to use Docker, so we made that process easier. It generates a best practice Dockerfile with all your dependencies, and resolves the CUDA versions automatically. It also includes a queue worker, which we found was the optimal way of deploying long-running/batch models at Spotify and Replicate.

Bento is more flexible – the models can be used outside of Docker, and it has built-in support for deploying to lots of deployment environments, which Cog doesn't have yet.

Hello HN! One of the creators of Cog here.

We built this to deploy models to Replicate (https://replicate.com/), but it can also be used to deploy models to your own infra.

Andreas, my co-founder, used to work at Spotify. Spotify wanted to run models inside Docker containers, but Docker was too hard to use for most ML researchers. So, Andreas built a set of templates and scripts to help researchers deploy their own models.

This was mixed in with my experience working at Docker. I created Docker Compose, which makes Docker easier to use for dev environments. We were also joined by Zeke, who created Swagger (now OpenAPI), which is used to define a model’s inputs/outputs. Dominic and some other contributors have since joined! https://github.com/replicate/cog#contributors-

It’s still early days, so expect a few rough edges, but it’s ready to use for deploying models. We’d love to hear what you think.

Hello HN!

Cog lets you package machine learning models in a standard, production-ready container.

We built this to deploy models to Replicate[0], where it has been battle hardened with hundreds of models, but it can also be used to deploy models to your own infrastructure.

Andreas, my co-founder and one of the creators of Cog, used to work at Spotify. Spotify wanted to run models inside Docker containers, but Docker was too hard to use for most ML researchers. So, Andreas built a set of templates and scripts to help researchers deploy their own models.

This was mixed in with my experience working at Docker. I created Docker Compose, which makes Docker easier to use for dev environments. We were also joined by Zeke, who created OpenAPI (née Swagger), which is used to define a model’s inputs/outputs. And, many other contributors have since joined! https://github.com/replicate/cog#contributors-

It’s still early days, so expect a few rough edges, but it’s ready to use for deploying models. We’d love to hear what you think.

[0] https://replicate.com/

I learned the difference between Philips and Pozidrive at school, thank goodness, and this has saved countless stripped screw heads. I am surprised how few people know about it, even professionals.

It’s particularly important when using power tools. An electric drill with Philips bit will instantly munch a Pozidrive screw.

It is the 45 degree markings you’re looking for, in case that isn’t clear: https://shop4fasteners.co.uk/blog/pozidriv-vs-phillips/

Here's a web version if you don't want to read a PDF: https://www.arxiv-vanity.com/papers/2201.00223/