Show HN: Realtime LCM tool | Hacker News Reader

Show HN: Realtime LCM tool

twitter.com

12 pointsdvrp12 comments

Are y'all running the diffusion in PyTorch eager mode?

AITemplate, stable-fast, or even a torch.compile could get that down to 60ms, I bet. Though I'm not sure the example implementations would work on a non SD architecture.

LCMs are actually exactly SD architecture. LCM is initialized from a regular SD unet and finetuned on a new objective. We are already compiling to get to these times. A lot of other people getting sub-100ms times are using fewer inference steps than we do, at a quality tradeoff.

> already compiling

Hmm, well if you mean torch.compile, y'all should still check out stable-fast, which is claiming ~16ms/iter on a 4090, twice that of torch.compile:

https://github.com/chengzeyi/stable-fast#rtx-4090-512x512-ba...

some people even claim they get ~50ms on (powerful) consumer hardware (e.g. RTX 4090).

we still have not been able to replicate those results; also because we want to do it in a distributed way

They might be "cheating" with methods that significantly degrade quality.

i worked on the gpu/infra side of this, so feel free to AMA Ultimately the LCM is just a SD Unet trained with a new objective, so a lot of SD optimizations are transferable to LCMs

i also worked a bit on the infrastructure!

there's a very cool consistent hashing and probability stuff going on for the GPU routing logic but again, i'll write it after this fire ends similar to a post-mortem.

will write an essay about it later, but servers exploding even with experimental alpha users.

I'm excited about what H200s or B100s will mean for technology like this one.

Is it running on H100s or A100s now?

right now A100s, we are still benchmarking 80gb and 40gbs (of VRAM) ones to see what makes more sense to scale this.

Can you use custom images?

yea!

there was another video i put on my twitter account but it's already 24 hours old; the UI has changed a bit.

here's the tweet: https://twitter.com/asciidiego/status/1723804366434627611

also, how's infisical doing?

Reply on news.ycombinator.com