Show HN: Create your own video clips with Stable Diffusion | Hacker News Reader

Show HN: Create your own video clips with Stable Diffusion

neuralframes.com

247 pointsnicollegah72 comments

A few months ago, I started making video clips with stable diffusion and noticed that the tools to do this were too complicated for everyday people. That's why I built neural frames. Enjoy.

Open articleView on HN

Hi all, thanks for all that attention to the site. I am super happy about that. A word of caution: I can only rent 4 GPU instances currently on AWS due to service quota limits. "Unfortunately" the traffic due to hackernews is too high for that. Sorry for any inconveniences. If you are having troubles at the moment, come back later or so. I asked for service quote increase but those take usually 24h or so.

Also, very little people actually pay so I can only afford so much.

I don’t usually comment, but aws is actually rather expensive and i’ve hit this annoying quota problem as well. Been back and forth and still hasn’t been raised. I can also recommend coreweave banana.dev and pipeline.ai as great alternatives. This service looks awesome good luck with the launch!

hey! FYI i think this limit is usally per geograhic site, so if you can only rent 4 in us-east-1 try us-west-1 etc

ah sick, i didn't know that. thanks!

Great project! We can probably host this for you for free until you work out your AWS credits. Email me at community@together.xyz.

The more AI-generated art I see the more convinced I am that it will generate entire new modes of art creation, rather than make creative work redundant.

I'm now waiting for a creation that could not have been done without AI, the labour that would be involved to create these works manually not being considered.

I've been experimenting with creating image datasets via Stable Diffusion, then training StyleGAN2, and compositing in After Effects. It's a new mode of art creation for me that I cannot recreate otherwise. https://www.jasonfletcher.info/vjloops/

This is some truly amazing art. I kept reading one post after another of yours.

Awesome! It reminds me of time-lapse movies in developmental biology, showing, for example, the various shapes an organism has as it develops.

This takes me back to the demo scene days. Seeing something that’s both a technical and artistic achievement that could only have been borne out of the cutting edge. Thanks, awesome stuff.

One video brought me memory of Autechre - Gantz Graf music video (warning, hard IDM style ;) [0]. It was made manually in 2002, per Wikipedia: "Rutterford also stated that there was no generative element to the imagery; every three-dimensional object in the agglomeration was painstakingly and manually synchronised with a specific element or frequency range within the track" [1].

[0] https://www.youtube.com/watch?v=ev3vENli7wQ

[1] https://en.wikipedia.org/wiki/Gantz_Graf

Very cool!

I wonder if this could be merged with the work done to build music from inverted FFT’s to do Aphex Twin type visualization, or alternatively visualizations through oscilloscope music like Jarobeam Fenderson.

Very cool! I just released a tutorial on how to do this using the computerender api!

https://github.com/computerender/tutorials/tree/main/python/...

If you’re interested in saving money on expensive cloud gpus, our api is much cheaper than this. (only $0.001-0.0025 per frame)

looks cool. somehow the website looks screwed up in my browser (chrome) and I cannot get an API key after signing up.

Hm, what device/os are you browsing on? The site should be mobile-friendly except for the account page. Also please feel free to reach out by email or discord.

How is it so cheap? I think that's a factor of 3 cheaper than others.

The GPUs are rented from vast.ai The individual machines aren't as reliable or well-integrated with other cloud services as typical cloud machines, but multiple can be put behind a queue to create a highly reliable service.

do you plan on adding dreambooth? I would give it a try.

I would like to support dreambooth, if there is a way to store the fine-tuned models more efficiently. The challenge is that each trained model is quite large and a bunch of models can't fit into one gpus memory at once.

textual inversion maybe? lightweight embeddings and I haven't seen any API offering it at the moment.

Try the Lora fine-tuning method with few MB results:

https://github.com/cloneofsimo/lora

This looks great! Thanks for sharing.

I suggest that people simply run Stable Diffusion Deforum themselves (there's an extension for automatic1111's web UI). You can run it in a google colab notebook and the cost will likely be cheaper or the same, though I haven't bothered to compare.

Reminds me a lot of the "Ghost" music video for by Gunship[1] made by aiplague[2].

1. https://www.youtube.com/watch?v=aUJuwNxNUWQ

2. https://aiplague.com/

Reminds me of something you would see in a newer modern remake of Decasia.

https://en.wikipedia.org/wiki/Decasia https://www.youtube.com/watch?v=hDa-mmSldDg

Very cool, your demo video looks great, would love to try it when it's working again. You can also do this type of thing on colab with Deforum Stable Diffusion: https://colab.research.google.com/github/deforum-art/deforum...

I've been messing with it myself, here is an example: https://youtu.be/FsVskNtNazk

It should absolutely work now. Doesn't it?

Seems to be overloaded.

"There's a lot of requests currently and our servers are overloaded - sorry. I am trying to increase the capacity. Please try again later."

I put it on for hundreds of hours to try out a 24 fps video myself: https://www.youtube.com/watch?v=f3GfUKJBUYA

Sadly YouTube’s compression absolutely crushes the quality out of your video. Love the concept though.

Pretty cool, I would upscale to 1080p or even resize as youtube will compress it better or less for hd videos.

Couple suggestions after playing with this:

1) the experience mostly works on mobile (where I first tried it). With only minimal changes in the fixed sizing I think you could make this mobile friendly.

2) for posts shared via your Twitter, would be interesting to see details about the prompt(s) used vs “new post”

3) I’d like to have a bit more customization in the options

Overall really nice and good luck with monetizing it. I’d love to see a blog post write up on the technical implementation. That’s something I’d more be willing to pay to see personally.

Those are very good suggestions. I built it with mobile in mind but somehow I failed to get it right, yet. Will continue to work on it. Would also be nice to make a native app at some point.

Yannic kilcher had created a videoclip for lyrics made of imagenet labels last year: https://www.youtube.com/watch?v=rR5_emVeyBk

This is very cool! What sort of values are you using for the denoising strength/guidance scale? Each frame is a nice level of similar/different to the last to create flowing video.

Nice! This is much simpler than using Deforum

Is everything set up on the cloud yourself, or do you use an API?

I've set it up myself. The APIs I saw didn't seem suitable for this real-time inference thing.

I’d much prefer to run this locally on my own. Any chance you’re willing to share code (or suggestions / links since it looks like you might be trying to monetize this per your Twitter posts)?

I understand and have been thinking about it. Would somehow like to monetize this but if it doesn't work out I might just open-source it.

You might be able to do both. Lots of people don't have high-end video cards or an M1, and will pay for the convenience you offer.

Completely agree

What were the APIs you looked at and how were they not suitable?

Curious to know, since I'm currently developing a stable diffusion API at Evoke: https://evoke-app.com/

Is there an example video somewhere?

Edit: Nevermind - I don't have autoplay active and there was no way to see that the "image" on the landing page is actually a video (no UI like a play button).

it's on the landing page directly. You can also check https://twitter.com/neuralframes

Lol did you mean to past that last one on your twitter? I think you might need some content filtering.

hahaha, i think you're right.

What prompts did you use to make the video?

The description of what you have to do reminds me the instructions on how to draw an owl (https://knowyourmeme.com/photos/572078-how-to-draw-an-owl)

What will be interesting is whether we will soon have a model that can actually follow these steps.

The site should have a demo video of what an output looks like instead of a static image.

I stopped at the signup form.

It's a video, not a static image. You should try reloading the page.

I don't see an image or a video... :shrug:.

I’m just getting a static image on iPhone as well.

Is there a youtube channel?

It's an embedded mp4 file:

https://neuralframes.s3.eu-central-1.amazonaws.com/202212141...

Might want to put notice:

“There's a lot of requests currently and our servers are overloaded - sorry. I am trying to increase the capacity. Please try again later.”

…prior to making users do multiple clicks, opt to not provide an email, etc.

Even a prerecorded demo posted to Youtube would be a better experience.

Might want to put notice:

“There's a lot of requests currently and our servers are overloaded - sorry. I am trying to increase the capacity. Please try again later.”

…prior to making users do multiple clicks, opt to not provide an email, etc.

Then, if I finally try to render a 5 second click I get:

“There's high demand on the servers currently. Sorry for any inconvenience. I am trying to scale up the servers.”

RIP

...no noise overlay on the input images to at least get some sort of frame by frame consistency? -.-

Do you have some resource to learn how this works? I'd love to implement it.

It's baked into Deforum, the bit you'd want to look into here is the recent changes for Perlin noise.

The company that makes stablediffusion is being sued for copyright infringement. Don't use this.

Everyone’s being sued for something somewhere. You can use this.

That’s a strange statement.

Who cares about frivolous lawsuits. Stable Diffision and the like tools will prevail. The main hope is that it will be their OSS versions and not corporate (Dall-E).

The works and ways of lives that are threatened by SD and the like are not worth preserving

I wouldn't call them frivolous. Quite a few people believe training generative AI models on their copyrighted work and providing others with access to that model violates their copyright. IANAL but it seems a weak argument to me. And it seems weak to some IP lawyers I know as well.

It also seems like technology that can't be realistically bottled back up. However, I wouldn't call lawsuits frivolous and it might actually be useful to get some legal clarity

I have yet to hear a copyright lawyer who actually understands how these tools work say anything other than that it's going to be fair use; the ones that I have seen comment on it as such seem to have extremely misunderstood how the tech works and how the images are used, they've just regurgitated the arguments that the artists opposed to it make (about how it copy and pasted images, etc).

Maybe I'm wrong, also not a lawyer, but I've heard from enough at this point (including ones that I've paid and are actually our lawyers as we evaluate internal use of the tools) that I definitely wouldn't bet on these suits succeeding.

Dumbest take I've ever seen. Congratulations.

sucks to have vested interests in the subject I suppose. And shilling for some no-good lawsuit across hackernews

It sure beats being stupid enough to leave a trail of self-incriminating evidence all over social media.

(You're not as smart as you think you are.)

It would be like banning Bittorrent at this point.

Stable Diffusion - or something functionally identical - is here to stay.

I would be shocked if there were any significant tech companies that are not currently being sued. Frivolous (or maybe even not so frivolous) lawsuits are a fact of nature, best not to waste time worrying about them until you are the target.

And Google was sued many times back in the day for the same thing, because of their indexing of the web. They won out under fair use for the exact same reasons that all of these lawsuits will fail, the way these images are used falls very clearly within the dead center of fair use, and is probably even less questionable than Google's, since Google really does store and display direct copies of pieces of the content that it organizes.

the filing of the lawsuit shows a clear lack of understanding by those doing the suing, it will be thrown out almost immediately unless brought to a sympathetic judge, which is unlikely since they filed in the bay area.

Reply on news.ycombinator.com