nice! Training models using reward signals for code correctness is obviously very common; I'm very curious to see how good things can get using a reward signal obtained from visual feedback
16 karma · joined April 16, 2024
Revideo is different to Remotion.dev in a couple of ways: First, we use generator functions to describe the flow of animations - every yield within the generator function corresponds to a frame in the video. As a result, our API is very imperative (animations described at the start of the function appear in the start of the video, animations described at the end appear at the end). Remotion's React-based approach is rather declarative - it gives you a frame number and lets you describe what your video should look like as a function of the frame number. Personally, we find our "procedural" API a bit more intuitive and easier to write than the declarative approach, but we might obviously be biased here.
Secondly, we render to the HTML canvas instead of the DOM. Both have advantages and disadvantages: Rendering to the DOM lets you define animations using CSS, which most programmers are already familiar with. On the other hand, an advantage of using the HTML canvas is that it should allow you to render entirely in the browser rather than using server-side rendering, as you can simply capture the current canvas using canvas.toBlob(). We have not yet implemented this for Revideo, but people in our Discord server have made good progress towards it. Also, capturing the frame for videos is a bit faster than screenshotting it (as Remotion does), so our rendering speeds are faster than Remotion's.
Thirdly, we're MIT licensed while Remotion is not FOSS (if your company has more than three employees, you need to purchase a company license to use Remotion). This was one of our original motivations to build our own video editing framework while we were building video products.
I will definitely follow you guys, I'm curious how rendley will develop :) Good luck!
It seems pretty unconventional to offer an sdk where people have to enter a license key, but I understand that it's hard to monetize differently when one of your key features is that you want to enable client-side rendering (not saying that this is bad - I relate to the challenge since we are offering a cloud rendering service and at some point when webcodecs is supported in all browsers, it might make sense to run revideo purely on a client device as well).
However, this is not the only change we made. Our goal is (as stated in the Readme) to enable developers to build entire video editing apps with Revideo. This encompasses a few more things than just triggering headless renders. For example, here are some of the changes we made that were quite drastic and were not possible without forking:
- we export the audio of <Video/> elements in a scene. Enabling this required us to modify parts of the core package: https://github.com/redotvideo/revideo/commit/d0f72b6df68b380...
- we made video-in-video renders a lot faster by requesting video frames from a backend process rather than seeking video frames using the HTMLVideoElement API. This required us to make all draw() functions in Motion Canvas async: https://github.com/redotvideo/revideo/commit/a6e1bcdf0ca8200...
While you can look at Remotion's code on Github (that's what we mean by source-available), the project has a custom license that doesn't grant you the permission to do anything you want with it. Remotion retains ownership of the code (i.e. you cannot simply fork it and give it away for free), and when you are a company with more than three employees, you have to pay Remotion if you want to use their code to render videos
As you mentioned, letting LLMs generate high-quality Revideo code is probably not yet possible, but something I believe could make a lot of sense is using LLMs to choose parameters for your video. Video parameters can get as complex as you like in Revideo (any JSON object is accepted), so you could build a very flexible template that lets you define pretty much any video by using a "universal" JSON representation (essentially, this would be a list of elements such as audio files, videos, images, texts, along with attributes describing the timing, position, etc. of the element)
Letting the LLM generate input variables for your template instead of the code would at least allow you to do type checking and ensure that there are no bugs in your video code - however, I still doubt that the resulting video will be of high quality, LLMs are probably not yet smart enough for this.
To learn how you would build a template for video splitting, feel free to check out this example: https://github.com/redotvideo/examples/tree/main/stitching-v.... It shows how you can render and concatenate videos for desired timestamps (e.g. show only second 5 to 10 of a video)
Given that our goal is mainly to let developers build their own editors, the Motion Canvas editor is not that important for us - we only use it for previewing code changes, so merely projecting the canvas without any of the remaining editor interface would also be sufficient.
I also agree that interactivity is super important. We have not yet started to work on this, but something we definitely need to make easier with the Revideo player is building drag-and-drop editing features (i.e. moving elements around on the canvas to modify their position).
Given that we are not a Retool competitor and operate in very different spaces, I would think and hope that this is not an issue for the Retool team
Ideally, we would like Revideo to be used to build any kind of web-based video editor. A lot of video editors exist, and many of them make a lot of money. Based on my experience, building a video editor from scratch is also really hard - I would think that people would choose a framework that makes it easy to build them if it exists.
My biggest worry is more about the technical difficulty of the problem (becoming the standard way to build a category of products is probably not easy), rather than this being an actual problem. I'd love to hear your opinion though!
First, we use generator functions to describe the flow of animations - every yield within the generator function corresponds to a frame in the video. As a result, our API feels quite "procedural" (animations described at the start of the function appear in the start of the video, animations described at the end appear at the end). Remotion's React-based approach is rather declarative - it gives you a frame number and lets you describe what your video should look like as a function of the frame number. Personally, we find our "procedural" API a bit more intuitive and easier to write than the declarative approach, but we might obviously be biased here.
Secondly, we render to the HTML canvas instead of the DOM. Both have advantages and disadvantages: Rendering to the DOM lets you define animations using CSS, which most programmers are already familiar with. On the other hand, an advantage of using the HTML canvas is that it should allow you to render entirely in the browser rather than using server-side rendering, as you can simply capture the current canvas using canvas.toBlob(). We have not yet implemented this for Revideo, but people in our Discord server have made good progress towards it.
Thirdly, we're MIT licensed while Remotion is not FOSS (if your company has more than three employees, you need to purchase a company license to use Remotion). As described in the post, this was one of our original motivations to build our own video editing framework while we were building video products.