Drawvg Filter for FFmpeg | Hacker News Reader

Drawvg Filter for FFmpeg

ayosec.github.io

184 pointsnolta30 comments

drawvg is very useful. Before drawvg, I was always fine using the stable FFmpeg releases such as 8.0.1 but when I saw drawvg added to the master branch[1], I didn't want to wait for the next stable release and immediately re-built ffmpeg from master to start using it.

My main use case is modifying youtube videos of tech tutorials where the speaker overlays a video of themselves in a corner of the video. drawvg is used to blackout that area of the video. I'm sure some viewers like having a visible talking head shown on the same screen as the code but I find the constant motion of someone's lips moving and eyes blinking in my peripheral vision extremely distracting. Our vision is very tuned into paying attention to faces so the brain constantly fighting that urge so it can concentrate on the code. (A low-tech solution is to just put a yellow sticky know on the monitor to cover up the speaker but that means you can't easily resize/move the window playing the video ... so ffmpeg to the rescue.)

If the overlay was a rectangle, you can use the older drawbox filter and don't need drawvg. However, some content creaters use circles and that's where drawvg works better. Instead of creating a separate .vgs file, I just use the inline syntax like this:

  ffmpeg -i input.webm -filter_complex "[0:v]drawvg='circle 3388 1670 400 setcolor black fill'[v2];[0:a]atempo=1.5[a2]" -map "[v2]" -map "[a2]" output.mp4

That puts a black filled circle on the bottom right corner of a 4k vid to cover up the speaker. Different vids from different creators will require different x,y,radius coordinates.

(The author of the drawvg code in the git log appears to be the same as the author of this thread's article.)

[1] https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/016d767c8e9d...

Couldn't you use a video player like mpv to achieve a similar effect? Not sure if you can cover a specific part of the image but you sure can crop the video however you want and bind the commands/script to a key.

>mpv to achieve a similar effect? Not sure if you can cover a specific part of the image but you sure can crop the video

mpv doesn't run on iPad so it's better for my situation to just burn the blackout into a new video. I actually do a lot more stuff than drawvg (also rescale, pts, framerate,etc) in filter_complex but left the rest of it out for the HN comment so the example is more readable.

I suppose it might be possible to use mpv with a custom shader mask glsl code to blackout circular areas of the screen.

>you sure can crop the video

Cropping the video is straightforward with no information loss when the geometry of presentation and the speaker is laid out like these: https://www.youtube.com/@MeetingCPP/videos

But cropping the following video by shrinking the boundaries of the rectangle until the circle overlay is not visible would result in too much of the text being cut off: https://www.youtube.com/watch?v=nUxuCoqJzlA

Scrub that video timeline to see the information that would be chopped off. For that, it's better to cover up only the circle overlay with a blacked out disc.

> mpv doesn't run on iPad so it's better for my situation to just burn the blackout into a new video.

I would love to stop using the YouTube client on iPadOS. Do you just d/l the video with yt-dlp+ffmpeg and then post process it based on your needs and then watch it from the Files app from iCloud or whatever?

With mpv you could draw over the video with libASS, but it would be more straight forward to use ffmpeg filters from mpv or otherwise.

The quick way I'd solve that is to open any program window (like calculator or whatever), mark it as being on top of other windows, and resize it and place it on top of that area. It seems quick and easy enough for the effort.

Cline just added a useless Note icon animation wherw it used to say "Thinking..." or whatever to signal contwng incoming. I think its nothing about faces. Our UI/UX designs put action items in lower right. Youtube skip button the most prominent..

The cropdetect example made me wonder if they're thinking about including support for yolo or similar models. They're including Whisper for text to speech already, I think yolo would enable things like automatic face censoring and general frame content aware editing. Or maybe Segment anything, and have more fine grained masks available.

On the other hand, when I compared the binaries (ffmpeg, ffprobe, ffplay) I downloaded the other day with the ones I had installed since around September, they where almost 100 MB larger. I don't remember the exact size of the old ones but the new ones are 640 MB, the old ones well under 600 MB. The only difference in included libraries was Cairo and the JPEG-XS lib. So while I think a bunch of new ML models would be really cool, maybe they don't want to go down that route. But some kind of pluggable system with accelerated ML models would be helpful I think.

One of the strangest discoveries of my life was that vector graphics is a solved problem, and the solution is turtle graphics that I was taught in primary school.

I still have a copy of Hobby Electronics from 1982 that my dad bought when I was about 9 or so and in primary school, with the article on building Hebot II.

I got various bits and pieces together and experimented with doing things like driving Big Trak gearboxes (remember? J Bull Electrical used to advertise them in every magazine, along with all sorts of fascinating old shite) with an interface plugged into my ZX Spectrum, but I never actually built one.

Funnily enough I was thinking about that the other day, and how sad it is that schools like my son's primary school just have very locked-down iPads for the children to use instead of the BBC Micros we grew up with (I'm guessing you're more approximately my age than primary school age, and those things were in schools well into the early 2000s. Bombproof.) that could be endlessly tinkered with.

Anyway the guy next door does a lot of 3D printing and it's never been easier to draw PCBs and get them made or even etch them at home (it's the drilling bit I hate). So maybe now EBv2.0 is five, it's time to dig out that issue of HE and start transcribing stuff into Kicad and Blender :-)

Well turtle graphics is not implemented in drawvg. But it should be easy to implement it.

SVG path syntax [1] mentioned as being an inspiration in the article is very similar to Turtle graphics (move, lineto, etc.)

[1] https://developer.mozilla.org/en-US/docs/Web/SVG/Reference/E...

Very many vector graphics standards use the idea of a current position. A distinguishing feature of turtle graphics is to have a current direction as well.

It's a shame SVG doesn't. Many shapes can be specified much more concisely.

plotter languages have the same concept.

Which ones? I've just skimmed through docs for HP-GL, DMPL and Gerber, but I can't see anything.

I was trying to do a bit of generative art for a pixel display in my window, and feeling creatively stuck, then realized I wanted turtle graphics. Implemented that and I was unlocked.

This doesn't make any sense. Vector graphics aren't specified by rotating and walking a turtle, and that is obviously a tiny tiny trivial part of the vector graphics field. Try rendering a vector font directly on a GPU and see how far RIGHT 45 FORWARD 10 gets you.

Pretty far? No idea about the sota in GPU vector rendering, but if you triangulate shapes, RIGHT 45 FORWARD 10 would produce a vertex buffer of points on the CPU, which could be rendered on the GPU.

GPU font rendering isn't done by converting the letters to polygons like that. That would be prohibitively expensive.

That's quite cool. I guess ffmpeg would kind of technically be a replacement for avisynth at this point.

Still, I find the syntax it uses horrible:

  ffmpeg -an -ss 12 -t 3 -i bigbuckbunny.mov -vf 'crop=iw-1, drawvg=file=progress.vgs, format=yuv420p' -c:v libvpx-vp9 output.webm

I understand that most of this comes from simplicity of use from the shell, so if you take this point of view, the above makes a lot of sense.

My poor, feeble brain, though, has a hard time deducing all of this. Yes, I can kind of know what it does to some extent ... start at 12 seconds right? during 3 seconds ... apply the specified filter in the specified format, use libvpx-vp9 as the video codec ... but the above example is somewhat simple. There are total monsters in actual use when it comes to the filter subsystem in ffmpeg. Avisynth was fairly easy on my brain; ffmpeg does not, and nobody among the ffmpeg dev team seems to think that complicated uses are an issue. I even wrote a small ruby script that expands shortcut options as above, into the corresponding long names, simply because the long names are a bit easier to remember. Even that fails when it comes to complex filters used.

It's a shame because ffmpeg is otherwise really great.

Maybe it helps to view ffmpeg as a DSL and the ugliness is caused by the CLI constraints / conventions.

For what it's worth, LLMs are a great tool for both composing and understanding ffmpeg commands.

And if you want something more verbose / easier to read you can use something like https://github.com/kkroening/ffmpeg-python (with LLMs) as well

It looks like that project is dead, some googling turned up this one which seems active and popular

https://github.com/pyav-org/pyav

They are not comparable, ffmpeg-python just abstracts away the CLI, pyav is a low level binding of the ffmpeg libs.

It may seem "dead" but ultimately it just helps you build CLI commands in a more sane way, the CLI interface to ffmpeg has been consistent for a long time. Only thing that may change is individual filters which you can just give raw to ffmpeg-python.

I remember when I was heavily using it last year I found a fork that seemingly had more sane typing or something but since LLMs last year didn't know about the newer lib but could write decent ffmpeg-python code I stuck with it and it did the job.

This could be a great and easy way to add annotations to technical instruction videos in a way that is self-documenting and programmable.

Nice readme example, I would also like to see time of execution for these tasks, I mean how much time for handling filters adds comparing to non using it

Didn't know this was a thing- thanks for posting.

Is there something similar that supports shaders? Like metal / wgsl / glsl or something?

Sounds like a fun project...

At this point we almost need an entire GUI video editor frontend just to understand and properly drive ffmpeg itself.

Looks very cool. I wonder if drawvg filters can be modified in real time using zmq.

How far are we from having ffmpeg make automatic rotoscope versions of our videos?!

Oh, I was not expecting an entire DSL for this. This looks really useful.

Reply on news.ycombinator.com