Show HN: WTFfmpeg – Natural Language to FFmpeg Translator | Hacker News Reader

Show HN: WTFfmpeg – Natural Language to FFmpeg Translator

github.com

88 pointsycombiredd68 comments

Open articleView on HN

In case anyone is looking for a FFmpeg command builder that's not ai-generated:

https://github.com/alfg/ffmpeg-commander

Haven't updated in a while, but it's a simplified web UI with a few example presets.

Doesn't all the web-related ffmpeg stuff boil down to https://ffmpegwasm.netlify.app ?

Edit: no, unrelated. Got confused with https://ffmpeg.app .

This has to be at least the fifth LLMpeg I've seen posted to hacker news in the past few months.

This whole repo is a single 300 LoC Python file over half of which is the system prompt and comments. It's not even a fine-tuned model or something, it's literally just a wrapper around llama-cpp with a very basic prompt tacked on.

I'm sure it's potentially useful and maybe even works, but I'm really sick of seeing these extremely low-effort projects posted and upvoted over and over.

At this rate one could probably automate the low effort project -> HN post pipeline

Hmm … {thinking face emoji}

{money-mouth face emoji}

{Eye roll emoji}

How many Show HN have we seen of low effort genAI image tools that are less capable and less in every way than the umpteengillion previous versions of a genAI tool? People use the Show HN like it's their parent's refrigerator to hang their preschool art looking for validation or something. If anyone was looking to hire someone in AI, would you see one of these subpar projects from a Show HN and think that's someone you would be interested in working with?

Using yet another LLM to generate the project code!

I bet it is very often that people upvote based on the title and perhaps comments, not the actual content or its utility. Besides, this karma business can really get one hooked as a sucker for the high grade…

> This whole repo is a single 300 LoC Python file over half of which is the system prompt and comments.

You accurately described many "AI apps" of this era.

At some point, one would imagine just making a less confusing FFmpeg CLI would be a better use of everyone's time. (I've sort-of understood it now, but the learning curve is pretty steep.)

that's akin to making a "less confusing git" - the reason ffmpeg/git are so widely used is largely due to how powerful and fine-tunable they are. the learning curve is an unfortunate but necessary side effect

It's really not, things can be powerful and easy to use. I'd even say some parts of ffmpeg are well designed in this way like `ffmpeg -i ./video.mp4 ./video.wav` working is well designed.

Think the fact all the commands are shorthand doesn't help because no matter how many ffmpeg commands you copy and paste in your life unless you put the effort in you're not going to begin to remember what -an means in the sea of all the other two letter switches and the copywriting in the output and error messages is very hard to tell whats going wrong for someone who hasn't used it a long time.

Not saying it should all be super wordy just that it's difficult to pick up things though osmosis when the commands look like this -ss -t -rc:v, respect to anyone who actually learnt how this works so they could type it without sitting there with the documentation and hitting a wall for an hour.

Will say though the raw tech inside ffmpeg has always meant figuring out getting it to do the thing has always been worth it because it's insanely powerful.

> It's really not, things can be powerful and easy to use. I'd even say some parts of ffmpeg are well designed in this way like `ffmpeg -i ./video.mp4 ./video.wav` working is well designed.

I'd argue that even this command line isn't well-designed at all; it should be `ffmpeg video.mp4 -o video.wav`. How is it sane that anything without an -i in front is an _output_ filename?

It's not an output filename.. The -i is the input, the .wav is the output filename in that example.

Yes? That's not in conflict with anything I said.

You're right! I misread your "without" as "with". I need better glasses, or reading comprehension.

You could make both git and ffmpeg much less confusing without sacrificing their power and tunability.

Case in point, git is already much less confusing than it was at 1.0 and nobody complains it has gotten less powerful. (“Less confusing” does not mean “not confusing”, of course.)

You could also write a wrapper for non technical users without a rewrite or comprising core functionality.

In my experience that never helps because the wrapper is never comprehensive or well documented enough that you can completely avoid dipping below it and as soon as that happens it's worse because you have to learn the original thing anyway plus some poorly documented wrapper.

It was vibe coded too, the doc comments and pokemon try-catch are dead giveaways. It's a slop wrapper around a slop generator to farm github stars. Welcome to the future.

> pokemon try-catch

I've seen LLM's do this in other languages as well but didn't realize there was a term for it. Wrapping entire function bodies in try/catch, at the very least please just wrap the caller so you don't have to indent the entire body for no reason. Not to mention a lot of commands inside can't even throw.

You can just use ChatGPT, etc already to generate the ffmpeg code. I just did this a few nights ago. There's no wrapper or any need to tune anything as far as I can tell.

Look at it a validation of user demand.

There are whole startups that are just a system prompt and a thousand lines of UI these days...

No one is stopping you from creating a Nobel Prize winning ffmpeg wrapper yourself to show how its done ;)

You aren't doing anything related to ffmpeg at all. The only thing "related" to ffmpeg is the system prompt, which anyone could just paste into their chat window. Please stop posting low effort content like this.

Why is it so hard to read the manual or even a cheatsheet? Many people use ffmpeg, it's not like there's a dearth of information out there...

> Why is it so hard to read the manual

https://ffmpeg.org/ffmpeg.html

Let's be honest, it kinda sucks. The commands are barely explained it feels more left as an exercise for the reader to do the puzzle solving of whats trying to be communicated.

Honestly if these processing chain diagrams just had a rollover where if you roll over parts of the command or the block in the chain and the other part highlighted with a description of what the switch was actually doing then a lot more people would be able to understand this, especially if real world before and after examples of outputs were included.

Instead it's <diagram of the chain> <raw string of the command> "Note: one caveat about something"

I am sure there exist people who live and breath media/codecs and they're reasonably fluent at getting ffmpeg to do what they want because of a tremendous amount of practice.

But for the vast majority of folks who only occasionally use ffmpeg to do something, the complexity of it is so outrageous it feels like a parody. Literally (I mean literally) THOUSANDS of options/flags. It's just too much for a human to navigate. Of course we're going to "cheat" or just google up something similar to what we want. If an LLM can handle it, even better.

I sympathise with the overwhelming sensation of the ffmpeg command line arguments.

But the more you familiarize yourself with a/v streaming and transcoding, you soon realize why you need such amount of control.

I mean, with ffmpeg I can easily combine 3 audio clips, 5 subtitles and a separate video, cut away first 25 seconds and the last 5 minutes of the resulting clip, resize it and change the aspect ratio, reduce audio to mono and specify output codecs for audio and video.

And this is still a pretty simple example of what one could want to do.

Ffmpeg has countless other amazing features, demanding more arguments.

How about for example camera stabilization? (-vf deshake)

How would one even start to explain all of this to an app without thousands of command line arguments?

The whole subject is incredibly complex and ffmpeg is by far the most amazing project in this space.

Without ffmpeg, there would be no youtube in 2005, no plex at all and really the whole of modern social web would probably have happened later if not Fabrice was such a fantastic guy :-)

And still I cannot create .ts files (with mp4 inside) using FFmpeg that will be accepted by my dad's TV media player. I have to put them through Avidemux, because somehow it uses a better TS muxer. More compatible with the TV.

The dumbest part is that this entire app can be replaced by simply piping the help page or man page into the LLM.

Question: what makes creating a GUI for ffmpeg so difficult that none of sufficiently high quality seems to have gained relevance so far?

Also: I understand the privacy concerns, but basically any LLM that's large enough can act as a conversational UI to ffmpeg nowadays. Why would I want to add a specific one to do that?

Instead of LLM, Python and whatnot, it could have been a cheatsheet: https://github.com/scottvr/wtffmpeg/blob/12767e7843b9fd481ba...

I can't help but find it kinda humorous that if the 119 lines of the system prompt wasn't there, it would just be a generic script that takes your input, sends it to ollama and then system("...") the response after some light processing.

https://github.com/scottvr/wtffmpeg/blob/main/wtffmpeg.py#L9...

>You are an expert at writing commands for the `ffmpeg` multimedia framework.

>Respond ONLY with the `ffmpeg` command. Do not add any explanations, introductory text, or markdown formatting.

Fragility of it aside, and the fact that more is written to try and force it to do less, this is basically the gist of the whole thing.

The first example from your link

- User: "convert input.mov to a web-friendly mp4" - Assistant: ffmpeg -i input.mov -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k output.mp4

Isn't exactly a web-friendly mp4. the fast start option is not used. this means the MooV header is at the end of the file instead of the head. that means the entire file must be read/scanned to get to the metadata when the browser requests it which means a long delay depending on the size of the file.

- User: "create a 10-second clip from my_movie.mkv starting at the 1 minute 30 second mark" - Assistant: ffmpeg -i my_movie.mkv -ss 00:01:30 -t 10 -c copy clip.mkv

this is another poor example, as it is again the slowest option by having the -ss after the -i. placing the -ss before the -i will result in the command being faster.

not really sure who is training this system on how to use ffmpeg, but it doesn't fill me with confidence that simple things like this are being missed. after this example, i just stopped looking

> that means the entire file must be read/scanned to get to the metadata when the browser requests it which means a long delay depending on the size of the file.

Not really, unless your server doesn’t support range requests, browsers are smart enough to request the end of the file where a non-faststart moov atom typically lives. But yes, you should use faststart.

You’re right that this appears to be the work of someone who’s not very adept at ffmpeg. Which shouldn’t be surprising; as a power user, maybe even expert at ffmpeg, unless I need to write a complex filter graph, consulting an LLM will just slow me down—people like me have no need for this.

> Not really, unless

That's great if you built the server and know this. If you are just providing files for someone to use in a location you know nothing about, best practices for web-friendly is to have the MooV at the head of the file. Placing it at the end is just asking for trouble. Anyone that's been there done that and experienced MooV at the end one time will always move it to the head from then on. An LLM should know this. Someone training an LLM should know this. Making a tool like this that does not do best practices is just a bad tool. Arguing against best practices verges on making one a tool themself. Don't be a tool

It would be helpful if this printed out the relevant sections from the ’man’ page of the command line options it suggests using, or told the user this option is undocumented in the man page.

This way user could directly review if it is suggesting something they want to go on with.

> --- Generated ffmpeg Command --- > ffmpeg -i my_video.avi -an -c:v libx264 my_video.mp4

The example itself shows a naive conversion, ending up transcoding to default h.264 params. This should have been -c:v copy for copying the input packets, as-is.

No, -c:v copy would copy the video stream from the input .avi (which could be DivX or some other obscure codec for all we know).

I know this from RFTM. I don't ask LLM to second guess me.

Yeah, you're right. Depends on what's in the AVI, of course. If the input is already something that can be housed inside mp4, i.e. standard codecs, the packets can survive by just copying them.

Sure, you can mux DivX into a mp4 container, but don't be surprised if it wont play in various equipment that don't support that specific codec. H264 should be working mostly everywhere, I hope :-)

A rough estimate of the disk space required for the model + all other dependencies would be helpful in assessing this tool's utility. It looks like the recommended model alone is 2.4Gb?

Seems tiny for it to understand natural language, technical terms and their meanings (deinterlacing, pulldown..) and the ffmpeg commands related to it. Assuming it works.

But also absolutely massive for a tool that tells you how to use another, considerably smaller tool. A cheat sheet would be a few kb.

If you don't want learn ffmpeg syntax, is better use a visual gui how handbrake that a frontend for a fatty LLM.

OP here.. I guess nobody got the joke. The last paragraph of the readme flat out says it was intended as amusing performance art.. Ludicrous is as ludicrous does.

  >Disclaimer This was largely made to amuse myself; consider it a piece of humorous performance art but it so borders on being actually useful, I went to the trouble to document all of this. YMMV. Use at your own risk. The author is not responsible for any damage or data loss that may occur from using this tool. Always review generated commands before executing them, especially when working with important files.

I'm on the fence on this one as the HN community thrives on novel interesting and sometimes humorous content, yet you got the ire of most.

Burying the line about it being a useful yet parody of a project at the very end helps no one understand as exactly for that point, this was sold as the latest and greatest, oh disclaimer, it really isn't.

Don't stop the work, but please remember to keep you intentions clear and your audience will understand.

Best of luck on your future projects!

Edit: Typo

Thanks.

I often find that my dry sense of humor and tendency toward the inane lead to a delivery that falls short, coming across somewhere between absurdist humor and some sort of ironic reality, when received by other people. These sorts of attempts at subtle comedy typically result from thoughts provoked by something that was completely serious but I found funny. (Take for example my "peanut buttor vector display" I spent entirely too much time on ( https://github.com/scottvr/PBVD/wiki/Ch-1.-PBVD-%E2%80%90-Th... ), which was inspired by a real news article about edible robots where the inventors were talking about receiving grants for sustainability research or some such, so PBVD made me (and only me probably) laugh and its creation was both reactionary and entirely pointless... but I digress..) I don't think I need to enumerate the points of amusement and commentary I was trying to manifest into existence by making wtffmpeg. It's not that the audience is daft, it's that the comedian isn't funny.

In the big picture, and to this specific post's point - from all the "lolol that's so dumb because" comments I suppose I can call it a success because even if they think they're laughing at me, and it's not the level I'd hoped to reach people, at least it generated laughter. Not unlike the servo-controlled arm I made to "hand"-write the Bible on paper - a critic might focus on the fact that it is not a very good robot, and that there is no need for it to do what it does and entirely miss the intended commentary on technical progress (the invention of the printing press putting biblical scribes out of work, Gutenberg, etc) and I think some of my failure to hit target comes from the fact that these sorts projects technically function, if not perfectly well, distracting from the obvious humor or social-commentary context. This reply has gotten long enough and the list of misunderstood projects coming to mind keeps growing causing me to realize that though I've thought I'm not bothered by such, chalking it up to being "niche", maybe my failure to connect and deliver is at least mildly troubling to me. Lol. shrug

Tangentially related, I have a tendency sometimes to litter my written communications with ASCII winky faces to ensure people know when I'm kidding about something because to my surprise and chagrin, I am often misinterpreted. Again I digress. And thanks again for the thoughtful comment. Your mention of clear intentions resonates.

fwiw I've been dealing with a lot of ffmpeg lately and it's like the most obtuse API I've ever used, and I'm now using Warp for it, and it works amazingly every time

https://github.com/profullstack/transcoder - we should integrate.

Superb. I always refer GPT to learn about what I need to do with ffmpeg. Glad to see someone made an AI wrapper out of it.

PS: Yeah, ffmpeg is not an easy tool to use.

LLMs are pretty good at ffmpeg already. No need to put ridiculous amount of examples.

Actually no, they are not good at remembering cli args precisely. The docs must be in the context

Yes, but from my personal experiences they're pretty amazing at ffmpeg by itself. Figures, considering it's one of the most commonly used and well documented clis.

> wtff "convert my_video.avi to mp4 with no sound"

English is not my mother tongue but I think the model should correct the user that it should be: "convert my_video.avi to mp4 without sound"

All cli programs (or shell itself) should soon have:

- argument syntax autocorrect

- natural language arguments instead of the actual ones should be accepted

- whenever there's an error executing, instead of just erroring out, the error should go through an LLM and output a proper explanation plus suggested fix

Doing command by command seems the wrong way about it though.

Having a fuzzy interpretation of configuration, and fuzzy input/output, on something designed for repeatable tasks? This doesn't sound like a great idea.

If you really want to LLM everything, I'd rather have a dedicated flag that provides correction/explanation of args while doing a dry-run. And another to analyze error messages.

I'm not saying if its a good idea or not, I'm saying it's my prediction of what will happen. Nobody will use old style terminals in a few years where you need to type exactly, is my prediction.

This is what agent do, aichat [1] can do this. What you want is a wrapper for it to pipe the result back to LLM and make sure the command succeeded.

[1] https://github.com/sigoden/aichat

It would be insanity for every cli tool to wrap its own llama.cpp

There are plenty of terminal apps with this functionality, e.g. https://www.warp.dev/

Lollll ‘dd’ with autocorrect will be a hoot

Will be much useful than now, obviously it doesn't run it for you.

How so? A potentially highly destructive command like `dd` (it can literally destroy all your local data in seconds) should be either touched with lot of care and having an idea of what you are doing, or not touched at all. Like some heavy machinery or a scalpel.

Sure, but I'm pretty sure people will still do it.

Yes, but I would not put it in the hand of a robot that might stab you with said scalpel one thousand times per second in the neck.

Reply on news.ycombinator.com