Ffmpeg Buddy
evanhahn.github.io
evanhahn.github.io
- play facetime camera (or any avfoundation camera, like a capture card) to mpv
ffmpeg -f avfoundation -video_size 1920x1080 -framerate 30 -drop_late_frames true -i "FaceTime HD Camera" -c:v rawvideo -f matroska -r 30 - | mpv -profile=low-latency -untimed -
ffmpeg -f avfoundation -video_size 3840x2160 -framerate 23.98 -pixel_format uyvy422 -drop_late_frames true -i "Cam Link 4K" -c:v rawvideo -f matroska - | mpv -profile=low-latency -untimed -
ffmpeg -f avfoundation -video_size 3840x2160 -framerate 23.98 -i "Cam Link 4K" -c:v hevc_videotoolbox -q:v 85 -r 23.98 -tag:v hvc1 <filename>_x265.mp4
mkfifo /tmp/fifo1
cat /tmp/fifo1 | mpv -profile=low-latency -untimed -
ffmpeg -f avfoundation -video_size 1920x1080 -framerate 30 -drop_late_frames true -i "FaceTime HD Camera" -c:v rawvideo -f matroska -r 30 - > /tmp/fifo1Overall I just found it better to leave pretty much everything to the more battle-tested `ffmpeg` and then use mpv to play the content (or can also use ffplay, but I do have have some customizations in mpv which I like).
Is that really true!? 1.8 GB per minute?
Ever since I started playing with video recording I ended up getting a NAS with 10Gb/s ethernet card, definitely helps!
ffmpeg -f avfoundation -i ":0" -flush_packets 1 -flags low_delay -muxdelay 0.01 -fflags nobuffer -f wav -c copy - | mpv - --profile=low-latency -untimed --lavfi-complex="[aid1]showcqt=fps=30:s=1280x720:count=5:axis=1:axis_h=6:bar_h=100:timeclamp=0.13[vo]"
here's a screenshot: https://dl.dropboxusercontent.com/s/yx5rlgpsmai3kej/showcqt_...
@echo on
setlocal
set ID=IDIDID
set VIDFILENAME=%date:~-4,4%%date:~-10,2%%date:~-7,2%_%time:~0,2%%time:~3,2%%time:~6,2%
echo "-segment_atclocktime 1 "
ffmpeg -loglevel error -rtbufsize 150M -y -f dshow -i video="YOUR HD Camera" -r 10 -f segment -segment_time 1800 -segment_format mp4 -vcodec libx264 -preset fast -threads 1 -crf 28 -tune zerolatency -thread_type slice -slices 1 -intra-refresh 1 -r 10 -g 60 -s 640x480 -aspect 4:3 -b:v 512k -minrate:v 256k -maxrate:v 2.5M -bufsize:v 5M -pix_fmt yuv420p -vf "drawtext=fontfile=/Windows/Fonts/arial.ttf:text='%%{localtime\:%%x %%X}': fontcolor=white@0.8: x=7: y=10" %ID%_%VIDFILENAME%-%%03d.mp4
So many bad ffmpeg commands, it's hard to filter them by quality
Here's what I use to GPU encode/decode multiple IP camera feeds using motion on a 15W haswell system with enough CPU left free for other tasks.
#motion.conf
movie_extpipe_use on
movie_extpipe ffmpeg -y -f rawvideo -pix_fmt yuv420p -video_size %wx%h -framerate %fps -vaapi_device /dev/dri/renderD128 -i pipe:0 -vf 'format=nv12,hwupload' -c:v h264_vaapi %f.mp4
netcam_params decoder=vaapi,rtsp_transport=tcp,keepalive=off,tolerant_check=onIf you use -2 instead it will adjust so that it is even.[^1]
For example:
ffmpeg -i input.mp4 -vf scale=720:-2 output.mp4
This has saved me a lot: https://stackoverflow.com/a/53024964/511200
I suspect you could still make it work with clever usage of the File System Access API as a cache, and process larger files in chunks. Then you’d mostly be limited by the Blob storage limits [1], and memory required to merge processed chunks together.
[0] https://nft-inator.com/app
[1] https://chromium.googlesource.com/chromium/src/+/224e43ce1ca...
https://wasm-audio-clipper.netlify.app/
Check the repo: https://github.com/marclundgren/wasm-audio-clipper. I ended up lazily registering the context. The memory footprint stays lower even after trimming multiple clips on the same session.
But the execution here is a bit lacking. This only lets you input basically 10 parameters.
If this were to be expanded to include all FFmpeg params, it would be INCREDIBLY useful.
The filter chain system, for instance, is quite complicated.
But of course there could be some compromise.
Very Much this. Even the `ffmpeg` incantations I use frequently are tough to recall -- like creating a gif
`ffmpeg -i input.mkv -vf "fps=8,scale=512:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 output.gif`
How can I mux arbitrary data streams into a container? Let's say I have a drone and I want to stream audio, video, and some avionics telemetry.
I know FFMPEG can pass through data encoded, and it lists a few binary stream formats:
ffmpeg -codecs | grep '^ ..D' # shows all -c=d codecs
..D... bin_data binary data
..D... dvd_nav_packet DVD Nav packet
..D... epg Electronic Program Guide
..D... klv SMPTE 336M Key-Length-Value (KLV) metadata
..D... mpegts raw MPEG-TS stream
..D... otf OpenType font
..D... scte_35 SCTE 35 Message Queue
..D... timed_id3 timed ID3 metadata
..D... ttf TrueType font
Although it's not clear if you want to display your additional data as an overlay (like this example) or keep it as some kind of metadata stream. I'm not sure if the second option is even a thing.
It's definitely a thing that exists, I've seen stackoverflow issues talking about working with containers that have these sorts of streams, but for some reason, I've scoured the net and haven't found almost anything about actually writing those streams.
ffmpeg -i input1.mkv -i input2.mkv -i input3.mkv -map 0:v -map 1:a -map 0:d -c copy output.mkv
...will take the video from input1.mkv, the audio from input2.mkv, and the data stream from input3.mkv. If you telemetry isn't already packetized, you'll need to do that yourself. Converting your drone's telemetry into something that ffmpeg likes is generally non-trivial.
At my day job I maintain a tool that takes a .csv file and a raw video file and outputs a single file. Which can then be viewed, with the data displayed on the map and such. https://pro.arcgis.com/en/pro-app/latest/tool-reference/imag...
> if you telemetry isn't already packetized, you'll need to do that yourself. Converting your drone's telemetry into something that ffmpeg likes is generally non-trivial.
Yes, this is exactly the non-trivial step I'm struggling with. Let's say I have every time step a msgpack-encoded binary blob with a timestamp. I want to packetize it and pack it into a data stream of some timestamped/sequenced packet/page/atom structures. I might be able to build those packets myself manually if I can find a viable format (mkv and ogg seem the easiest to reason about).
And ultimately I do want to do this with real-time streams, but I'll definitely look at this Video Multiplexer to start with! That at least scratches the itch of archiving this data.
There does exist a standardized way to encode the data: MISB (Motion Imagery Standards Board) ST 0601: https://nsgreg.nga.mil/doc/view?i=5093
Unix has something called pipes that lets you send data as 'files' but they're really supplied as a stream by another process.
Even if you could do less I/O by keeping stuff in-process - and I'm not sure that's true if you build your graph in ffmpeg, it's just the source and sink of data which are external, and they'll almost always be external.
However, I’d note that in my use case, I ended up going back to the shell after running some benchmarks and seeing that for my use case, I wasn’t saving any performance.
(And sometimes there are oddities about how they interact, or there is one option that seems very deprecated. It's like Eclipse, it's an engineer's tool, it's extremely powerful but god had no hand in the creation of the UX whatsoever, it is the unholy amalgamation of hundreds of modules that usually work, usually, mostly.)
Give me checkboxes for "resize this video", "convert framerate", "decomb filter", "burn hardsubs filter", a reasonable set of x264 or x265 transcoding options, audio transcoding and downmux, and options for controlling how it all gets reassembled.
It might help to list out some simple use-cases and then look at the ffmpeg features that would be required to get there, and then support those. But again, it's sort of hard to capture everyone's use-cases, ffmpeg is such a swiss-army knife.
Personally I use it for:
* to direct-stream cut (no re-encoding) slices of videos for memes
* to handle game replay captures (sometimes with re-encoding, sometimes not) and combine separate audio tracks into a file for youtube
* to handle game replay captures (always with re-encoding) but with separate audio tracks preserved
* to cut out audio tracks/subs that I don't want from a given movie/show when it becomes annoying
* to transcode a movie/show for archival at the minimum size/maximum quality
etc
In particular I'm not sure how to really tell ffmpeg to handle multiple audio/sub streams without programmatic interaction - you kinda need to know there are 2 audio streams and 4 sub streams, because you need to tell it what to do with each stream you want included in the output file, even if you do the exact same thing to every stream. If you have one show that's a little weird, fine, but without some kind of front-end it gets difficult to handle arbitrary content that is similar but not quite.
Then there are those that want a more generic version that allows all the bells & whistles to become available in the GUI. However, now you need to have a much better understanding of FFMPEG which the vast majority of FFMPEG users don't know (I myself learn new things everytime a new task is given).
At the end of the day, give it a go yourself to build a UI on top of FFMPEG to expose all of the things you just requested and see how painful it can be. I've done it multiple times for multiple jobs and not one of them was the same. It gets even more fun when you have sources with a combination of discrete and interleaved audio streams with high channel counts. Now, you have to probe the source and provide that info to the user to allow for output audio considerations. Based on those, you then need to potentially merge/join/split/map the audio. It's maddingly fun to those with those sense of perversions like me.
This isn't THE only issue with FFMPEG. There are many many problems with FFMPEG, but it's the best thing we have in this area. Warts and all, I would probably, no, definitely would not have been able to achieve many of the career accomplishments I have without it.
As you start to reveal the multitude of switches/options/etc available to FFMPEG via GUI, the GUI becomes gnarly and complicated very quickly. This then becomes just as daunting to the avg user as trying to build the command itself. There are many practical uses for a very simplified GUI to achieve a limited set of functionality. There's nothing wrong with it. In fact, I'd be willing to guess that if a UI came along that exposed every single thing that FFMPEG could do, fewer people would use it than a much more simplified version.
There are infamously a lot of tasks that "look easy upfront but have a lot of underlying complexity" and ffmpeg is not one of them. It's a task that looks hard upfront and then when you get down into the weeds it's an eye-bleeding eldritch monstrosity.
The consensus around MP4 and MKV containers and some relatively small number of codecs (H264, H265, VP1, AV1, etc) has substantially eased things but video codecs, audio codecs, and container files are just complicated af.
And believe it or not - this is still way, way better than it used to be! Flaskmpeg was never stable, Virtualdub was nice but development ended with avi container files, there were some forks that added various features, etc, but ffmpeg and OBS Studio both actually just work incredibly well compared to what came before.
except not everyone has decided to use that easing of the things. it seems like they go out of their way to make things difficult
My favorite is in how MP4s are not all the same. A video captured on a DSLR or other acquisition camera source might have +/- in the video/audio sync when imported into an NLE. Yes, the container utilizes PTS syncing to playback the video/audio streams in sync, but this +/- slop in sync is annoying AF to professionals capable of seeing the sync slop. Average consumers probably never notice. I thank all the gawds in all the heavens that I no longer have to edit MP4 source captured content and all of the manual slipping in the NLE.
But...sometimes you want one particular thing, that thing is relatively simple (compared to e.g. H.264 decoding), and you have needs ffmpeg doesn't meet. Then you probably can do better. A couple examples that have come up for me:
* specialized .mp4 muxing. My NVR project does this in a way where it can do HTTP range serving, so put the moov up front, produce the length quickly, and produce a given byte range without iterating through all the bytes before. ffmpeg can't do that, understandably. Hard to do in a general-purpose library; the container muxing input interface doesn't expect all the necessary information is available up front. But it's not that bad as a one-off. As you say, the consensus around a small number of containers really helps.
* RTSP. Consistent with your "modules that mostly work, usually, mostly", ffmpeg has various long-standing bugs in its RTSP handling (e.g. https://trac.ffmpeg.org/ticket/5018), has the risk associated with protocol handling in a non-memory-safe language, doesn't have much of a test framework for this, etc. This problem is relatively approachable (though hardly trivial), so I had success making my own library.
One thing I hate about CLI tool though, is that it's very hard to edit long commands in terminal.
For example, to replace the input filename from the previous command. I can do it in one second with my keyboard + mouse (use the mouse to select the old one, and type in the new one which would replace) in editor but you can't do so in terminal.
Another example is to change the order of some switches. Again very easy in text editor, not so much in terminal.
So, everytime I use FFMPEG for some complex one (especially when I need to go through trial and error, I have to write the thing in a text editor and them paste them into terminal.
Only a madman would do this. I don't know anyone that's in the middle of R&D that builds directly in the CLI that's sane. Everyone tries it, and just at the point of losing sanity, does the rational thing of building the command in a text editor for a simple copy&paste into the CLI later.
> It's a small key sequence: ctrl-x e . What's this do? It takes the existing command that you currently have on your command line and opens it in your shell's editor
Apart from the "wrap it in a shell script" advice, I would also note that in a bash-like terminal, home and end will jump to the start or the end of the current command buffer, and ctrl-del (or alt-del maybe?) or ctrl-arrow keys usually delete or move the cursor a whole word at a time, which makes it somewhat quicker to work rather than sitting there holding an arrow key or backspce.
Many here probably already know this, some may not!
Now this got me thinking: the main problem with terminal is that it doesn't seem to support "cut" (and consequently, "paste to replace selection"). So re-arranging a command string is always gonna be a pain regardless the complexity.
I can see why it was this way historically, but not exactly sure why no modern terminal emulator gives this feature, even as optional.
The antidote to ffmpeg's arcane command-line has always been finding someone else who already did it and seeing what they did differently. There are, of course, many ways to skin the cat in ffmpeg, but you often just need to see some finished, working commands and start from there.
You could of course build that on a wiki somewhere, ffmpeg does have various examples in their docs, but bropages is a place to curate that sort of stuff.
You'll need a client for it (tldr itself is just the database). I recommend tealdeer [2].
Example:
$ tldr tar
Archiving utility.
Often combined with a compression method, such as gzip or bzip2.
More information: <https://www.gnu.org/software/tar>.
[c]reate an archive and write it to a [f]ile:
tar cf target.tar file1 file2 file3
[c]reate a g[z]ipped archive and write it to a [f]ile:
tar czf target.tar.gz file1 file2 file3
Often combined with a compression method, such as gzip or bzip2.
[...]
[c]reate a g[z]ipped archive and write it to a [f]ile:
tar czf target.tar.gz file1 file2 file3
There's no real problem, but I never like it when I'm given an example that seems to be going out of its way to suggest something untrue.
But yeah you're right, it's all a question of how much capability you want to surface, because the more you surface the more quickly it gets complex and inscrutable to anyone not in the know.
At least some of your requirements are directly implemented in Handbrake, e.g., there are different transcoding presets [0] and burning in subs [1]. It uses ffmpeg under the hood as well.
[0] https://handbrake.fr/docs/en/1.5.0/technical/official-preset... [1] https://handbrake.fr/docs/en/1.5.0/advanced/subtitles.html
My problem with Handbrake was that it wasn't correctly copying creation datetime/makernotes/camera info into the new video and I had to manually later on issue bunch of ffmpeg commands for copying stream metadata, at which point I might as well do the whole conversion directly in ffmpeg.
But if you are dealing with repeatable, consistent file format (like files coming from the same camera), you are way better off going the command line route, and chances are the command will be pretty simple.
I could have dived into the log files and looked more closely at what it was doing but, eh, I'm at the level where I just want a batch script where I can see exactly what it's doing upfront. If I want batching/task management that is more advanced than a batchfile can do I'll set up SLURM workload manager.
What's the joke? I thought using VFW in 2022 was pretty hillarious of a suggestion. Maybe you got the punchline confused? ;P
AVISynth died to me long long ago as it was just no longer pratical. The work arounds to get 10bit video was never stable by the time I dropped it.
I did enjoy the scripting aspect of things like you suggest, but that's meh at this point. Now that my brain has been rewired to think FFMPEG like, the frustration goes away. It's a steep hill to climb without question, but eventually, it just goes "click".
2. The closest equivalent to VFW in linux is not ffmpeg but gstreamer. My experience with the latter is ... not good.
3. ffmpeg is great when it does what you need but VFW lets you draw from a vast ocean of third party plugins that do all sorts of niche things.
1)yes, if you're not needing 10bit, then AVISynth is fun and does some amazing things. Even it's features of doing technical restoration were useful, but once you step into professional, the 8bit only really holds things back and it was time to move on. FYI, a lot of the AVISynth filters are now available in FFMPEG!
2)gstreamer compared to ffmpeg to me personally is like comparing GIMP to Photoshop (dropping commercial vs FOSS). Yes, GIMP can do nice things, but Photoshop does what I need and I know how to make it do it. No questions asked. Muscle memory. I'm at that point with FFMPEG.
3)I'm long past trying to do creative things CLI that those 3rd party plugins allowed. I'm utilitarian with workflows and need to churn it out, so ffmpeg plugs in the holes as if it were designed for it ;-) If I'm going creative, I'm in a nice UI and doing things. However, I'm not afraid to use CLI to generate some content to messing with later in a more interactive environment.
This is what gets me every time. I'm relatively proficient with ffmpeg (relatively!) but for the life of me I can never remember how to switch audio sub streams or drop some of them, etc.
And yes, one first needs to find out what's where, but there are many UI tools that can do this (in addition to ffprobe).
Once I have identified a target CRF then I just apply that value to all similar files. I'm personally not concerned about exact size, I'm not squeezing them onto discs or anything, so having one file be 690mb and the next one be 695 mb is fine.
Contrary to the name, and contrary to popular belief: yes, placebo does still improve quality noticeably even over veryslow, when you are squeezing bitrate very hard. Actually typically you will get a noticeably smaller file than medium/slow (10% reduction is not atypical) and the quality will be better than veryslow. This applies less when you are giving adequate levels of bandwidth but when you are really squeezing the bitrate to the absolute limit (I've been known to use values in the CRF 31-37 range at some times to really go hard), going deeper into motion search yields more benefits. Like, you probably could still benefit from going even farther (placebo is just a preset tuning some motion-search parameters and you could tune them even higher).
At a minimum, veryslow does produce very significant quality gains over medium even if you don't want to go all the way to placebo though. Worth it for archival for sure.
And typically, the situation where I'm hand-tuning a file to hit an exact file limit... are ones where I'm seriously pushing the limits in terms of what I can fit into a given attachment-uploader file size limit or something. So when I am targeting a fixed file size I am almost always squeezing hard.
There's no perfect solution to avoid the trial-and-error process though, imo. Automated solutions can produce something reasonable but not optimal in terms of exactly hitting your filesize limit at the maximum possible quality. Tuning it down to the last MB, or deciding to resize a video to give a bit more motion bandwidth, isn't something that they can generally do.
Personally I don't worry about audio too much - 160kbps AAC or MP3 is fine and anything too far below that starts to get potentially noticeably poor. If something has DTS or AC3 yeah, I'm transcoding because nothing will play that on the internet, but if it's reasonably close I'll just copy source to avoid generational losses fromt anscoding. Yeah, sometimes a reduction in resolution can produce big gains with little noticeable quality gain... and sometimes (especially with anime/cartoons) a high-res file has so little high-frequency data that going big barely affects file size at all. Tune mode can also help a little bit at the edges.
I never mess with framerate or bit depth/color space unless I absolutely have to, though. Too many pitfalls there.
If ya can maybe help em out and send a pull request? I think it's a neat little app and will take a look at it myself.
Not associated, just find it easier to link to something useful.
I made a script that converts all the videos in a directory above a certain bitrate to H.265 to save space and found that just using the flags `-map 0` to grab every stream, and `-c:s copy` was enough to avoid having to probe the source for every stream since subtitles were the most problematic if any streams were bitmaps. It then checks if there were any errors before replacing the original video. I still get the occasional outlier that I either fix by hand or just leave as is.
Also the script was written with python and argparse, so it would be relatively easy for me to create a GUI with Gooey[1] if there was interest.
A trivial example is to try to convert the subtitles rather than defaulting to no subtitles (Handbrake). Another trivial improvement would be warning that the subtitles can’t be brought over to the container format before the 20 hour transcode rather than letting users find out after that their subtitles weren’t converted.
This isn’t rocket science.
Another tool that this is nice for is imagemagick. Is there something newer than imagemagick for image generation these days? I'm having trouble with some non English scripts and draw text
In my experience, you're just better off searching for what you need, in my experience you'll always find a bunch of results with the exact ffmpeg incantation you need for your specific rare usecase.
My current workflow is to open the image in a viewer that automatically reloads on change, like feh or sxiv, and then in my terminal use <M-a> to run the command without clearing the command line.
[0]: https://magickstudio.imagemagick.org/scripts/MagickStudio.cg...
One thing that might help: ffmpeg is, in fact, the Unix philosophy of "do one thing and do it well", it just so happens that all the commands are prefixed by the ffmpeg command, but you can break things down really easily after that. Each codec is independent, mappings are independent, filters, etc.
Recently I was working on processing large images (100-200MB per jpg) which needed a crop, some added text and "progress bar" for a timelapse. I started with imagemagick, it was super slow, so tried graphicsmagick, also slow, just somehow I thought, why not try ffmpeg. Man, it's night and day difference. It processed my images super quick (from ~30 seconds down to ~1 second), and had surprisingly good documentation (compared to IM/GM), and allot of support on stackoverflow etc.
I use this flag because I don't need accurate seek, but sometimes it works, sometimes it doesn't. When it doesn't work, there will be a few seconds of frozen image at the beginning of the cut. Don't know how to solve it.
Proper terms are i-frame and p-frame. https://en.wikipedia.org/wiki/Video_compression_picture_type...
Though according to the ffmpeg docs on -ss, it should be doing noaccurate_seeks if you're doing stream copy.
Also I'm unsure about this, but I think the audio stream can be cut at a different location than the video stream, leading to the frozen frame, and different video players seem to handle it differently. I've got a clip I've made, with transcoded video stream but copied audio stream, that is 1:06 on Firefox with the extra audio at the beginning, but 1:03 on VLC, MPC, Chrome, and desynced in Edge. Extracting the audio with ffmpeg again gives a time of 1:04.02 compared to a video stream of 1:03.
- x264 settings as well as copy video codec
- aac re-encoding settings
- mixing 5.1 to 2.1 audio
- adding multiple subtitles
- rendering hard subtitles
I was just using ffmpeg to convert videos to watch on the TV and I needed all of those.
As soon as the carefully curated string of option configurations went into a script, it also evaporated from my brain. Oh well.. til next time.
Multiple -filter, -af or -vf options specified for stream 0, only the last option '-filter:v transpose=1' will be used.
https://github.com/FHPythonUtils/Cli2Gui/
It's 2022, shouldn't CLI to GUI conversion be an automatic and core feature of every OS by now?
---
[1] https://avidemux.org. Avidemux has a "built-in libavcodec library from FFmpeg (i.e. Avidemux contains its own copy of libavcodec, external FFmpeg is not used)"
I knew how to in Photoshop etc, before they moved to the cloud so they can be paid in perperuity even though a lot of the advanced/useful features like layers or cropping etc are in more basic tools like Preview or GIMP now.
(GIMP needs to get a macOS friendly rewrite similar to how folks did KeePassXC imho, but I'm not a strong enough coder and no large open source friendly companies seems to want to pay me a salary to direct projects like that, so all I can do is halfway troll here on HN, but it comes from a place of love.)
"ffmpeg.exe" -i "in.mp4" -c copy -movflags faststart "out.mp4"
That's already available: https://handbrake.fr/
Something like this for PANDOC would be great as well
the difficult part is to combine vaapi, codecs, re-encode audio without re-encode video (and vice versa), codec lists, etc