FFmpeg and a thousand fixes
googleonlinesecurity.blogspot.com
googleonlinesecurity.blogspot.com
The fuzz testing they mention is based around constructing malformed (or at least "exotic") input files and then monitoring for failures... ie simulating exactly the kind of attack someone might use against YouTube's transcoding infrastructure.
This cron has been keeping ffmpeg in check for over 4 years (not 6, whoopsie) in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system.
1 <?php
2 /*
3 ** Cron responsible for detecting failed/hung ffmpeg
4 ** instances and killing them.
5 */
6 require_once '/lib/class/dbmysqli.php';
7
8 $output = shell_exec('ps -aeo pid,etime,args | grep ffmpeg | grep -v grep');
9
10
11
12
13 preg_match_all("/^[ ]{0,}([0-9]*)[ ]{0,}(.*?) .*?([0-9]*?)\-[0-9]*?\.flv /m", $output, $preg_out, PREG_SET_ORDER);
14
15 if (!empty($preg_out)) {
16
17 $db = new DBmysqli('dbmaster');
18
19 foreach ($preg_out as $process) {
20 $pid = intval($process[1]);
21 $etime = $process[2];
22 $queue_id = intval($process[3]);
23
24 $etime = intval(str_replace(':','',$etime));
25
26 // elapsed time >= 60:00 (1 hour)
27 if ($etime >= 6000) {
28 $fail_sql = "DELETE FROM media_upload_queue WHERE queue_id = $queue_id";
29 shell_exec("kill -9 ".$pid); // kill hung ffmpeg process
30 $db->query($fail_sql); // remove file from DB
31 // debate if life is worth living...
32 }
33 }
34 }
35 ?>
"This cron has been keeping ffmpeg in check for over 6+ years in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system."
why do we programmers always feel we need to apologize for something that we did quickly, but has been running without incident for a number of years.
take a bow my friend. that was an awesome patch you came up with all those years ago.
Because if we don't, our programming brethren will assume that we thought the fix was correct and optimal. And we'll get bashed for it.
Haven't you ever heard of the quip (summarized): "The quickest way to get help from a technical crowd is to claim what you're doing is correct. People will bend over backwards to prove you wrong."
The script is only treating the symptom, not the problem.
It would be better to detect the files before they waste an hour of time, so you could tell the user instead of having them silently disappear. Maybe there is something you could do to fix the files. The programmer part of me says there is a real fix that needs writing.
It's obviously a great script if it's worked that long. Who knows how long it would take to track down and fix the bug(s) causing the issue. If they haven't needed to fix the bug in all these years just writing that script was obviously a good decision.
Exactly this. There are an unbounded number of bugs which will cause this single symptom. I think detecting the symptom is exactly the right solution.
I agree that mitigating the problem is a good first step though.
r8645 | * | 2010-02-09 18:11:35 -0500 (Tue, 09 Feb 2010) | 2 lines
- NULL pointer dereferences,
- Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access,
- Out-of-bounds reads and writes to stack, heap and static-based arrays,
- Invalid free() calls,
- Double free() calls over the same pointer,
- Division errors,
- Assertion failures,
- Use of uninitialized memory.
But hey, any good programmer always writes perfect C code.
Meanwhile, you are just being snarky on a message board while using a whole stack of software that was built by the programmers whose work you are criticizing.
Rust looks great for apps and middleware. I don't see ffmpeg or x264 (both of which are implemented with significant amounts of architecture-specific machine code -- an environment where neither C nor Rust is going to be able to help avoid the kind of bugs we're talking about) jumping ship any time this decade.
I'm not claiming it's production quality, but I did write an MP2 decoder in an early version of Rust: https://github.com/pcwalton/fempeg
> Rust looks great for apps and middleware. I don't see ffmpeg or x264 (both of which are implemented with significant amounts of architecture-specific machine code -- an environment where neither C nor Rust is going to be able to help avoid the kind of bugs we're talking about) jumping ship any time this decade.
I grepped through the commit log for bugs marked "j00ru", as suggested in the post, and the first 20 I found were all mostly-architecture-independent, straightforward C code (often bitstream parsing code). This is the kind of problem that Rust is generally good at securing and making performant. None of the 20 patches I looked at touched any assembly code at all.
But the macro point is just empirical: the proof that Rust is an appropriate tool for codec development needs to come in the form of a production-quality codec implementation of some non-trivial standard (VP9, say), not a HN post.
I don't understand this. If you write in a safe language, the compiler rules out the memory safety bugs, regardless of the problem.
> But the macro point is just empirical: the proof that Rust is an appropriate tool for codec development needs to come in the form of a production-quality codec implementation of some non-trivial standard (VP9, say), not a HN post.
Of course, no argument there. I am reasonably confident that we can get there with tuning though, based on our benchmarks so far. :)
For performance sensitive code, you sometimes need to deliberately step out of the strongholds of the compiler.
...or just add more primitives to the language.
If you mean "constant" primitives, i.e. things which don't require to expand the language grammar or type system, then it's a question of future prevalence of usage of said primitives. If they become almost ubiquitous to any project, then they obviously need to be included in the standard libs. If you're talking about a new language feature, that depends on how much that new feature may help design and write future libraries, and also what are the aims of said language. I understand that rust aims to become a safer system language, so any feature helping the design of low level, performance critical code should be welcome.
OP is criticizing without a plausible alternative; Rust is at least more plausible of an alternative. But I also suspect that C+ASM will continue to rule the day here, especially now that these fuzzing techniques are become more widespread and industrial-strength.
"You will never find a programming language that frees you from the burden of clarifying your ideas."
> If you're a C programmer and somebody says like: I've that optimization that makes your program 3% faster you go: wow! Then you come and say: well, now I make it half as fast you go: w00t?! But then you remember that people actually use ruby to serve web pages.
I'd say this would be actually worth it for a far more secure FFmpeg.
It's been around since 2006, but I haven't really heard of anyone using it. I wonder why.
It never compiled on 64bit architectures, and to get it to compile with 4.0 or 4.1 (newer versions of gcc won't work) you need to fetch it from SVN (last commit is dated May 2009)... and obviously: no distribution still ship such an old version of the build tools... you'd need something like debian lenny, which does not even receive security updates anymore
This is like saying "my bulletproof vest will stop nearly every round fired at it". Too bad it only takes ONE...
But Modula-2 had Lilith to offer, while C had UNIX.
Compiler support across various platforms: are there compilers available and do they generate good code?
Mindshare: how big is the intersection of people who know the language and have the domain knowledge to contribute to the project?
Or are you perhaps just wishing that history had taken a different turn?
There'll be a HN topic A FFmpeg reimplementation in JavaScript soon enough ...
There is also Intel Lab's HRC (Haskell Research Compiler) that does have SIMD capabilities [2]. Unfortunately, there isn't a public release of HRC yet.
---
[1]: https://ghc.haskell.org/trac/ghc/wiki/SIMD
[2]: http://dorchard.wordpress.com/2013/10/14/automatic-simd-vect...
ISO/ANSI C also don't support SIMD, you have to go down to Assembly when writing portable code across C compilers.
also, I think you may be able to do reasonably well with icc and not much assembly
You don't need to work at the assembly level for this; you just need a language that can express vector operations in a way that doesn't require a compiler to solve the halting problem, and you need a compiler that knows how to compile such operations to SIMD.
Yes, ANSI C doesn't have such a construct, but GNU C (which I'd bet 95% of open-source code uses anyway) does.
(Plus x86's intrinsics are just plain ugly. It's easier to read asm.)
As for the architecture-independent SIMD in GNU C, it's something, but not quite flexible enough. Even C is really not a good enough language to consider implementing this in, because it has an overly-conservative memory model when it comes to char *, which aliases all other memory. For a library dedicated to parsing things which come in the form of bytestreams and 8-bit pixels, this means that most compiler optimizations are defeated right in the inner loops where you'd need them.
But restrict is only good in specific situations - if you have three pointers, and #2 can alias #1 but not #3, there's no way to express that.
It also tends to be used on function arguments, not at some higher up point of declaration. So when compilers inline the function and can see more global info about it, the restrict gets lost and optimization gets worse.
I think some kind of stronger 'typedef' would be better.
[1] http://msdn.microsoft.com/en-us/library/5ft82fed(v=vs.80).as...
map (uncurry (+)) (zip [1,2,3] [4,5,6])
The compiler needs to be able to see that the data conforms to all those restrictions (possibly more), and that's not a trivial thing to do.
Here's an example in Haskell: consider the case that the list of numbers was generated from a list of some other structure; something like "map x coordinates" to get all the X values from a list of X-Y-Z coordinates. For vectorization to work, those values need to be packed together in memory; however the structure isn't organized as such (at best, you'd have XYZXYZXYZ). That means a nontrivial memory copy, which will cost much more than the vectorization will gain you. So the structure needs to be reorganized, but then that might cause other pessimizations.
For the remaining 95% of the code, since 1978 one could use Modula-2, Ada, Pascal dialects, SafeC and many other system programming languages.
Back in the early 80's, C compilers were as crap as any other system programming languages when generating code for home computers.
- One hell of an awesome encoder/decoder library that can hit more formats than I knew existed.
:)
> There were lots of dynamic allocations
This surprises me; a lot of codecs are specified in such as way as to not require dynamic allocation at all.
I hypothesise that if it was written in another language, we would just see bugs appear in a different form.
Exactly! Programming is hard to get right, no matter what the language. I hate these people who come along saying "hurr, durr, C sucks, project X should have been written in language Y!" All the while ignoring the facts that a) they can go and rewrite X in Y any time they like, and b) if it was written in Y, people would be bitching about issues with Y (eg, speed). Shortsightedness and lack of will to put your code where your mouth is are two of the most despicable traits a programmer can have.
Language has a lot to do with what sort of issues programmers will have to tackle.
Using python as an example here makes little sense, since the constraints of the problem domain make it a particularly poor choice. Try choosing a language that doesn't suffer as much in performance (future Rust, maybe?).
I'm just making a point that language is relevant when discussing the type and frequency of errors that are surfacing in a project.
But the OSs created with them weren't as fancy as UNIX was at the time, or used hardware more costly than the PDP-11.
If you managed to execute code, what privileges do you have in Chrome? I hope that Chrome was using OS sandboxing for video playing. After all, if they found 1000 bugs, there are probably a few more zero day's available.
Does playing Flash video in Chrome/Firefox end up using ffmpeg? I'm not that knowledgeable about how video works, but there are probably at least two execution contexts: playing .mp4 natively in Chrome, or playing it via a Flash container.
Quite a few. We're often affected with VLC, and code execution is easy to get to. But with VLC, you're "only" in userland.
(I know you know that, I'm just spelling it out for people).
Especially since you can see often fake videos circulate around, that tell you to download a special software or codec to play them (with malware, of course). WMP also had a scripting/runtime system that could be abused greatly.
Be careful what you download :)
A sandboxing system provided by either ffmpeg or VLC would be a very good idea, though it would be some work... encoded data in, decoded frames out via shared memory. Negligible performance impact.
Given that there are apparently thousands of bugs in the video parsing code, it seems like a no-brainer.
Section 5.2 in this DJB paper talks about (portable) isolation of plain transformations. Video playing is already close to "pure" or could be made pure pretty easily.
Not sure if serious or sarcasm, to be honest, since this seems very far from what we see.
A media player is not simple to sandbox, (as the MacOS X sandbox showed us for example), because:
- you need to open files by yourself, without user interaction, to support playlists,
- you need to open connections by yourself to support video protocol like RTSP, RTMP, RTP,
- you need raw device access to support Webcams, Capture devices, DVDs, DVB tuners,
- you need to access GPU buffers for direct rendering, and/or shaders to do fast filtering or just plain chroma-conversions,
- you need to be able to access the audio output, at low-level, for libsync which is not always doable with the simplified APIs,
- and I don't understand what you mean by "very limited gui input"; how is that less than other programs?
Sure, it can be done, with performance costs but it's clearly not a "no-brainer".
http://www.chromium.org/developers/design-documents/multi-pr...
The entire app isn't sandboxed -- just the code that does video parsing, i.e. with the thousands of bugs and hundreds of remote code execution exploits (!).
See my other comment on this topic. ffmpeg is already very modular, and used in many video players (user interfaces), so this separation is more than natural -- it already exists in the codebase.
BTW, some people seem to be unfamiliar with the multiprocess/Unix design approach (usually people with a Windows background, which I came from as well). I recommend http://www.catb.org/esr/writings/taoup/ for a great intro to this design philosophy.
The exploits are usually on the protocol (access) level, the demuxer (format) level but also the decoder level.
While in theory the first 2 are what you call parsing, many security issues appear also at the decoder level.
And if you want to split the video decoder from the rendering, you need to introduce an additional memcpy (or two) of full decoded frames, which has an important impact.
I agree it would be nice to try, with a correct, separated processes architecture, but it means changing also a bit the usual architecture, where there is a direct-rendering between the decoder and the output.
You can open your X11 socket (or whatever your GUI uses) in the I/O process and hand it off to the renderer before dropping privileges, and hand it off without any loss of performance. Another thought -- although I don't generally touch 3d rendering, or know enough to know if it's possible off the top of my head -- is something like cross-process pbuffers: you render into one, and composite the result to the screen from another process. Since your OS already does this sort of thing (X11 compositors, for example, do this sort of cross-process compositing), this can't be too expensive.
You might need to use the XACE extension to restrict the renderer to just the rendering output window in case it gets exploited, so that you don't have access to the rest of the UI.
The places where vulnerabilities can do damage are when communicating with the rest of the OS: File I/O allows you to get user data, permanently install malware, and delete things. The window system allows you to snoop keystroke/mouse data, synthesize input, and watch what the user does. Drop privs on those, and your app is fairly effectively sandboxed.
Off the top of my head:
>video output
Meaning: RW access to /dev/dri/cardX on desktop. In an embedded system, replace that with /dev/fbX. In most cases, we're rendering to GL buffers. Do you trust the GL drivers to be completely bug free? (Hint: from some embedded drivers I've dealt with I find it remarkable they work at all.)
>doesn't need to [...] open network connections
How about RTMP streaming? Or DLNA discovery and playback?
Some of the cases may be protected with sandboxing, but I have serious doubts about getting anything near complete coverage.
I think you could also provide less than full access to the graphics driver. You could have a file descriptor or shared memory protocol, with the sandboxed process outputting video frames, and the parent process actually communicating with the driver.
In addition to being more secure, it's also better software architecture. VLC/ffmpeg are already way more modular than say the Microsoft equivalents.
Memcpying complete video frame in HD at 30 or 60 fps is not negligible performance impact. But I agree it would be a good idea.
It might get hairier if the platform graphics api wants to supply you a buffer to render into, and you can't pass it a shared memory buffer yourself (that is accessible by that sandboxed process).
The linked blog article [1] suggests 10% - 20%:
Our personal feeling is that between 10% and 20% of the
problems could be considered easily exploitable security
issues; however, the estimation has not been formally
confirmed in any way.
Well, it's not like I've got a whole lot of options for video processing. Note that they didn't recommend alternatives that they say are fuzz clean.
I use FFmpeg quite extensively and will continue to do so.
I wasn't even aware of the split at the time, but this shenanigan definitely gave me a strong negative initial impression of libav.
I really miss ffprobe, I can never get avprobe to work on the first or second attempt.
What issues were you having, and with what file types?
* I presume all this infra is used for more than just ffmpeg so why this software specifically? Im conscious its included in everthing from web browsers to games. If this was an attack vector, its a BIG vector.
* 1121 (or whatever it was) is a curious number to make a post about. Why not at the 1k mark or 1250?
* Explicitly calling out use of an unpriv user to a readership likely already well versed in basic security practices but notorious for casually ignoring on personal machines
Snarky, whiny, bitching comments do not a discourse make.
For instance, one fix mentioned was a fix the demuxer for .4xm files. 4xm is a proprietary container format for a now defunct company. Chrome would never have to worry about this bug, because it doesn't need to support playback of .4xm files.
More specifically Google is recommending you don't use it to "process untrusted media files"
And our update mechanism suck. We're working on improving the process because of that...
Hm, privacy was not on top, I guess - or we're toasted.
1. flvrunner uses ffmpeg
2. biggest result for flvrunner searches is how to remove flvrunner (including browser toolbars)
3. doubleclick runs webads for flvrunner.com (google own doubleclick)
4. such ads even run on onlinebehavior.com, which is owned by Google's analytics guy.
Someone more conspiratorially minded than me might make other deductions.
BAHAHA What a joke.