The fuzz testing they mention is based around constructing malformed (or at least "exotic") input files and then monitoring for failures... ie simulating exactly the kind of attack someone might use against YouTube's transcoding infrastructure.
This cron has been keeping ffmpeg in check for over 4 years (not 6, whoopsie) in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system.
1 <?php
2 /*
3 ** Cron responsible for detecting failed/hung ffmpeg
4 ** instances and killing them.
5 */
6 require_once '/lib/class/dbmysqli.php';
7
8 $output = shell_exec('ps -aeo pid,etime,args | grep ffmpeg | grep -v grep');
9
10
11
12
13 preg_match_all("/^[ ]{0,}([0-9]*)[ ]{0,}(.*?) .*?([0-9]*?)\-[0-9]*?\.flv /m", $output, $preg_out, PREG_SET_ORDER);
14
15 if (!empty($preg_out)) {
16
17 $db = new DBmysqli('dbmaster');
18
19 foreach ($preg_out as $process) {
20 $pid = intval($process[1]);
21 $etime = $process[2];
22 $queue_id = intval($process[3]);
23
24 $etime = intval(str_replace(':','',$etime));
25
26 // elapsed time >= 60:00 (1 hour)
27 if ($etime >= 6000) {
28 $fail_sql = "DELETE FROM media_upload_queue WHERE queue_id = $queue_id";
29 shell_exec("kill -9 ".$pid); // kill hung ffmpeg process
30 $db->query($fail_sql); // remove file from DB
31 // debate if life is worth living...
32 }
33 }
34 }
35 ?>
"This cron has been keeping ffmpeg in check for over 6+ years in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system."
why do we programmers always feel we need to apologize for something that we did quickly, but has been running without incident for a number of years.
take a bow my friend. that was an awesome patch you came up with all those years ago.
The script is only treating the symptom, not the problem.
It would be better to detect the files before they waste an hour of time, so you could tell the user instead of having them silently disappear. Maybe there is something you could do to fix the files. The programmer part of me says there is a real fix that needs writing.
It's obviously a great script if it's worked that long. Who knows how long it would take to track down and fix the bug(s) causing the issue. If they haven't needed to fix the bug in all these years just writing that script was obviously a good decision.
Exactly this. There are an unbounded number of bugs which will cause this single symptom. I think detecting the symptom is exactly the right solution.
I agree that mitigating the problem is a good first step though.
Because if we don't, our programming brethren will assume that we thought the fix was correct and optimal. And we'll get bashed for it.
Haven't you ever heard of the quip (summarized): "The quickest way to get help from a technical crowd is to claim what you're doing is correct. People will bend over backwards to prove you wrong."
r8645 | * | 2010-02-09 18:11:35 -0500 (Tue, 09 Feb 2010) | 2 lines