Hacker News Reader

Comment by sandGorgon | Hacker News Reader

Interesting technical question (disclaimer: I have no financial interest in these things). How is stuff like "Then stuff to automatically convert audio & image files to the right formats." usually done ?

Does the webapp push stuff in a queue and call a commandline tool like mencoder or something is there an industry standard tool ? How do you deal with concurrency (some kind of Actor model ) ? And most importantly, do you have to tune the linux kernel to achieve performance on this (just saw the LinkedIn NUMA post as well, so thinking about that)?

I am sure Youtube and all do it using the enviable Google infrastructure, but how does someone else do it

View on HN

All the stores have different requirements. One store requires 2500x2500 artwork, another requires FLAC audio, etc. So when users upload audio or video to DistroKid, everything gets converted to appease all the different stores.

Right now there's a cron that runs every few seconds, finds the next unprocessed file, and processes it via command line. If/when volume gets really high, I'll probably have to do this somewhat differently to make sure it scales.

DistroKid uses MediaInfo to figure out what the user uploaded:

http://mediaarea.net/en/MediaInfo

Then uses SOX for audio conversions:

http://sox.sourceforge.net/

And native Railo (the backend programming language I use) functions for image processing/resizing.

http://getrailo.com

thanks ! very interesting - I have never heard of any of the things that you mentioned .

could you talk about why you chose Railo, which looks to be a fairly esoteric stack. Is it something you chose specifically for its media capabilities ?

I love Railo. And I love the CFML language, which is what it uses. Railo is free (as opposed to Adobe ColdFusion - the other CFML platform), open source, has a great community, and does everything I need. I'm a speed demon in it. And the pages load super fast.

>Right now there's a cron that runs every few seconds, finds the next unprocessed file, and processes it via command line. If/when volume gets really high, I'll probably have to do this somewhat differently to make sure it scales.

Hah! Really cool. I wrote a very similar script/program to do that for huge video files using tons of different case/switch methods that carve up mplayer2.

Really awesome project, BTW.

How did you decide on using MediaInfo and SOX, rather than say eyeD3 and ffmpeg?

I forget why I didn't use ffmpeg. There was a reason. I evaluated both and decided SOX was better for me. It's working well so far.

+1 for the simple & pragmatic solution w/ cron and not over-engineering this at the current stage!

Railo claims to be based on JavaEE, so you have lots of options for scaling this if needed (TimedTasks can be run on more than one server, JMS provides queuing when jobs are waiting, etc).

I can't speak for them, but yes it almost certainly stores it in a database and queues it for processing by a command line tool like ffmpeg. You could just have a script that checks to see if there are songs in the queue, then runs the command line tool against the next song in the queue when one of a specified maximum number of processing slots is available. You could fire that monitoring script with a cron.

This isn't the type of service that would do massive Youtube-like volume, so I cannot imagine that they are doing anything special to handle high volumes of uploads. I would guess that a single lower-end AWS server would do an adequate job for the volume they'll be handling.

My does that kind of "stuff", we automatically convert audio and embed inaudible watermarks to prevent piracy,leaking, etc.

AWS makes a lot of the concurrency issues easy (and scaling). Basically you can use their SQS ("Simple Queue Service"), add tasks to it, and when the individual drones check out a song from the queue, it's no longer available for a set amount of time.

If the drone finishes the process completely, it removes it from the queue permanently, but if the drone fails, dies, whatever, after that time-out it gets bumped back into Queue for the next worker drone.

We use FFMPeg for conversion.

--- note: tracktrack.it, if you're curious about watermarking.

Store all the assets/data however you like, then append a job description (referencing the assets/data) onto a queue. Separate servers have a worker process running that tries to fetch a job from the head of the queue and start processing, taking as long as it takes. When it's done, it fetches another. A separate overlord process monitors the queue health. If the queue grows too large, it fires up more servers. When the queue is small, it shuts down idle servers.

I've solved this kind of problem before, albeit for video.

https://github.com/jrgifford/delayed_paperclip/

https://github.com/owahab/paperclip-ffmpeg/

Plus a delayed job worker process managed by upstart, tada, got yourself a video-encoding-and-streaming system.

> And most importantly, do you have to tune the linux kernel to achieve performance on this (just saw the LinkedIn NUMA post as well, so thinking about that)

Ha, no. Definitely not until you have really significant scale - it's remarkably fast on a reasonable dedicated server and scales well across cores, run N/2 to N+1 delayed_job processes for N cores depending on how well ffmpeg et al make use of your cores. Well faster than realtime.

thanks - one tangential question: why upstart and not runit or supervisord or all the other tools ?

Running ubuntu, don't rock the boat, just use the system default unless you have strong opinions.

From my brief last.fm experience, industry standards are a huge mess. I think we actually had our own program for doing at least some of the conversions, but it was just a couple of hundred lines calling standard libraries; commandline ffmpeg would not be unreasonable.

And yeah, it's an actor-like model, except you don't really have a problem with concurrency. You just need some kind of task queue that you add encode jobs to and a bunch of worker machines that take tasks off this queue, run them, and respond. Almost every big system seems to start looking like this.

It can convert all those formats, but it can't convert a .jpeg to a .jpg.

Reply on news.ycombinator.com