This pretty much answers my question but what do the wizards here in YC think? http://www.sitepoint.com/blogs/2006/10/15/binaries-belong-in-the-database-too
In short: It depends on your application, and probably only you can know for sure (and then probably only with experimentation).