Shrine – A new solution for handling file uploads in Ruby | Hacker News Reader

Shrine – A new solution for handling file uploads in Ruby

twin.github.io

52 pointslucisferre20 comments

The creator of carrierwave started the new project refile [1] as a better alternative. Is there a comparison between the two approaches, i.e, refile vs shrine?

[1]: https://github.com/refile/refile

I was actually contributing actively to Refile before, so in a lot of ways Shrine is similar to Refile (the idea of backends is basically the same). However, the main difference is that Refile does processing on-the-fly, while Shrine (and CarrierWave) do it upfront, so it goes in that direction. Basically in Shrine I added all of the features I was missing and wanted to change, which I wasn't able to add to Refile in a compatible way. One major addition is the support for background jobs, which I just couldn't see how it could be added to Refile. Also, I wanted Shrine to be far less opinionated than Refile, that you can have much more options how to do uploads.

Can you explain what you mean by "upfront" vs "on-the-fly"? I'm somehow not getting it.

This looks very nice. It'll need lots of docs with examples to understand how to put together the nicely flexible building blocks you provide.

"Upfront" processing means that you're doing the processing during upload, and on the "on-the-fly" processing means that you first upload the file, and then it's processed dynamically through the URL. For example, in Refile you make an URL to the uploaded file like this:

/attachments/sdkd/resize/300/300

That will make Refile trigger the specified processing at the moment the URL is requested. You should then put a CDN or a reverse proxy in front to kind of cache these processed files, so that they're not processed each time.

About the documentation, I wrote documentation for each of the plugins, and I linked them all on http://shrinerb.com. The plugin names should hopefully indicate what feature they accomplish, and since they're all individual, it's up to the user to choose what features they want (i.e. combination of plugins).

So "up front" still includes doing the processing in a background job async, as long as it's triggered at point of upload, not at point of request?

"upfront" but async background does seem to be the best way to approach it to me, at least if you had to pick one -- that's what Shrine does?

Yes, you got it right, this is exactly what it means. Yeah, there are some rare use cases for on-the-fly processing (think a WYSIWYG editor where the user can choose the size of the image), but for me upfront definitely works best.

Cool. I'd call those two choices "upon upload" (instead of 'upfront'), and "upon first view" (instead of "on the fly"), but I'm sure my terminology would confuse someone else too.

I'd also be interested in a brief comparison. This is the first I've heard of both Shrine and Refile. I'm really excited that backgrounding and direct uploads are being simplified in both projects.

I haven't done much Web work in Ruby, but in my head the first thing that comes to mind for file uploads is Paperclip[0]. I'd be curious to hear thoughts on the various alternatives (e.g. CarrierWave, Refile, Shrine).

[0] https://github.com/thoughtbot/paperclip

I've only used Paperclip a few times for some really basic uploading without any processing. By design, it really aims for simplicity which can be a bit of a pain if you try to do more so I'd always opt for Carrierwave in those circumstances. It's a lot more flexible, and it's nice to avoid cluttering up your models. Refile has always kind of rubbed me the wrong way; processing images on-the-fly reminds me of a really nasty experience I had with a PHP app a few years back that I was asked to "fix." Given the choice, I'd prefer to avoid it altogether though I realize there are a few use cases where it makes more sense. Chalk that one up to a personal bias, I suppose.

I feel the same as you. Refile was first very exciting for me, because if you wanted to change a version, you don't have to reprocess them, you only have to change the URL in the HTML. But that means that if you have a page with lot of these images, it would be super slow and can really easily DoS your server. And CDNs also cache the images for like 4 hours, which is terrible. I realized it's much better to just have the thumbnails done immediately.

Yes, I was inspired by CarrierWave's uploaders, the goal of Shrine is to have all of the uploading logic in the uploader. And it even allows you too hook up to ".included" of the model that receives the attachment methods, from inside the uploader (http://shrinerb.com/rdoc/classes/Shrine/Plugins/Included.htm...).

I've used Paperclip in production. It's good and does what it's supposed to. The main issue is if you have a lot of different image sizes. It will basically process all of the images inline + store them to s3, etc. My ideal flow for image processing is to create the thumbnail (which the user sees right away) and to throw the rest of the image processing work on a background queue.

This minimizes the time spent processing a request and keeps the front-end snappy.

I had exactly this scenario in mind when I was building Shrine. It has a recache plugin which allows you to do additional processing after validation but before storing, and then you can do the rest of the processing in a background job as part of storing. https://gist.github.com/janko-m/cfc50049c360d5c8be34

You could do that with https://github.com/jrgifford/delayed_paperclip/ I guess.

The part that I mind about both delayed_paperclip and carrierwave_backgrounder is that it doesn't really allow you to write your own job classes. This can be quite limiting for example if you want your jobs to send information to some monitoring services.

Another part that I mind is the user experience. Ok, both of these gems allow you to know when the backgrond job is in process, and then you can display some placeholder image until the processing is done. But that's not really nice user experience. In Shrine the user immediately sees the image they uploaded, because it has that image already cached (usually on the filesystem), and then from the user's point of view the uploading is finished, before the background job even started. CarrierWave also has the cached image, but by design it's not possible to use it, while Paperclip doesn't cache images.

This looks like I very cool project, thank you for all the work you put in on it, I've definitely run into some of the problems you're trying to solve, and have come up with various custom-hacked solutions to make it work. I can't wait to try this out on a project.

Here's a take down of all approaches. https://www.slideshare.net/choonkeat/file-upload-2015

TLDR: forget shrine too, use an image server

Very interesting presentation, really on a high level, I enjoyed reading it.

I noticed that many problems that you mention other uploading gems have are actually solved by Shrine:

- "Transformation juggling" -- Versions are processed at instance-level, so you don't need to remember a class-level DSL.

- "File path config" -- If you change the location where your files are stored, the existing files on old locations will still work normally

- "Form validation dance" -- Very simple with Shrine, you just add an "<attachment>" hidden field.

- "Schema pollution" -- Shrine stores uploaded files exactly in the way that you described, in a single column as JSON

The idea of an image server indeed looks very interesting, but I think you're forgetting that this solves only one part of file uploads (although I agree it's a big part, and imgix looks really sexy). If you're using an image server, you still need to do:

- Caching the file (this still needs a direct upload endpoint)

- File validations (most notably preventing big files)

- Storing the file to image server (this still needs a background job)

- Model attachment logic (all of the assignment/callback logic)

One problem that I always had with on-the-fly processing is, if you have a page with a lot of photos, and you want to change the URL so that they're processed differently, how will that page look when a first person visits it? I think it will be horrible, the user will have to wait for a really long time to actually see the images, which is not nice user experience.

Also, the "image server" strategy can be used only for images, so if you want to upload documents, audio or video, you still need do regular processing on the application side. What's nice about uploading gems is that they provide a general solution.

sorry didn't see this reply. i'll copy-paste my reply http://blog.choonkeat.com/weblog/2015/10/file-uploads-2015.h... for posterity ;-)

> If you're using an image server, you still need to do...

yes and we should move towards solving issues in the image server, not yet another client gem

> File validations (most notably preventing big files)

big file upload can equally be prevented on image server as on a rails server, settings should be in the webserver fronting it.

validations are "easy" since you're only validating json attributes, but i guess as u mentioned, shrine has this benefit too.

> Storing the file to image server (this still needs a background job)

even better than background job: we always direct upload from browser/client (regardless of whether we are using s3 or not).

> Model attachment logic (all of the assignment/callback logic)

a client lib can do this for your rails app. and it isn't much https://github.com/choonkeat/a...

> One problem that I always had with on-the-fly processing is, if you have a page with a lot of photos, and you want to change the URL so that they're processed differently, how will that page look when a first person visits it? I think it will be horrible, the user will have to wait for a really long time to actually see the images, which is not nice user experience.

we can still "pregenerate" after the author uploads his images, the browser just have to make the requests (not necessarily visible) and the image servers can cache those generated images for a long long time. then when the "first other person" view the page, those images are as fast as anything.

storing the generated image or not is still an image server's decision. my own implementation chose not to store because i find those generated files more trouble to carry around.

again, these are things for different image servers to iron out and one-up each other with.

> Also, the "image server" strategy can be used only for images, so if you want to upload documents, audio or video, you'll need to use something else. What's nice about uploading gems is that they provide a general solution.

actually, there's nothing in an image server that makes it inherently image only. i'd say file servers provide as general a solution as those uploading gems

for example, attache server handles pdf uploads well and even renders the thumbnail when requested via <img src="">. the same applies for any other file types - the goal is actually to render thumbnails for everything, as per how your desktop renders the same files are thumbnails.

again, these are things for different image servers to one-up each other with.

Great work! This looks like it addresses my issues with image uploads, specifically the backgrounding. Thanks for open sourcing this. I'll throw it in my stack and give some feedback.

Reply on news.ycombinator.com