"complicate" your lives? nginx is an amazing piece of technology and it's extremely simple. It's built for serving static files (and doing other things) better than anything else that exists. Use the right tool for the job.
Otherwise, even if you can speed it up, and/or minimize its memory usage, you're still wasting cycles and memory in your app that needn't be spent on it. To a point, this is perfectly fine, but at some point, you're (hopefully) going to want to scale, and I promise that this is the first thing you'll want to look at.