Yes, I think the way to do that is to have nginx switch new requests to the updated app server, and shut down the first one when it's not serving requests any more.
I don't know how you'd communicate with nginx inside the container to tell it to switch, though, or how you would be able to know when all requests were done on the old container. Hopefully there's an easy way.
Too bad that a single SIGHUP becomes so complicated with containers, but I guess it's a tradeoff.
About the Beanstalk, I haven't looked into it too much, maybe that can help with deployment. I'd be grateful if you could let me know how it worked, if you ever try it out.