TL;DR - Vanilla ubuntu, configure firewall, install nginx, install wordpress, turn on wordpress caching, install and configure varnish.
I'm sure my configurations aren't perfect, but they're a lot better than the defaults they ship with, and there doesn't seem to be a lot of "Here's some sensible settings" discussion on either project's website.
https://www.varnish-cache.org/trac/wiki/VCLExamples
At the bottom are some example templates that are a bit more tuned for production sites and are in use on some relatively large WordPress sites - last August, they claimed 8m pageviews/day on 3 frontend machines.
While I used to advocate W3TC, his support for Varnish purging has been ignored and I published a patch to fix it. Also, there are a number of tuning tweaks you can do - I don't know if you mounted your shmlog in ram or adjusted threading. The Debian packaged defaults are not very well planned - I don't know if Ubuntu blindly accepted Debian's defaults or repackaged with their own settings.
Nginx also includes its own proxy-cache which can eliminate the need for Varnish if you were still RAM starved. There are things you can do in Varnish VCL that you can't do in Nginx without writing your own module if that is an issue.
Since you're not running Apache with mod_rpaf, did you alter $_SERVER['REMOTE_ADDR'] processing? If not, most plugins and even commenting/trackback when used with Akismet would break.
Your worker-processes are probably not set well for nginx and there are a number of other tweaks that can help nginx quite a bit mostly with worker_connections and the rlimit_nofile (which if I recall, the default with a large site with w3tc's object caching would end up causing a bit of churn).
somaxconn might also have given a bit of a boost as you start having more traffic hit the backend. Not sure what version of PHP you get out of the repo or whether the backlog is patched, but, at some point, you'll need to adjust the backlog there for php-fpm - though, you would be well beyond the point of being able to run it on the ECS instance you tested with.
Good job benchmarking and actually including your config files.
Didn't touch shmlog configuration, or threading - everything I changed is on the post. I tried to keep it simple enough, in the end Varnish alone is probably enough for 99% of people.
You're right though, I should look at those VCL examples you posted, and thanks for taking a look through the files themselves, very helpful :)
* Not actually the original, but definitely the best.