I tried after seeing this benchmark [0] and thinking "that must be wrong". I got the same results.
Benchmarking on a 1GB, 1vCPU cloud server (to get an idea of what it would handle), with minimal VCL and one 15kb file in Varnish's in-memory cache it used over half the CPU (with hitch, unsurprisingly, using the other half for SSL termination).
[0] https://twitter.com/BlackIkeEagle/status/1274977171400847361
But as you can see, it's not much CPU at 450'000 reqs/sec, and the response time is just fine, at 1ms average. The benchmark is at first hitting one of my emulated VMs, but it gets cached, which is a fast-path.
I think benchmarking anything other than a cache hit, is not doing a cache justice. Generally, the cache hit ratio is very high when properly configured. I did not look into what the Litespeed benchmark was doing though, so I can't comment on that.
What exactly is your VCL here?
And how Varnish is initialised ExecStart=/usr/sbin/varnishd -a :80 -a 127.0.0.1:6081,PROXY -f /etc/varnish/default.vcl -s malloc,64m -p feature=+http2
I've verified everything is a cache hit, and the ratio is 1.000 by the end of testing.
Screenshots and more in this thread: https://twitter.com/peterbowyer/status/1292765478100795393
I also think we should consider how CPU usage is calculated as well. I don't know how it's done inside VMs, but I can easily imagine that there are several cases where it would include time where the CPU is just waiting for things like access to memory or even waiting on I/O as long as that happens through a shared memory ring buffer, via spinning or something to that effect. So, if that is how it works, then over-subscription of VMs could cause them to show higher CPU usage. But, I don't know, and I hope someone more familiar with the subject will join the discussion.
I haven't used strace before but I ran it and perf top and have put images at https://twitter.com/peterbowyer/status/1294181276061241344.
The __memset_avx2_erms is present even with 5 concurrent requests (but at 11% overhead then) and the core dump still happens.
Very odd.