1 Million responses per second on a 8 core 2 Ghz mahine means you have 16,000 clock cycles to formulate a response to each query.
Considering that [1] claims 1.43 cycles per byte for Chacha20, and assuming requests & responses are typically 200 bytes (they will compress well with HTTP header compression), the actual encryption part of https only consumes 1.43 * 200 = 286 cycles.
The remaining 15,716 clock cycles should be plenty to read the actual responses from the cache and parse the HTTP/2 request bytes.
Obviously all of the above assumes you design your infrastructure purely for performance. Writing everything in Python and NodeJS might suddenly burn up all those 15,716 clock cycles!