I wrote something similar years ago in Go, and settled on writing the data to a WARC file on disk (you can gzip the individual requests and concatenate to get random access), and also concatenating to a warc index file. The working index was kept in memory, while the warc index was read at startup.

My version acted as a proxy and would serve the latest entry from cache if a copy was cached. I had a special X-Skip-Cache header for when I wanted to go around the cache. (I can't remember if it handled https or if sites just didn't use https back then.)

My use-case was web scraping, particularly recipe and blog sites. I wanted to be able to develop my scraping code without re-hitting the sites all the time. Structuring it as a proxy allowed me to just write my python scraping code as if I was talking to the server.

Previously I'd written a layer on top of the python requests library to consult a cache stored in a directory (raw dumps of content / headers, with v2 involving git). But I found that required extra care when more than one script was running at once, and I liked the idea of storing it in a standardized format (WARC) that could be manipulated by other tools.