That's not true. They do, but you have to add it to the Vary header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Va...
This way user-agents know that the Accept header is involved in the final form of the response. (As another example, Firefox also doesn't take Accept into consideration when caching locally by default, and it does with Varry: Accept)