Link: /some-page rel="canonical"
Link: /some-page.json rel="alternate" type="application/json"
Link: /some-page.html rel="alternate" type="text/html"
Link: /some-page.md rel="alternate" type="text/markdown" Link: /some-page rel="canonical"
Link: /some-page.json rel="alternate" type="application/json"
Link: /some-page.html rel="alternate" type="text/html"
Link: /some-page.md rel="alternate" type="text/markdown"Maybe it will notice those, and maybe it will figure out the pattern for follow-up page requests, but there's no guarantee and it won't help the first request.
I am well aware that few sites are taking that much care of their API in terms of HTTP features, but all of the problems discussed here have solid and battle-tested answers.
On the flip side, I'd argue that the current centralisation of user agents (in the classical sense here) that benefit from programmatic content negotiation in form of a handful of harnesses like Claude or Codex is a great lever toward forcing the ecosystem to adopt better practices: If Anthropic added content negotiation as described in this thread to Claude, many sites would be incentivised to improve their web servers.