Ask HN: How do companies like OpenAI, Perplexity fine tune rich output?

I see fine tune as one of the major ways companies like OpenAI, Perplexity, Claude companies differ when it comes to provide higher quality of answers (correct me if I am wrong).

One curious question is how do they fine tune rich data (markdown, html outputs, tables, graphs etc) at scale. Currently, performing fine tuning involves the laborious process of carefully editing inputs (prompts) and outputs one by one. Becomes more difficult as the data context increases and one has to carefully examine the input data and provide the right output including things like formatting, grammar, UI etc.

Considering such a wide variety of questions they are processing, it amazes me how are they doing it at scale. Any thoughts?