Was thinking about this today in context of hiring.

We have these amazing LLMs that are continually improving. Yet if you say to them, here’s my business, now takeover the marketing department. You will end up with so much output that’s not localized that the value of the whole output is worth very little. Yet when you have a highly experienced localized marketing leader use the LLM to speed up work the whole output is very valuable.

I don’t think this problem is solved by solely defining preferences better. It’s clear a human adaptation layer beyond solely RLHF is needed for at least the short term.