Such as with RLHF-boosted writing from LLMs, where side by side comparisons of human vs. AI writing have been used as proof of revealed preference for AI. But an A/B test of a few sentences doesn't mean that someone would enjoy reading a 5000 word blog post in the same style, as the densely packed rhetorical flourishes become exhausting and repetitive tells start to jump off the page. Nor does it capture the signal/noise problem where despite technical ability and incredible verbosity, realization sets in that nothing in 2000 words has said anything at all while continually stringing the reader along. A side by side comparison with a few AI posters might be preferred in a five 5 minute test, but seeing the same visual elements from three frontier models in a loop on every surface for your entire life could quickly become actively repulsive.
And the problem with agency arises when limited freedom exists to truly make a different choice because the available options are externally determined. I've seen the argument made on HN that people's revealed preference is for AI maximalism, pointing to the fact that consumers keep buying products with AI, hundreds of billions of dollars are spent building new datacenters every year, workers (including creatives!) and students report using AI for daily work, etc.
But consumers aren't really given true choices that could reveal a real preference. The tech giants behind the phone duopoly have crammed AI features into both OS choices. With few exceptions every large company made an "AI strategy" their main focus since 2023 and competed to put chatbots and GenAI features into every inch of their product lines. Hyperscalers started a bidding war on datacenter buildouts in the expectation of future demand back when really only ChatGPT had any real market penetration outside the tech industry. Students and workers feel pressure to keep up with their peers by using AI to avoid getting fired/worse grades, artists struggling for work worry clients will go elsewhere if they take the time for their normal process.
Revealed preference when used haphazardly could be used to come to absurd conclusions like "people want to get scammed", "people like to be overweight", etc. Broadly and incautiously applied it just becomes a way to justify any given status quo as inherently optimally and reflective of collective will. With a creepy paternalistic implication that regardless of what you think you believe, forcing you do something against your will is secretly what you really want, and will make you happy even if you pretend not to realize it.
It's pretty questionable if a preference truly exists as "revealed" for something like AI art when it's preconditioned on active deception. An AI art-hater who gets tricked into picking AI doesn't mean they're lying/confused and actually love it. That sort of reasoning can get uncomfortably close to an anti-democratic argument for mass manipulation of people who can't be trusted to advocate in their own best interests.