Learning a phrase from humans doesn’t prevent a model from overusing it or making it a recognizable stylistic habit.

Weird user preferences during RLHF. Also how we got bulleted lists and emojis everywhere.

All AI semantic tendencies were “learned” from us. That doesn’t change anything.