Weird user preferences during RLHF. Also how we got bulleted lists and emojis everywhere.