Gpt2-Chatbot Removed from Lmsys
lmsys.org
lmsys.org
> *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here].
That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section:
> Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing.
> Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps:
> 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users.
> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
> 4. Remove the model from Arena.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
So... make the users do the work for free, but never publicly publish the data? That seems kinda scummy to me.
I was really motivated with the whole public ranking thing, but knowing I'm just doing work for another company with the data never being published sucks the fun out of it.
If there was an opt-out for private model voting, I would choose that simply because I don't care to give some corporation extremely useful data for so little pay.