2,198 karma · joined June 22, 2016
From the system card someone linked elsewhere in the discussion
I don’t think we should assume answering a test would be easy for a Scantron machine just because it is very good at grading them, either.
If they can’t write an evaluation for the discriminator I agree. All the input data issues you highlight also apply to generators.
Fundamentally I’m not disagreeing with the article, but also think most people who care take the above approach because if you do care you read samples, find the issues, and patch them to hill climb better
Discriminating good answers is easier than generating them. Good evaluations write test sets for the discriminators to show when this is or isn’t true. Evaluating the outputs as the user might see them are more representative than having your generator do multiple tasks (e.g. solve a math query and format the output as a multiple choice answer).
Also, human labels are good but have problems of their own, it isn’t like by using a “different intelligence architecture” we elide all the possible errors. Good instructions to the evaluation model often translate directly to better human results, showing a correlation between these two sources of sampling intelligence.
- I don’t feel a right to decide the vibe of how a business curates its space. My bakery, coffee shop, local library, etc. all curate a space with an opinion. I don’t feel I have standing to assert that my preferences should dominate their choices.
As an aside, businesses are also an extension of the people, the best ones tend to just not be mode collapsed
A passing thought: the ethe of individuals in the 70s and 80s is important because of the people it informed in subsequent years. While many people still like to hack, code, etc., the relative proportion of people doing this and working in tech continues to diminish as the popularity and importance of the sector grows. I wonder if debt without values / a more cohered zeitgeist is better or worse?
Romance is for those who care, and most don’t. But it is so, so beautiful once you do.
Isn’t there a whole anti-trust case going on around this?
[0] https://www.nytimes.com/interactive/2023/10/24/business/goog...