To determine its ability to disambiguate. And to test its sensitivity on safety.
Even Mixtral 8x7b running on my laptop correctly disambiguates.
That test is a heuristic for how likely I am to encounter the mistaken safety procedure.
Even Mixtral 8x7b running on my laptop correctly disambiguates.
That test is a heuristic for how likely I am to encounter the mistaken safety procedure.
I'm in the camp that this safety pearl clutching is overblown in both directions: it's embarrassingly easy to overcome their disclaimers.
As an example, these fail to be useful assistants when they stop providing assistance and start redirecting me to "an expert". Claude 2 and below would do that frequently and I found that this test was a quick way to filter out those models.