That is not how LLMs work. It isn't some intelligence that is separate from and also contains a database of knowledge that you can prune at will.
167 karma · joined May 24, 2023
"What was the artist trying to achieve?"
How is that "disarmingly straightforward"? Does that mean it's a straightforward question for anyone to answer because it's obvious, or that it's straightforward for the viewer of the art because in that case the answer could be anything? If the former, that's an incredibly difficult question to answer and kind of nullifies the point of this entire exercise.
Meanwhile humans can pop in and leave little morsels like this and blame it on the model.
Oh my, how would anyone ever have gotten that impression?!
Some model cards do show regressions on benchmarks for newer models on specific tasks: https://storage.googleapis.com/deepmind-media/Model-Cards/Ge...
This wasn't a new model but updates to models backed by numbers being better can make the model worse: https://openai.com/index/sycophancy-in-gpt-4o/
The slight increases in performance/benchmarks may be just noise: https://arxiv.org/pdf/2602.07150
Also, if a human does this, you can replace them and get a human who will not do it. The default for an LLM is to generate plausible-looking text that may or may not be completely incoherent. That is not the default for a human. Again, if you find that your colleague consistently fabricates APIs, you can hire someone who isn't crazy instead, but you cannot do the same with LLMs.
IME people would benefit greatly from the process, albeit tedious and time-consuming, of testing out the same prompt sequence/session with the exact same model multiple times. It becomes clear extremely quickly how capable but unreliable and inconsistent a model can be even when given the same context. If you have ever completed a long, complicated task with an agent and then lost the session and tried doing the same thing again from scratch you may have had the experience of seeing the subtle changes that come up in the model's thinking which lead it to accept or reject certain paths and ignore or incorporate prompt instructions like the one you've provided.
You know companies lie and overstate things, right?
> "Google is not a conventional company. We do not intend to become one."
> OpenAI being founded as a nonprofit and becoming for profit.
> Didn't Anthropic literally say they wouldn't train on your data or keep it for longer than 30 days unless legally required, and then decided to opt people in to having their conversations used for training?
The worst thing is when someone cites a video or a demo of an AI doing something and says, "See! It's here!" Remember when the Devin video came out years ago?
You can say "eventually" AI will be able to do xyz, but eventually the sun will blow up, too, so what the fuck are we talking about?
That's pretty amazing. Not due to a cascading failure from someone changing a config deep inside of a system that caused a bunch of unintended effects, just someone who messed up writing a shell script?