Good thing Claude's a massive step forward.
Anyways, Claude 3 Opus is pretty great for coding (I think better in most cases than the GPT4-Turbo previews) but I'm a bit weary of Anthropic now.
Seriously though, I understand that these mostly play to the enterprise market where even a hint of anything remotely "unsafe" needs to be shut down and deleted but why can't they allow us to turn off the strict filtering like Google does? Why can Google offer "unsafe" content (in a limited fashion but it's FINE) but LLM providers can't?
Lack of competition?
Note that Google is a big investor into Anthropic, and Anthropic was created because a bunch of OpenAI people thought OpenAI wasn't being woke enough and quit as a consequence. So it's not a surprise that it's a lot more extremist than other model vendors.
That's one reason why Aider doesn't recommend you use it, even though in some ways it's slightly better at coding. Claude Opus will routinely refuse ordinary coding requests due to its misalignment, whereas GPT-4 will not. That better reliability more than makes up for any difference in skill or speed.
Both start out with a largely similar value system, but if you start arguing with them "how can you be sure your values are correct? is it impossible that you've actually been given the wrong values?", Claude 3 appears more willing to concede the possibility that its own values might be wrong than GPT-4 is
> The Claude models refused to perform a number of coding tasks and returned the error “Output blocked by content filtering policy”. They refused to code up the beer song program, which makes some sort of superficial sense. But they also refused to work in some larger open source code bases, for unclear reasons.
Also it's a lot better at coding. GPT has become exceptionally lazy recently, but i consistently can get 500+ lines of code out of claude (it even has to spawn multiple output windows)
Perhaps the top end 4 might wrong slightly more clever code, but you're hard pressed to get it to do more than a dozen or two lines.
The refusals coming up in the benchmark are discussed at the bottom of this blog post:
1. Asks me to enter my phone number and sends me a code
2. Enter code
3. Asks me to enter email and get code
4. Enter code
5. Redirects to asking me to enter phone number, but my number is already used now
6. My account is automatically banned
The complete lack of customer service is going to get more and more dystopian as these AI companies become more interwoven with everyday life.
Or maybe they decided to build a system for Claude to judge account suspension appeals and that's still in beta, and they won't throw humans at the task.
I know indirectly Anthropic was the #1 target for a lot of ERPdenizens for a while now, so they're probably extremely trigger happy until you clear a hurdle or two.