First impressions of the new Amazon Nova LLMs (via a new LLM-bedrock plugin)
simonwillison.net
simonwillison.net
> Anthropic’s decision to raise the price for Claude 3.5 Haiku isn’t looking great, given the continued competition at the bottom end of the market. Just a few months ago Claude 3 Haiku was the cheapest model available, now it’s hardly worth considering, undercut by even GPT-4o mini.
It's the opposite, Anthropic did the smart thing. The old Haiku 3 was competing with Flash, 4o-Mini and now Nova. And just like you correctly identified, that market has gotten very crowded very quickly.
Haiku 3.5 on the other hand has carved out a great pricing niche on its own. It's quite a lot better than the above models, but significantly cheaper than the flagship ones (Sonnet/4o/Gemini Pro). It's the best choice for many a prompt that falls into that bucket of difficulty.
FWIW, I've been building LLM-based products full-time for more than a year and use all of the named models, haven't had the chance to try Nova and Llama 3.3 yet, curious how they'll hold up. Especially Nova Pro vs Haiku 3.5 vs Llama 3.3.
Edit: Just ran a series of varied real-world language tasks on Nova Pro vs Haiku 3.5. Some were ties, on most Haiku 3.5 performed significantly better, and Nova never won. At this point I see no reason to use Nova Pro for general use besides extremely latency-sensitive usecases.
Anthropic's Claude 3 series was interesting in that they offered Haiku, Sonnet and Opus - cheapest, middle, best.
I had trouble back then figuring out when Sonnet was the right option - it was clearly interesting though, and I ended up experimenting with it for things where Haiku wasn't quite good enough but I didn't want to pay for Opus.
Anthropic released Sonnet 3.5 as the new "best" model, but then failed (so far) to follow it with Opus. You've made me realize that Haiku 3.5 is actually the new Sonnet - it fits that same spot, at least for the moment until 3.5 Opus appears.