Hacker News Reader

Comment by lhl | Hacker News Reader

I believe Amazon Q is running on Amazon's own Titan G1 model. I recently ran the "Premier" version (their highest end one) through my personal vibecheck test and was quite surprised by its RL. It was the only non-Chinese model I've tested to refuse to answer about Tiananmen Square and the only model I believe I've tested with this eval (over 50 at this point) that refused to answer about the LA riots. It also scored an impressive 0/6 on my reasoning/basic world understanding tests (underperforming most 3B models) but that's more capabilities than RL...

Amazon claims the Titan model is suitable for: "Supported use cases: RAG, agents, chat, chain of thought, open-ended text generation, brainstorming, summarization, code generation, table creation, data formatting, paraphrasing, rewriting, extraction, and Q&A." (it is not, lol)

View on HN

It is Titian under the hood. And it's absolutely crap.

Also fun fact, Titan's image generator will refuse any prompt that references Bezos because it "violates content policy"

If you want to do something useful on bedrock use Claude

I've been poking around this week and there's actually quite a few useful models on Bedrock (this is region dependent!) https://docs.aws.amazon.com/bedrock/latest/userguide/models-...

Claude Opus is supposedly only available in us-west-2, but is listed as "Unavailable" for me (Sonnet and Haiku are available). Cohere's Command R+ is also available and while less capable, for instruction following, I believe its superior to Anthropic's models. There's also Llama 3 70B Instruct and Mistral Large, both which are good for general tasks.

For those that haven't been closely following/testing the models available, I think Artificial Analysis' Quality vs Price charts isn't too bad a place to start https://artificialanalysis.ai/models although if you have specific tasks, it's best to eval some models are surprisingly good/bad at specific things.

Titan appears to be bad at everything though.

> cohere's Command R+ is also available and while less capable, for instruction following, I believe its superior to Anthropic's models

My experience recently is that its actually noticeably better for instruction following than Claude, but can be finicky if you're not careful about adhering to the prompt template. But between the RAG and multi-step tool use capabilities, even if it was slightly worse on the instruction-following side of things I'd still say, as you do, thats its much better than Claude on average.

Agree on titan as well. I recently was forced into a meeting with our AWS TAM, and they kept shoehorning Q into every conversation. I held my tongue knowing that titan was the model powering it under the hood.

Reply on news.ycombinator.com