Hacker News Reader

Comment by reenorap | Hacker News Reader

Boris, you're seeing a ton of anecdotes here and Claude has done something that has affected a bunch of their most fervent users.

Jeff Bezos famously said that if the anecdotes are contradicting the metrics, then the metrics are measuring the wrong things. I suggest you take the anecdotes here seriously and figure out where/why the metrics are wrong.

View on HN

On the subject of metrics, better user-facing metrics to understand and debug usage patterns would be a great addition. I'd love an easier way to understand the ave cost incurred by a specific skill, for example. (If I'm missing something obvious, let me know.)

Baking deeper analytics into CC would be helpful... similar to ccusage perhaps: https://github.com/ryoppippi/ccusage

This is useful if you want to keep an eye on what claude's actually doing behind the scenes: https://github.com/simple10/agents-observe

But the default 1M context window just rolled out a few weeks ago. If refreshing old sessions on 1M context windows is the problem, it's completely aligned with what Boris is saying.

We are taking it seriously, and are continuing to investigate. We are not trusting the metrics.

Thank you

The quantitative ux research team at Google was created for exactly this problem: a service which became popular before the right metrics existed, meaning metrics need to be derived first, then optimized. We would observe users (irl), read their logs, then generate experiments to improve the behavior as measured by logs, and return to see if the experiment improves irl experiences. There were not many of us and we are around :)

I worked with Boris in the past and in my experience, Boris cares deeply about the customer. I'd vouch that Boris really cares about the issue people are running into.

“ Hello. My name is Mr. Sirob,”

https://amphetamem.es/meme/?id=the-simpsons_04_12_89&timesta...

But no other user has yet come and said "I worked with ajma in the past ..." so how can we trust your judgement about Boris?

I saw this guy named Claude saying ajma is a genius!

Nice try boris

Google products ux is widely acknowledged to be a steaming pile of shit though, so I am not sure you should follow their example.

Many of the metrics they use are obviously actively user hostile.

Metrics and quantitative ux results in really bad software, making it rigid while optimizing for the wrong things.

The most obvious example is Google creating multiple steps for Login where you have to enter your password after you put in your user.

I wonder what metric lead to that decision or was it a political decision to make it seem like their "old" software has some new feature.

If you mean Google website login, that step is needed because the email address is used to determine which identity provider to use. E.g. I have three different accounts that branch off from that same initial login flow.

One is my person "gmail.com" account, and the other two go through enteprise identity providers related to my employment and their G-Suite licenses. So after I put in one of these three email addresses, I get prompted for the appropriate next step. Only one of them involves giving a password to a Google server. The other two are redirects to completely separate login systems operated by my employer.

I mean I get it logically makes sense. But it still seems like a waste of time for a small percentage of use cases.

Maybe a better approach is put in your login have it automatically detect if it requires an identity provider. Gray out the password to signal to the user password is not necessary and automatically redirect.

Less clicking, don't break flow and think of a smoother solution.

Reply on news.ycombinator.com