Hacker News Reader

Comment by drcongo | Hacker News Reader

Author of the post here. Thanks for posting it, it's little out of date in terms of the references in there but I stand by the sentiment.

Since I wrote the article I feel like I've seen something of a drop in usage of the kinds of techniques described, total anecdata though.

Patreon posted below looks very interesting. Thanks for the heads up.

View on HN

From your article, you wrote:

>If you want the whole story here you should also be sat in a room testing this modal overlay with real people. Ask them questions:

“Do you like that overlay..." [...] It’s extremely unlikely that they like it.

Your opinion may ultimately be correct but your justification for it is flawed.

If you survey people, you get what's called a "stated preference." (SP) [1]

However, there's another concept called a "revealed preference" (RP) [2] and this often contradicts the "stated preference."

The web analytics Ramit Sethi believed in may also be flawed. However, the point is that statistics collection attempts to uncover "revealed preferences". Since RP is often hidden, the tools for uncovering it are not surveys. Instead, they use hidden cameras, or Point-Of-Sales transaction data, or eye tracking, or web analytics, etc.

There are many examples where RP yields more insight than SP. This is counterintuitive so we end up thinking SP is the correct perspective because it "feels right." Many real world examples have proven SP to be incorrect such as retail sales events, airlines, dating & romance, etc. I can expand those examples, or you can search google if you're curious about them. Annoying ad overlays on web content may be another area where RP trumps SP. I haven't researched it enough to know.

Your article gives more weight to SP than RP with no justification why it should.

[1]https://en.wikipedia.org/wiki/Choice_modelling#Stated_prefer...

[2]https://en.wikipedia.org/wiki/Revealed_preference

Okay, I challenge you to discover a "revealed preference" that says they like overlays. Not that they click them, not that it "results in an action," but that they like them and they will be more likely to return to the site because of the positive and enjoyable experience of the popup.

There are a few dozen confounding variables in your analysis as well if you want to look for the revealed preference. Why are users clicking the modals? Are they certain that they have a choice? In the article, the author makes the excellent point that many people believe that they must complete the modal to continue reading the article. That's not preference, that's coercion.

So, sure, it's valuable to understand revealed preferences without bias of leading questions. Absolutely. But you still have to keep your head on and understand the variables you're measuring, the confounding variables, and larger effects at play.

This question comes up a lot in user experience design, with stated preference vs. revealed preference (how you do user testing and surveying vs. how users actually use the product unprompted), and the larger macro-scale question of what actually makes a quality experience. Often making all the decisions about the user experience from microinteraction tests leads to a poor overall experience, and it's exactly because of too much focus on 'what works' without understanding how the measurements are being made and how we should interpret that data.

>, I challenge you to discover a "revealed preference" that says they like overlays.

That's not my position. I never proposed that web surfers "like" modal overlays. I think we can all agree that intrusive overlays annoy everyone.

The "revealed preference" I'm talking about isn't referencing the modal overlays specifically. The RP reference is to the "whole story" (to put it in author's words). To me, the "whole story" is how the website measures the value of its content, how it monetizes it, what % of people never want to return, what is a sustainable audience, etc, etc.

There are multiple dimensions to what people "like/dislike" and people assign different (and hidden) weights to them.

The author wrote: >They will falsely conclude that people love these modal overlays.

I've never heard any sane UI developer or programmer expound that position. That looks like twisting the words of what they actually think (aka a straw man). The actual conclusion by them is more accurately portrayed as, "they conclude that greater % people love THE CONTENT MORE THAN THE INCONVENIENCES of these modal overlays. The % of people who tc;dr is real but small enough % to be acceptable collateral damage."

The RP behavior analysis attempts to answer that second thesis. Surveys and questionnaires are the wrong tool for it.

You are the only person who has even mentioned surveys or questionnaires. Arguing with yourself over something only you have mentioned is the very definition of straw man.

edit: to clarify - the asking them questions bit mentioned in the article was part of a user testing session. We'd already watched them walk through it trying to complete the tasks required, the follow up questions were for clarification on why they signed up for a non-existent newsletter that we didn't ask them to sign up for. At no point did we do a survey or a questionnaire.

>You are the only person who has even mentioned surveys or questionnaires

I'm simply using a generalization of something you wrote to ease the flow of the discussion here. You wrote:

  *"Ask them questions:"*

Why is my restatement of that to be a "survey/questionnaire" an unfair generalization? It wasn't a malicious intent to mischaracterize you.

If we did a search &replace for "survey/questionnaire" and changed it to acryonym "ATQ" for "Ask Them Questions", does the meaning of my post really change?

EDIT to YOUR EDIT:

>"At no point did we do a survey or a questionnaire."

Sorry, I wasn't contending that you did a literal survey. I was simply a placeholder label for discussion purposes.

If we can get past the misunderstanding of labels, I'd think it would be more helpful if you actually address the substance of my previous argument: why does "ATQ" about modal overlays trump "revealed preferences" about the value of web content?

>so it seems a little odd to pick that bit as the thing to take issue with.

It's not odd because the "Ask Them Questions" was your only visible justification to convince us.

We're in edit hell now!

Further edit in response to the reply below, which is too deep to reply to – In the interests of brevity, clarity and sticking to the subject, the entire process used to test the modals was not included in the article. I was much more concerned with writing a macro view of the issue than a how-to manual. There's plenty of things worth arguing about in the piece, lots of it is pure opinion, and some of it is probably plain wrong so it seems a little odd to pick that bit as the thing to take issue with. Maybe I should have been much clearer about the testing process.

Your argument is simply that analysts are trapped in a local maxima, optimizing for clicks as a proxy for consumer lifetime value. However, we have no knowledge of the actual analysis techniques. In fact, the company may be optimizing lifetime value with lightboxes.

My argument is that they are not optimizing for lifetime value, and in fact are wrong. It's difficult to measure, thus they go for the route that is easier to measure. Classic McNamara/quantitative fallacy, and I think it applies cleanly here.

If you could update the references, who's doing it right today?

I wish I could remember some of the good ones I've found, and I wish I'd started collecting screenshots of them and done a tumblr in support of their efforts.

Reply on news.ycombinator.com