Hacker News Reader

Comment by majewsky | Hacker News Reader

> People seem to like it — we are seeing 5,000 hits to our API a minute to serve those cards that show when you hover over any link.

Uhm, no. It just means that people are hovering over links with their mouse. It does not imply any opinion about the previews.

> The original idea was conceived four years ago

When I was active at Wikipedia/Wikibooks 12 years ago, there was a user script floating around that did the same thing, except I'm not sure if it included an image. (Mediawiki allows a user to define custom CSS and JS that get embedded in every page of their user session.)

I don't mean to express dislike or downplay the hard work that went into this feature, just to add some context.

View on HN

> Uhm, no. It just means that people are hovering over links with their mouse. It does not imply any opinion about the previews.

When you evaluate features using engagement metrics, there are only two possibilities. If the metric is high, users love the feature. If the metric is low, users don't know the feature exists, and more alerts or "unread" badges must be added to help them learn.

My point is that the metric is bad.

Alternatively, if it's a feature you don't like:

If the metric is high, users are spending too much time on it. Also, it is responsible for every single frustration that users have with your product.

If the metric is low, the feature can be safely removed (see: Windows Start menu).

> If the metric is low, the feature can be safely removed (see: Windows Start menu).

That turned out really well...

Whoever is responsible for putting the touch start screen on Windows Server is responsible for 90% of my frustrations...

The best way to run Windows Server is without a desktop installed, using PowerShell to manage it. Even one-offs. You might know this but it’s worth saying just in case. Also worth noting that some server applications assume/require a desktop during install or operation (usually because they’re trash).

Yeah. Time on site, for example, might mean they can't find what they need; as opposed to getting what they need and staying for more.

Numbers are helpful, but sometimes they can be misinterpreted to mask the why.

> When you evaluate features using engagement metrics

No-no-no. Stop right there. "Engagement metrics" are the worst kind of metrics. Engagement means next to nothing. It just means that a user interacted with something. Was it good? Was it bad? Was it intentional? Was it by mistake? "Engagement metrics" answer none of that.

And yet... They are the easiest metrics to collect and too many companies use them as if they were meaningful.

If you hadn’t stopped right there, you might have seen the sarcasm in the rest of the post!

I'm case you were curious, WM has done more user testing than just the engagement metrics:

https://m.mediawiki.org/wiki/Special:MyLanguage/Wikimedia_Re...

Not sure what it says about them that they're mentioning worthless numbers instead of the way more useful data they got back during this survey.

But when the feature only needs you to hover over a link to activate (which people do anyway) you have no basis to claim that this is users "liking" the feature - it might just be noise from everyday regular navigation.

i was using wikipedia to with my son yesterday for a project and found it really annoying, so i'm in that list of people who hit the api but aren't in the list of people who seem to be "liking it"

Linked from the article is the final A/B test they ran: https://www.mediawiki.org/wiki/Page_Previews/2017-18_A/B_Tes...

They had better metrics than just number of API requests

Hey! (author here)

I sincerely regret the use of that statement "people seem to like it" in my post. I've now removed it as I worry this confuses my message so thank you for pointing it out. This really trivializes all the work that went into A/B testing this and how we measured success. I really recommend a read of https://www.mediawiki.org/wiki/Page_Previews#Success_Metrics... Side note: the volume of traffic was also wrong by several magnitudes... actually 0.5 million)

My main motivation when writing this post was to share how small changes require magnitudes of effort not to express the merits of the feature. As a developer who works with product owners a lot and often get asked how I can build things quicker (I'm sure many can relate). I wanted to provide something useful to other developers that easy looking things are not actually easy to build, so thanks for flagging.

With regards to the user script, yeh that's been around for some time and was the seed for this idea. It's just taken a long time to get that from such a small audience to a mainstream one. It doesn't downplay it in my opinion, just shows how far we've come.

Thanks for reading.

I do mean to express dislike, as I'm one of the millions of users giving Wikipedia a false metric and sense of satisfaction. I've seen the pop-up hundreds of times, and have considered it a hindrance to my process every single time.

Something that just happens accidentally is a bug, no matter how useful it might be if it were not happening accidentally.

You can disable it either from the screen, or if you have an account, permanently from your settings.

I completely agree with this.

When you go from reading the Atlantic, The Economist, The Washington Post... any content site... then begin to read a Wikipedia entry everything you know about reading an article no longer applies. You are now forced to change what you do.

This is just the most basic no no in web design - unplanned interactions, forcing a user to interact, forcing a user to actually think about the interface.

The developers have just forced every person who wants to consume content on Wikipedia to do it differently than on every other content site.

And there are literally countless millions of people who will have no idea how to disable it, who use computers every day but have no idea how to change something.

Just an incredible UX failure.

I expect you're remembering User:Lupin/popups.js[1], which dates back to August 2005.

It soon became known as Navigation Popups[2], and is still available today.

The main documentation for Page Previews mentions Navigation Popups.[3]

[1]: https://en.wikipedia.org/wiki/User:Lupin/popups.js

[2]: https://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_pop...

[3]: https://www.mediawiki.org/wiki/Page_Previews

I personally do not care for the link preview feature as I am one of those people who like to hover over a link so I can see the url leads (which is also why I detest URL shorteners).

I also have to wonder how much bandwidth Wikipedia is going to end up using to display unneeded previews.

> I personally do not care for the link preview feature as I am one of those people who like to hover over a link so I can see the url leads

But that's exactly what these preview popups are for! A bit heavier than the plain URL, but much more useful and friendlier.

(EDIT: I see there's a bug in that it displays the preview even if you don't stop the mouse there. Still, this can be lighter than requiring people to click through to the full articles, considering the preview can tell you what you need or that the link is not relevant for you, and slows you going down the rabbit hole to more links. Maybe it should be located at the bottom of the screen like the plain URL popup though?)

> I also have to wonder how much bandwidth Wikipedia is going to end up using to display unneeded previews.

This is interesting. There's an external arrow for external links, so I'm still good with this use case for inspecting links; for internal links I've found myself hovering where I previously clicked (and loaded) an additional page at least 50% of the time. Where I'm OK with a quick paragraph description and can then decide if the linked paragraph requires / inspires a deeper read, about 50% of the time.

I think this is a net reduction of bandwidth for my personal use-case. Wide A/B testing would reveal more, obviously.

Also, is 5K hits/minute really that impressive?

According to https://stats.wikimedia.org/EN/Sitemap.htm , English Wikipedia gets 88K views per minute.

I'm thinking it has to be heavily cached.

"5000 hits to our API per minute" does not say anything about whether they're measuring in front of or behind the cache. For all intents and purposes, the cache's endpoint is the API endpoint.

And uncached hits only would be an even more worthless metric.

I, too, think either that number is wrong or this wasn't a feature available to all users.

I also agree that using this metric as affirmation the feature is liked is dangerous; count me among the many here inadvertantly contributing without enjoying the feature.

That was also my first reaction, but I did not mention it because judging this (in relation to pageviews) would require more knowledge about the audience, e.g. how many users are using a mouse at all (vs. a touch screen, or pure keyboard navigation).

I imagine most people spend more time reading on Wikipedia as opposed to trying to hover over every link available?

That's less than one API hit per 17 English pages viewed - the overwhelming majority of the time, users do not use this feature at all, based on these numbers.

This response is pure sophistry.

I wouldn't be surprised if a lot of the pageviews are bots/crawlers. They don't hover over links.

but how many of those views are on mobile devices?

this is a desktop feature. in general, mobile traffic > desktop traffic.

we’d need wikipedia’s browsing statistics be sure, since some sites see different ratios than others. What if 1:8 wikipedia visits are on desktop? That would make the 5K/minute much more significant.

Number was very very very wrong. Actually it's 0.5 million. I've corrected the post.

it's still shown as 5.5k in grafana. what am i missing? the unit is events per minute.

The sample size for those graphs is 1% so you need to multiple that by 100. I personally checked the access logs to check they are consistent. That was where the confusion came from!

Ah, thank you.

>Uhm, no. It just means that people are hovering over links with their mouse.

in my case every single time was by accident whilst moving the mouse around, off of other text I was trying to read.

frankly, its bloody annoying

it covered the text i was reading

the search for yet another "disable" setting begins....

The popup has a little gear icon in the bottom right. You can disable it from there.

They put a small click button on the edge of a hover control? really?

I just tested it and it's trivial to click. Are you actually having difficulty or just pretending it's hard so you can be condescending?

I think the best place for these controls are usually on the feature itself. I don't want to have to search in some independent preferences UI every I want to see if something can be tweaked, but it makes sense to offer it there as well.

cheers!

> the search for yet another "disable" setting begins....

Your inability to figure out how to disable it was already counted as success!

> the rates of disabling the feature are negligibly low — a strong indicator that people find it useful.

https://medium.com/freely-sharing-the-sum-of-all-knowledge/w...

> To ensure that these rates were not artificially low due to usability issues, we also confirmed in a separate qualitative test that users were indeed able to find and operate the disable functionality if they desired to disable Page previews. (From their AB article)

“We thought the floating pop-up billboard showing you a preview of what’s inside when you glance at a building was a great feature as you walk down times square”

— bad UX designers in 2070

As far as I know the Lupin user-script was the foundation of the PagePreview feature. I guess the Reading group at WikiMedia rewrote the extension as part of MediaWiki core and tried to fix most edge cases on the way.

Reply on news.ycombinator.com