Why did infinite scroll fail at Etsy?
danwin.com
danwin.com
I like knowing:
* how many items are in the results
* how many pages there are (a derivative of the first, sure, but helpful to know)
* where I was if I accidentally close the tab and re-open it, or follow a link and need to hit the back button, or share my position with a friend (yes, most paginated results aren't stable in the long run, but usually are in the short run)
* easily jump to either end of the list (or, near to the end of the list, e.g. < 1, 2, 3, ... 98, 99, 100 >)
For example, I follow way too many people on twitter (1700!). I know I'm getting tweets throttled so I miss out on people I care about.
I also know that when I first picked up twitter years ago I made some noob mistakes and followed things I, well, shouldn't have (@Tide? I think a friend was working at P&G or something...).
Anyway, "People you follow" on the web UI is an infinite scroll. Even worse, it's buggy so sometimes you can trigger it to not register you're at the bottom and it won't load. AWESOME. I just want to get to the LAST PAGE. BUT I CAN'T.
And since it seems to load about 10-15 at a time, given that it takes about 1/2 a second to scroll down and wait for it to load, that means it'll take at least 60-85 seconds to reach the bottom - IF it doesn't crash (a reload takes me back to the top). Which means I've never been able to do it.
I had to pay one of those "show the folks you follow that haven't tweeted in n-months" just to try and prune the list, which helped me go from 2,000 to my present number.
Yes, this could be solved by sorting and filtering, but in the truest MVP sort of the world, why do all that extra work just for the "infinite scroll" fad? Switch back to pagination and I could accomplish all I needed and have a nice pruned list. And I bet it's less effort and has far fewer bugs than the current implementation.
I hope that in a few years enough data against infinite scroll will have cemented it as a generally accepted bad idea, only working in a few particular cases.
That mindset of design I think is the next step after "responsive" design. Rather than just scaling the visual design of the page, how about truly designing the user interaction to fit multiple scenarios. If it were up to me, I'd call it Symbiotic Design (responsive 2.0)
How is this a negative effect? The amount of stuff that users are buying should be pretty much the _only_ metric you care about. And they are probably not using search as much because browsing is much nicer with infinite scroll.
It sounds like they just did infinite scroll wrong.
If you read the comments page on the forum, the biggest complaint is that you lose your place in the scroll when you return to the scroll page. If they had listened to their users and made it so that you return to the spot where you clicked on the item, perhaps it would have been a success.
Also to your point, Dan talked about how they also thought that opening results in new tabs would be a sure fire winner (because power users open up multiple tabs to do comparisons), but it was also a significant negative effect. So clearly Etsy users back up when venturing into results and apparently expect to be at the exact same state they were before clicking through
Which makes sense, given that the multiple-tabs-at-a-time is often a power-user only thing. Etsy's users are much more "typical" in their usage of the browser; similar to Tumblr (who also ditched infinite scroll).
At that point, there's no easy way to get back to what I originally liked unless I had the forethought to open it in a tab or remember it exactly.
As for the metrics, it depends on how the business wants to position itself. It's evident from the quote that they care more about just the sales and would prefer to encourage more engagement with their site.
When building sites now, I usually contemplate using target=_blank on any lists just to aid this kind of pain point.
I agree if Etsy is a 100% commerce focused, but they aren't. Key to their success has been their ability to build up a community and sell stuff. Even if you weight down the impact of clicks and favourites the effect is still negative with less exploration and interaction happening. What would be the long term impact of sales of this reduction?
Items ordered is most helpful for making decisions when in context. It's awfully difficult to gauge health or make decisions with in isolation. A superficial short-term return may not be as beneficial on other levels - strategically, operationally, vendor ecosystem...
The article and other commenters have pointed at some, a few more more basic things worth knowing:
- return visit behavior. If infinite scroll diminished visitors favoriting items, what's the relative conversion for visitors returning to a list of favorites vs not? (features like this, Amazon Wish List etc, usually perform pretty well).
- avg order value. Visitors are buying the same number of things, but are they making the same $$ per order? Preserving avg order value but increasing item count also increases operations overhead (which could be worth it to encourage a broader vendor pool...)
- Clicks to items helps Etsy identify interesting/trending items&sellers independent of sales (move your data knowledge upstream!). Also increases the data size on individual users to build a taste profile and surface more items of interest at an individual level
Anyone who has worked on infinite scroll knows that the back button is a "interesting" technical challenge.
Especially when some A/B testing seem to show that it is a bad UX idea (for shopping goods).
Side thoughts: I wonder if they did A/B test "My apps" page on Google play (https://play.google.com/apps), it manages to have all the wrongs of infinite scrolling (no fixed ground, no sane back button) while not being infinite scrolling.
Generally yes, we would trade an increase purchases for anything else. The test was a negative effect because we forced people to stop using search, which was the feature we were experimenting with. Making buying less efficient, even if people want to buy so much that they eventually succeed, isn't a great result.
> And they are probably not using search as much because browsing is much nicer with infinite scroll.
No for two reasons. First, browse pages didn't have infinite scroll back when this test was done. And second, the point is that people seeing infinite scroll purchased less from search as compared to the control. This was a controlled A/B test.
As gfodor says, yes the back button did return you to your place (or it did, once we ironed out all of the problems with it in early incarnations).
EDIT: In fact, even if by some miracle it didn't, it probably still triggered a twitch of stress and irritation in users who assumed it would.
It doesn't have to be that way -- http://news.ycombinator.com/item?id=2592741
Infinite scroll flies in the face of the way the human brain works with groups and sets, makes it virtually impossible to usefully search within the current page using the browser's search feature, etc. And the positives are? Nothing, other than novelty.
It is one of the many recent examples of webdev/designers doing something because it is possible and trendy and new rather than because it adds any value.
Paged results with a well designed indication of where you are plus good server-side categorization and server-side search to filter results is far preferable to infinite scroll in every practical situation.
Keep the infinite scroll for those purely arty non-commercial story-telling sites, if you make me try to use an infinite scroll interface to buy stuff from you I will buy nothing.
To be perfectly honest opinions like yours provide guidance of what to focus on but in general this type of intuition usually turns out to only capture part of the dynamics when something is actually used by millions of people, and in many cases intuition turns out to be the polar opposite of the reality the data tells you.
http://www.useit.com/articles/better-than-reality-a-fundamen... (note - this is from 1998 and is still relevant!)
For example, try "scrolling" this list of acrylic sheets on inventables: https://www.inventables.com/categories/laser-cutting/acrylic...
Makes me want to find whomever implemented the system and shoot^H^H^H^H^H err explain to them how unusable it is.
Are there any sites that were designed from the start with infinite scroll that also have footers?
When I start browsing the sort of page that web developers are prone to applying infinite scroll to, I usually begin with a sort of semi-subconscious assumption that I'll just read down to the end of the page, or just the first N pages (for some smallish N) or something. I start doing so, but for a while don't realize that I'm never going to reach that point because the page is continually growing as I read it, and thus lose my way of quantifying (admittedly very vaguely) how much time I've wasted^W spent on it and where I should stop. My usual solution is to hit the "end" key (or its moral equivalent) and scroll up from there, ignoring whatever loads beyond that point.
Now from the developers point of view this may be a good way to get me to spend more time on their page, but if they're actually "gaming" me like that, it really just makes it enormously more obnoxious than it already was to begin with. Gaming intended or not, it strongly disinclines me to continue visiting the site in the future.
For me, I cannot use Etsy because there are simply too many products. I have no way to narrow down the volume of products to something which anywhere near approaches my ability to make a choice.
(At least with Amazon I can filter by department, then filter by four stars and higher...)
- the little scroll bar changes sizes randomly and moves places in the browser. Users can't predict this or know where it is, so they have to keep hunting for it when stuff loads in the background.
- if I browse a lot of items, scroll down a ways and then decide to scroll back up, it is much harder to find items that otherwise I would remember as being on page 3.
- The scrolling gets jagged as my browser barfs trying to shove more things in the list. In other words, even if it's faster it FEELS slower and less responsive.
Users have a very different set of goals, with Google we know that the falloff from the top search result to the bottom is huge, people expect that the relevant results are the top couple.
With product search there's different goals, for example browsing an minimal set of criteria looking for a results--you're looking a pair of jeans and starting with 36x32; or the opposite you just want to know the price of a pair of New Balance 990s in a 11 4E.
In the latter case infinite scroll or not doesn't come into play, but the browsing case it does, and from the point of view as an Etsy customer, I have the most difficulties.
One of the things I've found the most is people do better with a task if they can understand how long it's going to take. If you know there's 100 search results over 5 pages, you can decide if that's too much to go through. As an aside, this is true of a lot of things, you'll do better with pull-ups if I tell you to do 20 and give you a count, than just tell you to keep going until I say "stop".
With infinite scroll you've got no idea how much effort looking through the total search results is going to take. Etsy don't make things easy, as an example Art > Custom Portraits[1]. I have no idea how many results there are, I can scroll all the way to the bottom and find out there are at least 8 pages, but that's it.
The search results themselves are pretty snappy, so I don't see a huge advantage of infinite scrolling. I do think the search results are pretty bad and the lack of filtering is a problem. From Art > Custom Portraits I can filter on just Pets or Silhouettes or More (which I assume is everything else). It's not obvious what these links do either, they are just floating at the top.
Some better filtering, e.g. by price, would help. I'm not sure what other metadata Etsy have that would be useful for an great faceted search.
As an aside, I've been working on (product) search for over a decade and I'm close by the Etsy offices; over a coffee I'd be happy to discuss search. Part of my New Years resolution is to do a better job of networking, so I'm happy to extend this offer to any other NYC based e-commerce shops.
[1] http://www.etsy.com/browse/art/custom-portraits?ref=br_nav_n...
Clicking is so much faster than dragging, (barring terrible page load times)... a part of me wishes no one ever invented scrolling.
I'm the biggest proponent of avoiding monolithic tests and having clear and testable hypotheses. I'm glad there's a high profile example to point to now. Thanks.
---
Q: Hey John Doe, check out page 3 at site.com, I totally want to buy item 3 and 6.
A: Obviously by the time John Doe checks the site/link item 3 and 6 might be something entirely different. People send direct links to items.
---
Q: Oh no, I don't know where I am in the scroll list.
A: Who gives a rats ass? Your content is right there. Interesting items you found along they you open in a new tab so that you don't have to backtrack.
---
Q: But I want to quick jump to page 45?
A: Why 45? You just pulled that number from your hat (read: ass) anyway. What you really should be looking for is the search field.
---
Q: But I never get to see the footer!
A: The what...? I don't think I've looked at a site footer since '94. Navigation is at the top, and bullshit at the bottom.
---
I just don't get peoples need to "know where they are", like it made a difference?
1. People only care about the first few search results.
2. Infinite scroll exists solely to make it easier to scroll past the initial view.
3. Therefore, infinite scroll made it easier to scroll away from the results people care about.
There you go.
By the way, I don't get why Google's "instant results as you type" was cited as a reason to pursue this infinite scroll feature. Those are totally different things.
"Instant results" is for faster display of the first few results, which is great for search. "Infinite scroll" is for scrolling through long streams of information -- great for newsfeeds and timelines, but not for search results where you only care about the first few.
Google doesn't even implement infinite scroll in their results.
This concept plays off the simple fact people are fundamentally bad at choosing between many options.
I'm curious if this is the factor that caused negative results with the infinite scroll. I'm also curious what would happen if you started only returning 5 results/search...
Infinite scroll is a good solution, if you are building lobster traps - sites where you want people to spend endless amounts of time. (http://gizmodo.com/5586337/pandas-and-lobsters-why-google-ca...). It clearly works for Facebook, Twitter and Quora.
Although Etsy has some aspects of lobster trap, it still is more transactional than FB and other social time killers.
Just like many people here, I hate infinite scrolling with passion.
In truth, your view all suggestion as a way to fix pagination is akin to me making a paginate button to solve infinite scrolling.
We shelved the infinite scroll at the time out of haste.
It turned out to be a happy accident, as active users refreshing would generate a ton of pageviews.
Not only that, but the act of clicking refresh to "get more" is a nice way to get the user to engage.
I tried to buy something that I knew was on Etsy during the holidays, and after about an hour of looking (their category breakdown reminds me of reading through the yellow pages - remember those?) I gave up and bought from Amazon instead.
If the code is not providing real value, it should be thrown out.
I'm so happy to see them measuring "median item impressions" rather than "mean item impressions." Many of the underlying variables describing consumer behavior aren't normally distributed. Talking about "average number of friends invited" when 50% of your users invite 0, 25% invite 1, 13% invite 2, etc. but 3 users invite 5,000 will necessarily lead you to propose bad product ideas and decisions.
However, I'm curious about the experimental design. For example, was this tested on new users, old users, or both? Changing a fundamental part of a site's experience like this will have some cost as users acclimate. I'd wager Etsy's audience is less technically inclined, too, so it might take them longer to acclimate.
They also commit a small fallacy when they talk about how they should have done it instead, and IMO it's a fallacy that frequent A/B testing encourages people to commit. They suggest that instead they should have determined first whether more items are better and faster items are better.
On the most surface level, perhaps there's something about more items AND faster items that outperforms either one or the other in isolation. That's easy enough to accomplish, technically. You use different statistical tests, but it's possible at the cost of perhaps a larger sample size.
On a deeper level, you're providing the users with a fairly different overall user experience. Their sense of where things are placed, what they're supposed to do when they want to "see more," how they know they have the opportunity to "see more," etc. are aspects of the infinite scroll design that aren't encapsulated in either rendering more items or rendering those items more quickly.
For example, can users bookmark specific search result pages under the current design? Can they still do the same thing under the "infinite scroll" design? I imagine there are lots of little things like this and that the UX difference alone would have a larger impact on the results than just changing the number of products per page.
To get more meaningful results from this, I'd run this experiment under the following assumptions.
1. Assume that existing users will be more impacted by this change than new users. Therefore the cost of "failure" for existing users is higher.
2. Assume that at the end of the day the #1 thing Etsy cares about is "dollar throughput" of the Etsy platform. Engagement, favoriting, searching, etc. are all positive indicators of an increased dollar velocity.
3. Assume they have information about what aspects of a users' first visit are indicators of their long-term ability to contribute to Etsy's dollar throughput.
4. Assume that eventually every user will have the same experience, new or existing alike.
So, I'd start by running the experiment with new users only. Over the course of a week or a month I'd put a % of the users who joined each day into the "infinite scroll" bucket. I'd then run the study as a longitudinal study.
Assumption (3) can guide us as to whether we need to cut off the experiment early. The length of the study would be determined by the particularities of an Etsy user's life-cycle, e.g., maybe given a cohort of users, we care about the length of time it takes 75% of the eventual purchasers to make their first purchase.
Because of assumption (4) we know that if the "infinite scroll" design is terrible for new users, we never have to bother testing it on existing users.
[1]: Non-technical users, in particular, are sensitive to sudden change. I forget where, but I read a research paper once that implied that the worst thing you can do to harm a person's user experience is change the placement of links, buttons, etc. You can change the color, text, icons, etc. but if you change the placement, they essentially have to "re-learn" the interface.
IIRC, the users were given a task (e.g., "create a document") and they measured two core variables: time to accomplish the given task and time until their "performance" at a given task was equivalent to the control user interface.
Changing the placement of a certain action in the UI had a deeper and longer-lasting impact on users' ability to perform tasks than changing anything else about the UI by a large margin.
http://image.slidesharecdn.com/continuous-experimentation-12...
I didn't rewatch the entire presentation but McKinley does discuss the user makeup in the experimental groups...yes they do account for different kinds of users, and the most drastic difference between user behavior are between sellers and non-sellers. Someone from Etsy would have to talk about how much slicing-and-dicing of the demographic that they do...but even if infinite scroll was good for some users (new users without pagination-related habits) and not for others, it's probably not a good idea to have two kinds of search experience in the hopes that the "oldies" eventually figure it out...based solely on how hard it is to implement infinite-scroll in the technical sense.
At Etsy, introducing infinite scroll resulted in "fewer clicks on the results and fewer items 'favorited' from the infinite results page". On an Etsy search results page [1], clicking on an item bounces you to another page. And on sites with "infinite scroll", this is typically a very uncomfortable experience, particularly when trying to get back. Depending on the implementation, you're wind up back at the beginning of the result set; even if you don't, it's usually a fairly bumpy ride, with the time taken for the page to reload its data and the jumps in scrolling as everything loads in. Even if it technically works, the kind of sensation this brings about is enough to discourage someone from actually clicking through. There's an negative association that develops with clicking on these items, the foreboding feeling that you'll end up losing your place, such that one tries to do so as little as possible -- in line with what was observed from the Etsy experiment.
On the Etsy search page, you can indeed "favorite" items without going into the item's actual page. But that's not something one is likely to do based on a tiny little thumbnail -- one would usually first click through, see a bigger picture, and possibly read the description below. It doesn't help that, on the results page, the "favorite" button is but a tiny little icon, that only appears if you directly mouse over the thumbnail; meanwhile, on an actual item page, it's right there under "Add to Cart" [2]. Perhaps users weren't even aware that you could do this from the search page.
With a site like Etsy, where lists are a means to an end -- a way to get to information on other pages -- it's no surprise that infinite scroll performs quite poorly, as opposed to content consumption sites, where browsing is a self-contained experience of its own.
(As an aside, it's fairly silly to compare infinite scroll with Google's Instant Search. Instant is well-liked because it gets you to your search results faster; this being Google, the user isn't there to hang around and enjoy the scenery, but to get to the information they're looking for as quickly as possible. And Google's results pages themselves still use pagination, despite their experimenting with infinite scrolling back in 2011 [3] -- a change that, quite clearly, didn't make the cut.)
[1] http://www.etsy.com/search?q=test&view_type=gallery&...
[2] http://www.etsy.com/listing/65114535/12-boracilicate-glass-t...
[3] http://searchenginewatch.com/article/2103479/Google-Experime...