The Potential Pitfalls of Pagination | Hacker News Reader

The Potential Pitfalls of Pagination

jade.ellis.link

17 pointsbubblehack3r6 comments

Back in my J2EE days I knocked out a paginator server-side component that constructed a guaranteed ordering of items. This paginator stored its paging state in memcached with a consistent hash for a key in order to share paging indices among similar requests. Staleness was handled with a periodic check for additional result, or cache expiration after a while. The client had to handle the stale check and the "oops, sorry, let me refresh the paged list..." after idle. This was only suitable for roughly half the use cases.

It was complex, but it worked. Thank goodness I didn't have to deal with infinite scrolls back then.

Good article, I do wish it would talked about big-O and arbitrary sort orders.

Let's say there's lots of columns/fields to sort by. Not quite an arbitrary sort (e.g. can't sort by area when given a width and height column) but its still pretty flexible. Is is possible to get a O(1) pagination while avoiding the rolling-shutter problem and deleted-anchor problem?

Getting around the deleted-anchor problem with ordered-ID's seems like just using ID's as a index. Not a bad trick, but seems like it isn't generalizable for all different kinds of sorts.

Author here! This might be a case for one of the methods I missed, mentioned on the Reddit thread [0]. I've heard it called Cursor-based pagination or searchID-based pagination. The basic idea is you sort everything once and take a snapshot of the result. The user then gets an ID that refers to this snapshot. This often means that `next_page`/`previous_page` are opaque identifiers.

The tradeoff here is now you've got to keep that copy around - your server has to maintain some kind of state, you've got to garbage collect unused snapshots and it's impractical to scale this to millions of items.

[0]: https://old.reddit.com/r/programming/comments/1ev9hha/the_po...

One solution that we've been implementing in our APIs is the Seek Method (https://use-the-index-luke.com/sql/partial-results/fetch-nex...) which scales very well with large data sets in postgres, while avoiding some of the mentioned problems.

I've been facing the issue of shifting data with the eBay browse api. Unfortunately they don't offer ID based pagination. What's the best way to deal with it, manually track queried data yourself?

This is something that can only really be handled on the server side, but you can implement some mitigation strategies, like manually deduplicating entities. It's a lot of work for marginal gain, and you can never completely make up for problems on the server side in the end if you can't change the server.

Reply on news.ycombinator.com