Hacker News Reader

Comment by verst | Hacker News Reader

Related: Do you think SDKs for these APIs should themselves require you to interact with the cursors?

If I ask a SDK to get me 100 items but internally it returns only 50 before needing to make another call with a marker or cursor It would be nice if the SDK had an option to handle this for me (possibly exposing cursors still for those who really want it).

View on HN

SDKs should typically return a lazy iterator in languages which support it, so caller do not even care about pagination and just iterate on it until they don't want any more data; while the SDK fetches new pages when needed.

In pseudo-Python:

    def get_stuff():
        url = "https://example.org/api.php"
        while url:
            response = requests.get(url)
            yield from response.json()
            url = parse_link(response.headers["link"]).next

and you call it with:

    for item in get_stuff():
        print(item)

The AWS Python API users “paginators” to allow you to iterate over results with a single for loop and it handles the pagination in the background. The underlying API - which is also exposes - uses and opaque “NextToken”

What if it fails after 50?

Maybe return the items it found so far and provide the cursor it failed on? Have a way to optionally provide the cursor? I can think of some ways I'd implement this in an SDK. Probably for simplicity I'd have some sort of JSON status field indicating the request (traversing all cursors) completed. If it's not complete I still would look for the next cursor it returned and do the usual thing manually.

Reply on news.ycombinator.com