Comment by deathanatos | Hacker News Reader

Comment by deathanatos | Hacker News Reader

Pagination is one of those things I feel like so many of these things get wrong. LIMIT/OFFSET (or as MS likes to call it, TOP/SEEK) style results in O(n²) operations; an automated tool trying to pull the entirety of a large collection in such a scenario is not good. I have to again recommend the excellent "Pagination done the Right Way" presentation[1] from Use the Index, Luke (an equally excellent site).

Just return a link to the "next page" (make the next page opaque); while this removes the ability to of the client to go to an arbitrary page on its own, in practice, I've never seen that matter, and the generalized URL format allows you to seek instead of by LIMIT/OFFSET by something indexable by the DB; HTTP even includes a standardized header for just this purpose[2].

I also think the generalized URL is more in line with Fielding's definition of REST, in that the response links to another resource. (I don't know if being in the header invalidates this; to me, it does not.)

If you get the metadata out of the data section of your response, and move it to the headers, where it belongs, this usually then lets you keep the "collection" of items you are return as an array (because if you need to put metadata along side it, you need:

  {
     "the_data_you_really_want": [1, 2, 3],
     "metadata, e.g., pagination": {}
  }

vs.

  ["the", "data", "you", "wanted", "in a sensible format"]

)

(I've seen metadata-pushed-into-the-body turn API endpoints that literally need to return no more than "true" or "false" into object that then force a client to know and look up the correct key in an object… sigh.)

[1]: http://use-the-index-luke.com/no-offset

[2]: https://tools.ietf.org/html/rfc5988

From the parent Microsoft guideline [1], they make the explicit differentiation between Server-driven and Client-driven paging, the Server-driven seems to be exactly what you describe in your first paragraph:

> Paginated responses MUST indicate a partial result by including a continuation token in the response. The absence of a continuation token means that no additional pages are available.

> Clients MUST treat the continuation URL as opaque, which means that query options may not be changed while iterating over a set of partial results.

> Example:

  {
    ...,
    "value": [...],
    "@nextLink": "{opaqueUrl}"
  }

They then explain Client-driven paging, if the client really wants to do it:

> Clients MAY use $top and $skip query parameters to specify a number of results to return and an offset. The server SHOULD honor the values specified by the client; however, clients MUST be prepared to handle responses that contain a different page size or contain a continuation token.

But if you want to obtain the entire collection, you would not use Client-driven paging, but only rely on Server-driven paging

[1] https://github.com/Microsoft/api-guidelines/blob/master/Guid...

Big +1 to killing offset in favor of order-by and an "after" or "before" criteria.

However, -1 to the idea of sticking any important "metadata" in headers. For one, it's dramatically easier to interact with response bodies than headers at the terminal or with simple programming libraries and tools. Having only one way of representing data requires fewer special cases. JSON is far superior to key/value pairs in terms of ability to represent structure. Lastly, if you ever decide to use a different "transport" protocol other than HTTP, you'll be glad you are already using a common representation without a side channel.

How do you feel about using StatusCodes to return api information.

/tacos/8 ... not found 404

vs { "meta":{ "errors":["not found"] } }

or something like that without getting into the form of the JSON returned.

The actual HTTP status should be relevant. I don't want to get HTTP/200 with {error: not found}. Instead, useful errors should be like:

HTTP/400 {error: invalid foobar in request. Foobar should be a string, but instead got '123'}

Or something like that. Tell developers what the problem actually was, and potentially how to fix it. http://jsonapi.org/format/#error-objects has some good ideas and suggests a standard.

That said, even discarding the body, you should be able to determine if a request was successful by the HTTP status code alone, IMO. If you made a new thing, 201, thing ain't there, 404, your request sucks, 400, etc. Don't make people using your API parse your [especially true if you've invented your own format] JSON errors if they don't want to.

"Success With Info" is something that just causes problems.

The number of examples we found of "Success with Info" going horribly wrong was abundant. Turns out checking return codes is something people have failed to do pretty much since the invention of the return code...

Is that a failure of the people or failure of the spec?

I think in many ways it's just tradition. Old browsers did wonky stuff with non 200 codes and that just created inertia that made people never quite take the effort to follow the spec.

Failure of people and tools.

Like the sibling comment says, often tools designed to consume HTTP will throw a more severe class of error upon 4xx/5xx, which then changes the codepath significantly, making it more difficult for the programmer to treat 2xx and 4xx/5xx responses similarly for parsing out the headers and the response body.

Web browsers are also guilty of some of these behaviors. They would inject a 'user friendly error page' upon 4xx/5xx. Since they are graphical applications, they have also been used as rudimentary debug/sanity-check tools at the expense of curl.

Together, these forces combined to drive some APIs to respond with a 200 for any response. Often, faithfulness to spec or faithfulness to the intent of the design takes a backseat to making a solution work. The combination of these traditions survives in the myriad APIs that essentially use HTTP as an browser-accessible RPC mechanism.

Which is irritating as all hell at times... the .Net http client does this, which makes snagging the actual response text and code difficult... Same wrt .Net for response codes, there's a lot of errors that can happen with the JSON deserializer for services, but it returns a 500 series error not a 400 series. I usually wrap the on error in the app file, and replace those responses for .Net projects.

Worth noting, not sure if this is improved/fixed in .Net Core based services.

As I mentioned in another thread, I'm in favor of the response always being an object at the top level, with a data property for any response data, and an error property for any errors, errors having a "message" property that is the error message string, and a "code" property that matches the numeric http response code.

This also helps if your error classes have these codes in them, that way the http response code can be set based on the error, defaulting to 500.

Together, these forces combined to drive some APIs to respond with a 200 for any response.

I once had to work on such an API, which was designed that way because of similar limitations in the (third-party?) iOS networking library used by the app that connected to the API (as it was in 2013, anyway).

One can see how thinking of an HTTP request as an RPC rather than a REST request/response would justify throwing an error instead of returning an object, but since REST is now more common than RPC, one would at least expect an option not to throw.

I tend to prefer a simple JSON object that wraps any response...

    { data: any, error: object }

So that only the presence of an error needs to be checked... within the error should always be a friendly message property and optionally a numeric code, that mirrors the response code (404), etc.

    { error: { code:404, message: 'Not found' } }

This tends to work well with error traps in server-side workflows... As to the data/result etc, it can be whatever is needed as a response to the client. For paged results, data/nextToken or something similar are perfectly serviceable imho.

To me an error should be top-level in the response object, as well as meta, and the data itself should be a consistent property and nested as such. Other platforms add their own wrappers which include the headers with the response object. ymmv.

in C# terms...

    //use CamelCasePropertyNamesContractResolver

    public class ApiResponse<T> {
      public T Data { get; set; }
      public Exception Error { get; set; }

      public ApiResponse(T data) {
        this.Data = data;
      }
   
      public ApiResponse(Exception error) {
        this.Error = error;
      }
    }

In this way, all API requests return ApiResponse<TypeOfData>

I agree with tracker1 about a top-level object with data+error keys. Always useful to have an "open" / "extensible" object at the top, so you can add extra stuff as needed. Pagination is a classic example, but another one is debug data, such as profiler timings.

I'm not opposed to having meaningful status codes, but I am opposed to 1) overdoing it and 2) omitting success/failure from the body. The only useful status code distinctions are the ones that browsers respect: success vs failure, redirects, etc. Differentiating 200 OK vs 201 Created, for example, causes nothing but headaches. What would I do differently given that distinct information? What if I do a POST to create an get a 200 when I normally expect a 201? Is that suddenly now a failure?

For API use cases, I only use 200 (success), 400 ("you screwed up"), and 500 ("we screwed up"). For HTML responses, I also use whatever 3xx the SEO people suggest these days.

That's exactly the way to do it. Returning the HTTP status code in the body of the message is just wrong.

Returning just an array from a json-endpoint that may contain sensitive data is a vulnerability: http://haacked.com/archive/2008/11/20/anatomy-of-a-subtle-js...

(edit: may be mitigated in newer browsers)

The obvious mitigation is don't visit untrusted endpoints from your API client, and/or don't run a script interpreter inside your API client.

That's not what the vulnerability is here. If you have a server endpoint that returns an array without wrapping it as a property on an object then an attacker can write a webpage that overrides the Array prototype and requests the json from your page as a script, thereby bypassing the cross origin check it would be subject to as an xhr request. So if you have a page which returns sensitive information as an array you (were, it seems browsers probably don't allow you to override the array prototype anymore) are vulnerable to the attacker stealing a signed in users secrets.

On the contrary, that's precisely the vulnerability:

1. The API server needs authentication, which gets cached in the user-agent.

2. The same user-agent, with the same authentication context, is then directed to a malicious site.

3. The user-agent contains a javascript interpreter, and the malicious website serves a script to override the Array prototype.

4. The same script then executes a CSRF request to the API server, reusing the cached authentication context, and therefore stealing the response.

A browser fits all the necessary conditions that must exist for this vulnerability to be exploited, but if you're the consumer of the API, you don't have to use a browser as your user-agent. You can write your own, which you'd do if you're making these requests programmatically in a non-web application.

Regardless as to which user-agent you use, a viable mitigation is to not visit untrusted endpoints with the same cookie jar or authentication context.

Another viable mitigation is to omit including a script interpreter in your custom API client. This is in fact usually the case.

Sorry, you are right.

There is no maybe for newer browsers. The site linked says it is safe for Firefox >2 , IE ≥ 6 and all of chrome.

Making the next page opaque as you said is a good thing also because limit and offset will often become unstable or incorrect if the dataset is changing without a proper timestamp. That is you may get more or less records then you are expecting which maybe important if you need to allocate size.

Almost all of my rest services I require sending a timestamp for batch processing (that is only records created/updated/removed before given timestamp are shown).

> while this removes the ability to of the client to go to an arbitrary page on its own

Yeah, that's the whole problem with this approach.

Sometimes, you can look at what the index is and find a better thing than page numbers to use; for example, if your collection is ordered/indexed on a timestamp, you can navigate to dates&times instead of page numbers. (e.g., in a UI presentation you might expose the ability to choose a date to go to, instead of a page)

Reply on news.ycombinator.com