Go, REST APIs, and Pointers | Hacker News Reader

Go, REST APIs, and Pointers

willnorris.com

117 pointswatermel0n20 comments

Here is a practical everyday example where languages that can represent emptiness in the type system come in handy.

He ends up using pointers and mentions: "Using pointers also means that clients of the library will need to perform their own nil checks where appropriate to prevent panics."

Looking forward to using Rust :)

> Using pointers also means that clients of the library will need to perform their own nil checks where appropriate to prevent panics.

In practice, as your Rust comment alludes, I find working with nil properties in Go types (used to shuffle between the DB and JSON) to be quite painful. I end up keeping track of a lot more state than I really want to, and errors can be introduced subtly.

In Go, you can usually use a type's zero value without doing anything special. But if that type contains any nil properties, you can't presume this anymore. If I have:

    type Thing struct {
        ID int
        Bad *string
    }

I then have to do nil checks in every method that might handle Thing, or I have to make it accessible only via functions which perform those checks for me in advance. Either way, much less convenient.

PATCH is a nightmare in reality, especially with the default zero nature of Go.

We opted for the JSON PATCH notation in RFC 6902: http://tools.ietf.org/html/rfc6902

Essentially there's a standard format for instructions on how to modify an existing JSON document, like this:

   PATCH /my/data HTTP/1.1
   Host: example.org
   Content-Length: 326
   Content-Type: application/json-patch+json
   If-Match: "abc123"

   [
     { "op": "test", "path": "/a/b/c", "value": "foo" },
     { "op": "remove", "path": "/a/b/c" },
     { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
     { "op": "replace", "path": "/a/b/c", "value": 42 },
     { "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
     { "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
   ]

We've implemented the operations "add", "remove" and "replace" in our REST API.

We don't have a meaningful way of doing "move" and "copy", and "test" can be done by performing a GET and looking at the document (and that can be used in the If-Match).

You can see the documentation for our stuff here: http://microcosm-cc.github.io/#events-single-patch

And that gives away why we chose to do this... booleans and the default value of false. We wanted it to be more explicit and no room for accidental expression of a value anywhere, regardless of the callee language/environment or ours. JSON PATCH makes this very explicit.

Of course there's the mild inconvenience of handling the value type, but that's relatively easily overcome.

What was really interesting was handling permissions for the PATCH instructions.

For example a user might have permission to issue a PATCH that changed a string, but only the admin could issue a PATCH that updated some special part of a resource.

PATCH is relatively easy with the above, and very predictable... fine-grained permissions of which part of a document someone can update... that's definitely where the fun is.

So you mean patch operations should be atomic right?

Atomic and not idempotent as per the HTTP spec.

Another approach is sql.NullString and related types from database/sql: http://golang.org/pkg/database/sql/#NullString

  foo := sql.NullString{"bar",true}
  if foo.Valid {
    fmt.Println(foo.String)
  }

Yes, but the blog post makes it pretty clear that easy JSON decoding/encoding is a primary goal, and this is not friendly to that.

   func (ns NullString) MarshalJSON() ([]byte, error) {
       if !ns.Valid {
           return []byte("null"), nil
       }
       return json.Marshal(ns.String)
   }

It would render `user_id: null` instead of `user_id: ""` which should work, I think.

sure it's valid JSON, but it really depends on the API as to whether it handles null values properly when it's expecting some other type. I honestly haven't tried that with the GitHub API, but based on what I've seen it probably handles it fine. Others may not.

And this is the problem about using null as a valid state for data. In many languages, using null as a valid data value just can cause all kinds of problem (extra checks, not supported, or problems accessing null data).

Seems like the issue is trying to keep type safety when doing a PATCH so instead of polluting the struct with pointers so it can be used to do a PATCH, why not create a separate type specific to that purpose:

  type Repository struct {
      Name        string `json:"name"`
      Description string `json:"description"`
      Private     bool   `json:"private"`
  }

  // response to blog post https://willnorris.com/2014/05/go-rest-apis-and-pointers

  type RepositoryPatch struct {
    Repository
    patched_fields []string
  }

  func (rp *RepositoryPatch) SetPrivate(private bool) {
    rp.patch_fields = append(rp.patch_fields,"private")
    rp.Private = private
  }

  func (rp *RepositoryPatch) MarshalJSON() ([]byte, error) {
    // marshal rp to []byte then unmarshal to map[string]interface{}
    // filter patched_fields and then marshal again
    // ;)
  }

yeah, that's similar to the field mask that was discussed on the issue (https://github.com/google/go-github/issues/19#issuecomment-2...), though your approach does provide some nice encapsulation. Given how large some of the GitHub data types are, you'd probably want to auto-generate the setter methods, similar to what goprotobuf does for accessor methods.

I'm designing a Go backend to interact with a backbone client. Our API receives form data and spits out JSON, so the situation is not exactly the same. But we still ran into a similar problem with the way revel does parameter binding (if fields were not included in the form data, they ended up being interpreted as the zero value).

Check out this solution we came up with in a testing/learning project: https://github.com/cranberryofdoom/peeps/blob/master/app/con.... Basically we just had to do an extra check to see if the fields actually existed in the form data. Will probably generalize it later on to make it less verbose.

This is actually a rather annoying problem. This is what I'm doing temporarily (all fields have the "omitempty" tag):

    Fetch struct A from DB
    Unmarshal JSON data into &A
    Unset read-only fields of A
    Save A to DB

It's very inefficient but it beats dealing with bugs caused by various flags not being set (or being unset).

I'll switch to using pointers to boolean, int and string since most other (non-trivial) fields are pointers anyway. I'm not happy about the indirection, but it's certainly much faster than the current database roundtrip I'm performing.

And for anyone shouting 'Rust' - there's always going to be that perfect language out there, at the end of the rainbow.

Interesting...point about using pointers to get nullable JSON values. Especially important for booleans, where false is the default value, and would not show up if "omitempty" is specified on that JSON field. Demo here: http://play.golang.org/p/K3sG6T2wDd

Mentioned this in the blog comments, but would anyone familiar with the Go internals know if a set/modified/etc flag could be added to the struct field and available via reflection?

(That would be awesome!)

Sure, it'd be possible, but it would also require changing the language. And that's not going to happen for something like this.

Isn't the problem here to absolutely want to manipulate a struct for the update ? What about a method like that ?

    func ( rep Repository ) Update( map[string]interface{} )

So why not just use a map (hash, dict, etc.) instead of an object? You're sending and receiving JSON data, that's maps in maps all the way down. Why this fetish towards mapping your JSON 1:1 to some inner program object?

Well in a real language your objects have guarantees associated with them; constructing them should imply things about their validity. You can draw a distinction between an unvalidated request to construct an object, a valid object, and an update that can apply to an existing object. And your type system lets you ensure that you don't mix them up, that all code paths pass through the assertions you want them to pass through, so you don't do redundant checking and don't worry about missing anything.

In go, shrug, hell if I know what anyone's doing with that language.

Reply on news.ycombinator.com