Hacker News Reader

Note on encoding/json: you only need to mention the fields you care about. If there are extra fields, the parser will just scan past them.

    >  I was decoding a large JSON data structure that could 
    > easily have thousands of different fields nested 10+ 
    > levels deep. Go's standard json library usually 
    > recommends that you define a struct type to hold the 
    > result of the decode, tagging field names with the 
    > `json` annotation. This obviously doesn't work when you 
    > only care about a few dozen of the several thousand 
    > possible fields.

Sure it does!

    type Response struct {
        LevelOne map[string]struct{
            LevelTwo map[int]struct{
                Value string `json:"eventual_value"`
            } `json:"level_two"`
        } `json:"level_one"`
    }

The response can have way more fields than just the ones you declare in your struct.

Compared to the Python version of:

  response['level_one']['level_two']['eventual_value']

?

I did ignore fields that I wasn't interested in; however, you do at least have to declare all the intermediate structs in the tree. With 10-ish levels of nesting, that's a lot of boilerplate; even in your example with 3 levels, you've still got 6 lines of code for what's a one-liner in Python.

It wouldn't be as much of a problem for a production system where you define your data structures once and then expect to amortize them over many, many changes. But I was prototyping, and the point of prototyping is to not spend time on boilerplate that isn't necessary to prove out your concept. A few hundred lines of type declarations just so I could decode JSON isn't exactly convenient.

Lines of code in type declarations isn't an interesting metric. Type safety, however, is.

Said otherwise, the python oneliner is likely not what you want, in real life you'd have to check for fields existence, purpose defaults or warn, etc. So you must likely end up with a few kings of code anyways.

Context of this thread is prototyping. For prototyping that one-liner is exactly what you want; if the fields don't exist just dump a stack trace and fix your code to use stuff that does exist.

I think the solution posed to you is at least much better than what your comment thread started with: defining the entire JSON structure with Go types. Actually, I think it's a lot better. Personally, I'd probably stop there---even if I were prototyping.

But, at your insistence, we can keep digging. It's easy to write a polymorphic function that arbitrarily picks out values from a Go `interface{}` value.

    func lookup(v interface{}, keys ...string) interface{} {
    	switch len(keys) {
    	case 0: return v
    	default:
    		dict, ok := v.(map[string]interface{})
    		if !ok {
    			panic(fmt.Sprintf("%T is not a dictionary", v))
    		}
    		return lookup(dict[keys[0]], keys[1:]...)
    	}
    }

And then we can use it like you would in Python:

    lookup(response, "level_one", "level_two", "eventual_value")

This doesn't completely remove the burden of type asserting (unfortunate), but it at least makes nested lookups much easier. Arguably, the nested lookups are probably where the type asserts hurt the most.

Full example: http://play.golang.org/p/4tWtqGRgU6

The use-case for dynamic typing was that I was decoding a large JSON data structure that could easily have thousands of different fields nested 10+ levels deep.

Yeah, prototyping with JSON is one place where dynamic languages are more nimble.

For example, the Python bindings to Amazon S3 let you access files through dictionary notation. BeautifulSoup lets you iterate over child nodes with a for loop, or access attributes as a dict. NumPy lets you use standard arithmetic operators on matrices. Judicious use of this in libraries makes the resulting code much briefer.

Yes, this is another specific area where dynamic languages do prototyping better and can produce shorter code. I did an exercise where someone implemented a routine in Clojure that I had written in Python. The Python was shorter! In part, this was due to list comprehensions. In part, it was due to an excellent library. What you describe is what many Smalltalkers did as well: create your own control structures and DSLs.

I've sometimes wondered what could be done with a dynamic superset language of Go, such that one could mostly add type definitions and compile to Go.

My biggest pain when programming servers in Go is when handling others' dynamic JSON. Multiple levels of switch-case to handle different types for the same JSON fields at different times. With the current eco systems, Objective C seems the best language, dynamic typing with optional static typing, or mostly static typing in the programs, with dynamic typing for cases like handling dynamic data. Other goods include ARC, Xcode, Apple's frameworks, etc. Concurrency seems to work well with dispatch queues etc. I wonder if Objective C could be as good as Go for servers.

Apple uses Java on its server-side.

My biggest pain when programming servers in Go is when handling others' dynamic JSON.

It should be simple enough to implement a dynamic JSON parsing DSL in Go. It would be slower and memory inefficient, but much more convenient for rapid development.