How Python Makes Working with Data More Difficult in the Long Run

Leaving aside the static/dynamic debate - there's no attempt here to suggest that the issues over readability and maintainability could be mitigated. He's just shown the simplest Python implementation, waved his hands and then said "Hey! Let's use Go".

I'm sure there's plenty of reasons to use Go but I'm not sure this article makes a terribly good case for it in this scenario.

How about using

While static typing surely helps, Go has its own issues working with polymorphic JSON, and don't get me started on working with XML. I have never liked struct tags. Why would a statically typed language relies on this fragile feature ? it's often a source of bugs since they are just strings that need to be parsed at runtime, with reflection, and frankly the std lib struct tag parser is extremely fragile. Go is full of these things. In order to make the compiler go fast, the developer has to do the compiler's job, at runtime. That's not right.

If some external data doesn't fit Go type system then Go makes it extremely hard to work with these data. Same thing with SQL, it's extremely tedious with Go when fields can be null.

If you want better type information for Python, you could use the new typing hints and the typing module(https://docs.python.org/3/library/typing.html).

You could also use mypy(https://mypy.readthedocs.io/en/latest/index.html) which is a static type checker for Python.

This was the subject of the talk that I gave with a cospeaker at RustConf. Effectively, the shortcomings of dynamic type systems when it comes to building data science oriented services.

You mention that it's much omre laborious to write a struct that matches your data exactly - however, I've found that, at this stage in the process you already have some sort of schema (mental or formal) of what your data is, and it's fairly trivial to write out a structured representation of it.

What this then buys you is a semantic understanding, in your code, of what 'bad data' is and how it's handled. You know that if you get a string where you expected an int, that's a problem. You also know that if data is missing, eg: null, that you'll handle it.

In Python, a function can erroneously return None. To your data, this is just a missing value, not a bug. But there is a semantic difference, and a cost associated with that bug.

All of these problems, limited strictly to working with data, are far easier to react to and deal with in a static language.

We absolutely conceded the exploratory phase to Python. It is, and will likely remain, an excellent and near-ideal tool for that phase.

> In Go, for example, to parse and return a JSON response from some web API, you first need to create a struct whose fields and field-types exactly match the structure of the response.

Right, but why can't these declarations be written by some Python code? (with a large set of examples of responses as input)

Can we please see a comparison with a statically-typed language better suited to data-science? Go is a long way away from the current state-of-the-art in statically typed languages. I would hope that F# with its type providers has a pretty good exploration story and possibly Scala too.

Reporting quite a similar experience as Daniel Whitenack spoke about, in his GopherCon talk about Go for Data Science:

https://www.youtube.com/watch?v=D5tDubyXLrQ

See also his answers to some follow-up questions:

http://www.datadan.io/common-go-for-data-science-questions/

... and his O'Reilly article on the topic:

https://www.oreilly.com/ideas/data-science-gophers

We would not have this conversation..if Excel supported big data :D :D :D

Use R