Hacker News Reader

Comment by hereforphone | Hacker News Reader

Pandas is marvelous. As is numpy, scipy, and all of that family. And they run essentially as C libraries, and approximate that level of performance. Someday I hope there will be golang analogues of these

View on HN

Pandas is great.

One funny thing, it's only in the Python world that C is a defined speed. When we implement libraries in actual C, C++, Rust etc, we benchmark and compare different libraries, they can have wildly different performance - depending on you know, work put into them. But Python is just a two-speed world: Python or C ;-)

As someone new to python trying to use these, I find them often too magic. Soo much overloading, magic, parameters etc, with virtually no help from a type system.

Like this standard usage which misuses all the operators:

  a = b[c == 1]

One would think that would be equivalent to b[true] or false, and what's that supoosed to mean? But if c is a numpy array it's instead doing element wise comparison on c, and the new array of booleans will be used to index into b. If you don't know this, it's impossible to figure out looking at the code. Too many footguns imo. Same with pytorch, broadcasting is nice, but makes it hard to figure out what's going to happen in practice with the shapes being calculated upon.

I think that this is something that you can get accustomed to pretty quickly. Vectorised operations and broadcasting are very common across languages too, it's not just the python data stack that uses them.

A more weird but still basic example from pandas is the way it handles indexing and the confusion between .loc, .iloc, .ix, .at and plain '[]'. That part of the API should be really basic but for some reason it's a giant mess.

Your complaint applies to array processing libraries in general. It applies to numpy. Yet those syntactic niceties are exactly what make them so convenient. Sure, it takes time to get comfortable writing array-oriented code. But it isn’t a good idea to optimize for complete beginners. Some familiarity with the general domain is a prerequisite to using array (and dataframe) libraries.

But instead of having X ways of using the [] operator, it would be almost just as convenient if that were actual named functions. Then the editor could understand whats going on, one could jump to definitions, look up stuff, more easily actually learn that it's possible to do etc.

I feel Notebooks aren't used as much because they're a great tool, but that they are used so much as a necessity since you need to look at live data to have any hopes of understanding what's going on. Why couldn't a compiler instead tell me what I'm trying to do will fail?

I agree with you in spirit. Having separate names is preferable to context-specific semantics for the []. In pandas, using [] alone is discouraged, in favor of using .loc or .iloc. But admittedly, that only partially improves the situation. Both accept multiple types (int vs int-list vs bool-list).

Last year, I had an in-person discussion with a numpy core developer in which I proposed adding pandas-like syntax to numpy, e.g. allowing the author to use:

a.boolmask[b == c]

(but maybe not so verbose).

A problem immediately arises: An array can have multiple axes, and each could be indexed with a different type within the same slicing call.

One idea might be to allow the author to explicitly wrap the arguments with some dummy class to make clear what types they’re using:

a[np.boolmask(b == c), d, np.intlist(e)]

(Again, perhaps choosing shorter names in practice.)

The idea would be that the wrapper merely “looks like” the thing it wraps, but contains an assertion to verify the type of the argument.

Anyway, I agree that notebooks (or REPLs in general) are great for verifying your assumptions about how little snippets will behave. But I think they would be useful even in a fully statically-typed compiled language, too.

It's just like learning any unfamiliar programming approach. Array processing has a long history, and it's supported by languages like R which are heavily used in data science. Once you understand the different uses of [] operator, it becomes second nature, and you realize how useful vectorization and broadcasting can be. Also, manipulating live data is an excellent way to learn about your data and figure out what you need to do. Python is a dynamic language, so interacting with a rich REPL environment makes a lot of sense.

This seems to come up every single time these libraries are brought up around here. People who are primarily programmers don't seem to understand these libraries weren't written for them. Wes McKinney was working at a hedge fund when he wrote pandas. The history of Numpy dates back to researchers at MIT and LLNL petitioning Guido to extend the Python syntax to facilitate array programming, because it is supported in programming languages primarily used by scientists and engineers like MATLAB, R, S-Plus. Broadcasting in these languages is fairly intuitive and well understood by the users, and not considered surprising, because it matches pretty closely the way you symbolically manipulate vectors, matrices, and tensors on paper.

The array syntax is not what you expect if you're coming from a background primarily in programming applications using other dynamically-typed languages or even Python itself, but it's exactly what you'd expect if you're a scientist used to using MATLAB, or even just used to doing math on paper. And at least back in the early days, that was how hedge funds and quantitative finance worked. They weren't looking for application programmers. They were looking for physicists and electrical engineers. It was a lot easier to teach them how to program than to teach programmers about Brownian motion and stochastic processes.

For their intended use cases, Numpy and pandas are among the greatest tools ever created, honestly comparable to MATLAB itself except also free.

If you're expecting a numerical library's operations/semantics to be intuitive to a reader not familiar with the library, you'll be disappointed by most such libraries.

What you've shown here is pandas 101. If it is confusing, it's a sign you need to understand the underlying basics of NumPy and pandas.

Once generics ship, there could be progress in this direction. Until then, though, very high level data structure type libraries, along the lines of pandas, are always going to be terrible in Go.

Reply on news.ycombinator.com