Numpy is absurd | Hacker News Reader

Numpy is absurd

gist.github.com

102 pointsmoelf50 comments

Here's a better resource for understanding how __conains__ works which actually looks at the source/documentation instead of trying to understand it intuitively: https://stackoverflow.com/questions/18320624/how-does-contai...

Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.

Well yeah you're right OP could have explained this better; but this argues their point. From the definition that `a in b` is `(a == b).any()` it's trivial to see things like `np.array([1, 0, 0]) in np.array([1, 2, 3])` will return `True` too, which is absurd, as the title says. I find it hard to believe when numpy team was designing the API they missed this.

It's trivial to see some things and other things use broadcasting and that's not quite trivial :)

My point is that even the trivial implications of the definition are absurd; without inspecting non-trivial broadcasting cases we can already determine the API is poorly designed.

Does == ignore dtype?

Yes. 1 == 1.0

Even that took me a while to understand it and, I guess, is another example of Numpy’s absurdity.

My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans.

But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence.

Reading https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.

I've worked with numpy for years and I don't think I ever used the 'in' operator with arrays. What would be the use case for that? For me a larger gotcha was around object multiplication (i.e. element-wise vs. matrix multiplication) and how it is different based on whether you have a "matrix" or an "array" type (or a mix of both). That caused some pretty big headache in the beginning.

But that just seems like a foot gun for the poor unfortunate soul that eventually does.

Meh. Footguns are underrated. Let’s return to the Wild West of the 90s.

I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.

All these talk about code safety just want me to stop wearing my bike helmet.

For most of my programs safety is of zero concern. I'll trade it for convenience any day. Just let me implicit cast signed to unsigned thank you very much.

I don't think people who use python are concerned with footguns. Numpy is very much not at fault here.

Why would you think that?

Because the typing system used in Python is a big footgun. If the basis of the language you are using encourages footguns then it's not surprising that a large library in that ecosystem doesn't care about that either.

iirc matrix is deprecated

Interpreting “in” as “is an element of”, all of these are type errors. It only makes sense to ask if an X is in an Array<X>. In a language that lets you ask nonsense questions, expect nonsense answers.

Numpy has the highly useful operation of `a < b` where a and b are arrays and the answer is an array of booleans - i.e broadcasted elementwise compare.

They changed the language to change what question `<` asks.

This gets ugly with !=. Instead of 'a != b' you usually want 'not (a == b).all()'.

It really doesn't if you think about it. a==b cannot by itself be used as an if condition. You would always need to use (a==b).all(). So it would be very difficult to make the mistake you are suggesting

https://stackoverflow.com/questions/10062954/valueerror-the-...

I think the general point is that it was a bad idea to implement the operators as elementwise because it breaks all of the properties they are supposed to hold.

Really there should be separate elementwise operators like `.>` and so on but you can only do that in some languages.

Sure, and languages like C++ let you do this kind of thing very broadly, but it has a lot of pitfalls. One of my biggest gripes with python is that function interfaces and "type signatures" are hard to discover even when documentation is good.

FWIW I think it's very hard to discover this from C++ code as well. It has to be documented.

> I always gripe about Python not having built-in array type

  from array import array

it’s right there.

> Numpy doesn't distinguish "vector of vector" from "matrix"

  numpy.matrix

though to be fair, it’s on its way out.

when built-in array type is not performant nor serves as an interface (abstract class), it's as if it doesn't exist. I guess the point is Numpy predates Python's array by a long shot.

Compare and contrast with languages with built-in N-dimensional array for scientific computing.

like matlab, that lies?

Julia

What do you mean?

in matlab:

  >> size(4)
  ans =
       1     1

which is 1. wrong and 2. evidence that matlab cannot represent numerical objects of less than rank 2.

Numpy can:

  >> np.shape(4)
  ()

I'm confused by what you mean. In Matlab, everything is a matrix. A naked integer is a 1x1 matrix, hence why the size is reported as [1 1]. Those are the semantics of the language, not a lie.

a naked integer is not a 1x1 matrix, it’s a 0x0. Matlab silently promotes it to one, though, which is a deliberately misleading thing to do because it’s convenient for matlab’s data model.

i think it’s fair to call that a (white) lie

> a naked integer is not a 1x1 matrix, it’s a 0x0

Do you have a link to where this is asserted in the documentation? I'm confused as to the basis for this.

https://mathworld.wolfram.com/TensorRank.html

it should be clear that i don’t care very much about what matlab has to represent things as. i don’t mean “matlab integers”, i mean the mathematical objects; matlab’s representation of them is poor.

Okay, well that's not Matlab lying, that's you not understanding how the tool works. It would be like saying that 0.1 + 0.2 == 0.3 evaluating to false is a lie, because floating point numbers don't accurately represent real numbers as a concept. But floating point numbers aren't actually real numbers, and integers in Matlab (they're actually stored as floating point doubles, is that a lie too?) aren't actually mathematical objects.

You have to work within the constraints of the tool. Matlab makes it very clear upfront that the semantics of the language mean that naked numbers are in fact matrices, because everything in the language is a matrix. Maybe you feel that's not a good representation, but it's not a lie, and in fact it's very consistent and useful in the context of the language itself.

> Numpy predates Python's array by a long shot

I don't think so. The array module was included even in Python 1.0.

Or did you mean to use another verb?

To be more specific, "git log arraymodule.c" reports:

  commit 778983b48165da25ee11fb97f6855af7c67f4ff2
  Author: Guido van Rossum <guido@python.org>
  Date:   Fri Feb 19 15:55:02 1993 +0000
  
      Added new module "array" (for now optional) defining array objects.

Op here, I'd also like to say I blame this on Python's lack of broadcasting syntax, (dot in Matlab and Julia)

Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container

Julia's broadcasting syntax depends on having Julia-style compilation to be efficient. It's not clear to me Python could ever provide the same behavior.

    import numpy as np
    print(np.__version__)
    a = np.array([1,2,3])
    b = np.array([1])

    print(a in b)
    print(b in a)
    print(np.isin(a, b))
    print(np.isin(b, a))
    print(np.isin(a.astype(float), b))
    print(np.isin(a, b.astype(float))))

    1.21.6
    True
    True
    [ True False False]
    [ True]
    [ True False False]
    [ True False False]

Core numpy is really weird, and has really sloppy logic around higher-dimensional objects. I've a module to work around lots of the warts. If somebody has smarter versions of intersect() or in() or whatever, contribute!

https://github.com/dkogan/numpysane

Apologies if this was explained in the talk, but why not upstream your changes to numpy?

No real good reason, honestly. I wasn't feeling positively about numpy devs at the moment, and didn't want to negotiate getting the code in. The library is solid though, and should be merged. I need somebody to pester me about it, to actualy work on that today.

For a language with "explicit is better than implicit" as part of its mantra[1], Python is chock full of surprising behavior like this where the programmer really has no idea what's happening under the hood.

I'm happy I moved on to Go, which actually adopts this as a core design tenet.

[1]: Even though PEP-20 is largely a meme.

To be fair this person's gripes are with the behavior of Numpy arrays, not anything that's native to Python's core language. Numpy arrays are tricky beasts because they try to balance fast operations on huge amounts of data, with expressive Python operator syntax.

Although, another way of looking at what you just said is that Numpy arrays exist because core Python fails to strike a balance between being both fast and expressive.

That does not necessarily follow. Python may be striking a balance between the needs of numerical and non-numerical computing, while NumPy arrays do not need to consider non-numerical computing.

Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing:

  >>> import numpy
  >>> numpy.polynomial.hermite.hermfromroots([2,2,3,4])
  array([ 7.075e+01, -4.625e+01,  1.175e+01, -1.375e+00,  6.250e-02])

This has a high-startup cost which makes NumPy use inappropriate for short-lived programs like shell scripts, even in cases where Python's own startup cost is otherwise appropriate.

Who needs this strange operation anyways? Deprecate the whole method and call it a day

that's what I'd go for, throw more errors when `__contains__` is called with ambiguity.

but, cough, mature and stable scientific computing foundation?

> I always gripe about Python not having built-in array type

Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html

Oh no! Undefined behaviour! What ever shall we do?

Finally! It had to be said. Im a huge Python fan and a huge Pandas fan and have written hundreds of scripts. But numpy is so "R" esque and technical and stiff and unintuitive for sure.

Reply on news.ycombinator.com