Weird Python Integers
kate.io
kate.io
Plugging my near useless Python library that does this and a lot of other subtle, annoying things to break programs. The library is essentially a display of how much Python actually exposes to the user and how modifiable it is.
>>> from exterminate import AltPrint
>>> print("Hey! How are you?")
Yo dawwwwg! Wuz crackalackin' yo?
AltPrint calls Gizoogle.translate():
https://github.com/adtac/exterminate/blob/f18862ed5b2e143e18...
And Gizoogle.translate() performs an HTTP call:
https://github.com/adtac/exterminate/blob/f18862ed5b2e143e18...
Apparently it is too hard to move python proper over to that method because of backwards compatibility issues with C extensions.
For example, the first output of the following program is 4, but the second is still 7.
program main
integer m,n
m = 7
call corrupt(m)
write (*,*) m
n = 7
write (*,*) n
stop
end program main
subroutine corrupt (a)
integer a
a = 4
return
end call corrupt(7)
program main
integer m,n
call corrupt(7)
write (*,*), 7
stop
end program main
subroutine corrupt (a)
integer a
a = 4
return
end
The reason was that in one subroutine, a parameter value was stored in a local variable, then used for computation and restored at the end. Since constants was stored in read only memory when using g77 on Linux, but not on the f77 compiler on HPUX, the Linux port would crash, but not the original HPUX version. In that the code you had above would have worked.
I was going to mention Indiana but was more hoping that it wouldn't be mentioned at all.
It is just a bad idea to implement the '==' operator using comparison on addresses instead of on values and I consider languages pretty low-level that do that.
In a sane language == is always a function adhering the attributes of an equivalence relation (reflexivity, symmetry and transivity), where a user can expect an actual value comparison - AFAIK one is advised to overwrite `equals` in e.g. Java, but I do not think this is good design.
//edit:
There is no obvious answer, as one could argue, that languages just use different names for the same concepts ('==' vs 'equals') and consider comparison by value or by reference more fundamental (and choose '==' for that).
I think Python is a good example here, where referential comparison is the `is` operator: `a is b`.
new Long(5) == new Long(5);?
If I have a graph data structure then `node == node` with identity does make sense to me in lots of cases.
And what's the value equality of `(new Object()).equals(new Object())` - they have no value.
It's not that reference semantics don't have a place, it's just that the place isn't numbers.
As for your second question, it looks like the type system has two embarrassing questions: what the heck does "new Object()" mean? Nothing worth saying.
Integer a = new Integer(2);
Integer b = a;
Integer c = new Integer(2);
a == b is an value-comparison of the identity "value" (in Java it's the memory reference).
Or in other words, how do I know two references are equal (identity equal) without comparing the values of the references?
Logical equality devolves to the same thing at the end.
Certainly, one can come up with examples of how a different operator == would be useful for certain types of objects (including Integer), but the architects of Java strongly felt that an operator should always behave the same, regardless of the type of its operands, unless the operator is + and the objects are Strings.
Um, nonsense?
class Intbox {
public static void main(String[] args) {
Integer a = new Integer(2);
Integer b = a;
System.out.println(a == b);
System.out.println(a.equals(b));
}
}
$ java Intbox
true
true > cat intbox.java
class Intbox {
public static void main(String[] args) {
Integer a = new Integer(2);
Integer b = a;
Integer c = new Integer(2);
System.out.printf("a == b => %b\n", a == b);
System.out.printf("a.equals(b) => %b\n", a.equals(b));
System.out.printf("a == c => %b\n", a == c);
System.out.printf("a.equals(c) => %b\n", a.equals(c));
}
}
> javac intbox.java
> java Intbox
a == b => true
a.equals(b) => true
a == c => false
a.equals(c) => true
That's separate from the decision Java made that == was for identity comparison. Some people disagree with that decision.
There is no argument to be made that "comparison by value is just as fundamental a concept as comparison by identity", or that this is just a matter of opinion.
The fact that it is possible to compare equal by value without comparing equal by identity does not contradict that it is impossible to compare equal by identity without also comparing equal by value, however you define "equal by value".
> Your point regarding that "there is no way to compare equal by identity without also comparing equal by value" is true but doesn't really say much. (I'm not sure if it isn't a tautology?)
It's not a tautology; it is one of three requirements defining an equivalence relation. (Equivalence relations are by definition reflexive, which is to say x is equivalent to x for all x.)
I agree that in some sense you can say that "identity comparison is more fundamental that value comparison". As I added in my first comment, I think this is likely a tautology, and doesn't do much work for you at the end of the day. The important thing is keeping the two straight and doing the appropriate comparison depending on the context.
[1] And as I've already mentioned, you can see that this is true by looking at the definition of an equivalence relation. They all treat everything as equivalent to itself.
Two NaNs are equal in identity but not in value, according to the IEEE spec.
A NaN value does not compare equal to itself, because NaN is not thought of as a value at all but rather an error signal ("Not a Number"). You can check whether something is NaN, but that's not a distinction between comparing by identity and comparing by value -- it's a distinction between comparing values and checking for the presence of errors. The analogous operations to (1) comparing NaN to itself (unequal) and (2) checking whether NaN is NaN (yes), for another value such as -0, are (1b) comparing -0 to itself (equal) and (2b) checking whether -0 is NaN (no). They aren't (1c) comparing -0 to itself and (2c) checking whether -0 has the same "identity" as -0.
`a==b` is always an analog for the C code `a==b`, not `* a==*b` and certainly not C++'s `operator==(a, b)`
> The current implementation keeps an array of integer objects for all integers between -5 and 256, when you create an int in that range you actually just get back a reference to the existing object. So it should be possible to change the value of 1. I suspect the behaviour of Python in this case is undefined. :-)
does anyone have any idea how they chose that range? it's a 262-wide block starting at -5, which seems incredibly arbitrary.
* Initial commit with -1...99 cached:
https://github.com/python/cpython/commit/842d2ccdcd540399501...
* Cache more negative numbers, -5...99:
https://github.com/python/cpython/commit/c91ed400e053dc9f11d...
https://bugs.python.org/issue561244
* Cache more positive numbers, -5...256:
https://github.com/python/cpython/commit/418a1ef0895e826c65d...
it looks like they searched the standard library for negative integer literals and that's how they settled on -5 for the low end.
the high end came after introducing the `bytes` object. it went up to 256 instead of just 255 for size/len checks.
> [...]
> It looks like there is a table of tiny integers and each integer is takes up 32 bytes.
It is the memory address but it's a "CPython implementation detail: This [return value of the id() function] is the address of the object in memory."[1]
Though you cannot use this to determine the size of an object, or rather you "shouldn't" because that assumes a very specific implementation detail, which isn't there.
If you'd like to get the size of an object, use sys.getsizeof().[2] Also keep in mind that containers in Python does not contain the objects themselves but references to them so the returned size is the size of the object itself only, non-recursively. Read "Is Python call-by-value or call-by-reference? Neither."[3] for some more details.
[1]: https://docs.python.org/3.6/library/functions.html#id
[2]: http://docs.python.org/3.6/library/sys.html#sys.getsizeof
[3]: https://jeffknupp.com/blog/2012/11/13/is-python-callbyvalue-...
BTW, I didn't like the way the difference between mutable and immutable objects was pointed out, it sounded like the language recognized and supported those two types, when in fact the difference is just in the methods exposed by the objects. Even immutable objects can be modified if you don't respect the contracts (like in the case of this article).
— Robert Anton Wilson, The Historical Illuminatus Chronicles, as spoken by Sigismundo Celine.
https://en.wikipedia.org/wiki/E-Prime
Kellogg and Bourland describe misuse of the verb to be as creating a "deity mode of speech", allowing "even the most ignorant to transform their opinions magically into god-like pronouncements on the nature of things".
Bourland and other advocates also suggest that use of E-Prime leads to a less dogmatic style of language that reduces the possibility of misunderstanding or conflict.
Alfred Korzybski justified the expression he coined — "the map is not the territory" — by saying that "the denial of identification (as in 'is not') has opposite neuro-linguistic effects on the brain from the assertion of identity (as in 'is')."
I suspect that with dependent types, nominal types are actually a degenerate type of dependent structural type: a dependent pair of a type and a proof that a string 'name' field has some particular value, or that a list<string> 'implements' field contains some particular string (the interface name).
Python is fine with duck typing.
a = 2**62 - 1
b = 2**62 - 1
a.object_id == b.object_id # true
a = 2**62
b = 2**62
a.object_id == b.object_id # false
Perhaps one difference from Python is that in MRI Ruby Fixnum doesn't really even allocate an 'object', the object_id is the value in disguise. In fact all 'real' objects have even object_ids and all odd object_ids are integers:
a = 123456789
(a.object_id - 1) / 2 # 123456789It caches small integers, but also literals used in the same interpreter context (I'm probably getting that last term wrong). You'll get different results if you run these in from the shell as opposed to executing a script, try it out!
Here's a fun example
>>> x = 256; x is 256
True
>>> x = 257; x is 257
True
>>> x = 257
>>> x is 257
False
>>> def wat():
... x = 500
... return x is 500
>>> wat()
True >>> def f():
... x = 2222; y=2222
... return x is y
...
>>> f()
True
>>> f.func_code.co_consts
(None, 2222)
I believe the REPL compiles each input in a new toplevel module context, so each input gets its own constant pool.
Functions get their own constant pools, which explains the following behavior:
>>> if True:
... def f():return 2222
... def g():return 2222
...
>>> f() is g()
False >>> def f():
... x = 2221+1; y=2221+1
... return x is y
>>> f()
False
>>> f.__code__.co_consts
(None, 2221, 1, 2222, 2222)
>>> dis.dis(f)
2 0 LOAD_CONST 3 (2222)
2 STORE_FAST 0 (x)
4 LOAD_CONST 4 (2222)
6 STORE_FAST 1 (y)
3 8 LOAD_FAST 0 (x)
10 LOAD_FAST 1 (y)
12 COMPARE_OP 8 (is)
14 RETURN_VALUE
>>> 100 is 100
True
>>> (10 ** 2) is (10 ** 2)
True
>>> (10 ** 3) is (10 ** 3)
False
>>> 1000 is 1000
True
The tricky bit is that everything is an object in Python, even "primitives" like integers, which can be surprising coming from some other languages. Then the trickier bit is that CPython (the most popular and standard implementation) on startup creates an object for every int from -5 to 256 and will re-use those instead of creating a new PyLongObject. This means that `256 is 256` works, because under the hood, it's short-circuting it to the same object. But `257 is 257` creates 2 different PyLongObjects, which have the same value, 257, but aren't technically the same object.
And in my above example, the reason `x = 257; x is 257` is True when on the same line, but False when on separate lines (in the REPL) is because of the reasons kmill says, the constants pool that the interpreter creates. If they're interpreted together, it will create and use the same constant.
Here is my entry on my company's blog about this. I won't guarantee you it's well written, but it does show bytecode and has links to the relevant C source lines at the very end.
https://www.everymundo.com/literals-other-number-oddities-py...
MOST-NEGATIVE-FIXNUM, value: -1152921504606846976
MOST-POSITIVE-FIXNUM, value: 1152921504606846975
In [7]: a = "foo"
In [8]: b = "foo"
In [9]: a is b
Out[9]: True
In [10]: b = "foobaljlajdfsklfjds l;kjsl;dfj ls;dfj l;skdj flsdjluejsklnm "
In [11]: a = "foobaljlajdfsklfjds l;kjsl;dfj ls;dfj l;skdj flsdjluejsklnm "
In [12]: a is b
Out[12]: False
https://hg.python.org/cpython/file/f254ceec0d45/Objects/code...
Its not actually length, its whether the string looks like it could be a python name, and it only happens for string literals.
http://www.ianbicking.org/blog/2008/12/the-magic-sentinel.ht...
But I'd estimate 99% of the times I've used `is` was for comparisons to `None`, and 90% of the rest of times for comparisons to other sentinel values.
Idiomatically, you generally only see `is` used for comparison to `None`. Sometimes developers want to check identity, and you'd use `is` there too (but I don't have a handy example offhand).
No, it is not good practice to rely on the interning of ints and short strings. It's implementation-specific behavior and relying on it can cause your code to not work on PyPy or even potentially a future version of CPython.
http://docs.oracle.com/javase/specs/jls/se8/html/jls-5.html#...
(here == means reference not value equality)
"If the value p being boxed is an integer literal of type int between -128 and 127 inclusive (§3.10.1), or the boolean literal true or false (§3.10.3), or a character literal between '\u0000' and '\u007f' inclusive (§3.10.4), then let a and b be the results of any two boxing conversions of p. It is always the case that a == b."
If you're going to have a conditional that will evaluate to True most of the time, sometimes (in Python) it's considered better to just go ahead and try the logic that the conditional is guarding, and then catch the exception and handle it if it happens.
Unfortunately this blog post seems to miss a great opportunity to show you how you should compare integers for equality -- using the equality operator `==` and not the identity comparison `is`.
EDIT: odd, this post attracted a lot of downvotes. Please help me learn how this post could be improved.
Then you look at something like C#, where == compares values for value types and references for reference types, except that == is overloaded for some "value-like" reference types like String... it's a mess.
OTOH, Java is consistent in a sense that == never dereferences, but awkward in practice because of the need to use equals() for strings, which is too verbose for an operation that's far more common than object identity comparison.
It's still confusing. Plenty of "reference types" behave like values in all obvious senses. Why should "foo" be a reference but 45L a value?
I fail to see the confusion, other than for newbies.
a) is not even true, see floating point numbers
and
b) will be obsolete with value types.
Needless to say, there is pointless confusion created by Java's design, and there are better approaches available.
All you have to do is to adapt the semantic model from reference equality vs. value equality to identity vs. equality.
Identity checks whether the "bits" are identical, irregardless of whether the bits are "references" or values, and equality is a user-defined operation.
"Foo" equality "Foo" // True
"Foo" identity "Foo" // Only true if they point to the same "object"
123 equality 123 // True
123 identity 123 // True
Double.NaN equality Double.NaN // False
Double.NaN identity Double.NaN // True
1.
There is a ton of code out there which does something like
def contains(that: Thing): Boolean =
this.value identity that || this.value equality that
2.
The idea to define a _sane_ definition of identity/equality across all types is there to avoid the "next-best" option: boxing primitives to wrapper classes which is both slow and has terrible semantics.
3.
I don't really think restricting identity to e.g. reference types makes sense given that equality is defined for every type. Either none of them should be available by default, or both should be.
There _are_ multiple valid ways to compare to things (consider floating point numbers for a second) and making one more privileged than the other feels wrong.
Maybe collections of values should be different from collections of references. The sensible use cases for the two are quite different.
> Additionally, you would run into issues with floating point numbers which would not be found/retrieved anymore if identity were removed.
Meh, just allow NaN to compare equal to itself. Equality is supposed to be reflexive.
> The idea to define a _sane_ definition of identity/equality across all types is there to avoid the "next-best" option: boxing primitives to wrapper classes which is both slow and has terrible semantics.
Unboxed primitives don't have identity, only value equality. They align well with what's being proposed.
> I don't really think restricting identity to e.g. reference types makes sense given that equality is defined for every type. Either none of them should be available by default, or both should be.
We could build the distinction into the language, so for every type you define you explicitly choose whether it's value or reference. Scala's already halfway there with the class/case class distinction.
> There _are_ multiple valid ways to compare to things (consider floating point numbers for a second)
Disagree; comparison is so fundamental to most types that it's worth privileging. Using the wrong kind of comparison is a very common source of bugs.
I think all existing code disagrees with that. There has been great value derived from being able to abstract over element types.
What you are proposing would double the required number of collection classes and all of its traits, because it would require separate ones for Collection[E <: AnyRef] and for Collection[E <: AnyRef].
There is literally no reason for introducing this complexity. Go has demonstrated how poorly this idea has worked out in practice.
Additionally, this approach would make it nearly impossible to migrate reference types to value types, because it would break all users of the code.
> Meh, just allow NaN to compare equal to itself. Equality is supposed to be reflexive.
That's a complete non-option. You might not like the IEEEs definition of equality, but this is what it is. Messing with it would break all existing code using floating point numbers.
> Unboxed primitives don't have identity, only value equality.
Their identity is the bits they consist of, just like identity on references is the bits of the reference.
> They align well with what's being proposed.
What is being proposed?
> We could build the distinction into the language, so for every type you define you explicitly choose whether it's value or reference.
We already have that: AnyRef and AnyVal.
> Scala's already halfway there with the class/case class distinction.
That doesn't make any sense. The case keyword is basically just a compiler built-in macro to generate some code. It is already doing way to much, and overloading it with even more semantics is not the way to go.
> Disagree; comparison is so fundamental to most types that it's worth privileging. Using the wrong kind of comparison is a very common source of bugs.
What I'm proposing improves the consistency across value and reference types so that it's always obvious which kind of comparison happens:
- identity: Low-level comparison of the bits at hand. Built into the JVM and not overridable.
- equality: High-level comparison defined by the author of the type.
Less than double, because not all collections make sense for both - e.g. a sorted set or sorted map only makes sense if the keys are values.
> There is literally no reason for introducing this complexity. Go has demonstrated how poorly this idea has worked out in practice.
It eliminates a common class of errors. All type-level distinctions add a bit of complexity, but we often consider them worthwhile to make.
> Additionally, this approach would make it nearly impossible to migrate reference types to value types, because it would break all users of the code.
Changing from one to the other is a radical change that should force the user to reexamine code that deals with them.
> That's a complete non-option. You might not like the IEEEs definition of equality, but this is what it is. Messing with it would break all existing code using floating point numbers.
Java already deviated from the IEEE definition with Float and Double. The sky didn't fall. Maybe strict IEEE semantics could be offered in their own type where needed, and that type would neither be value or identity-is-meaningful. (This would mean the type system wouldn't allow you to use the strict-IEEE type in any standard collection, which I think is correct behaviour; compare e.g. Haskell where for a long time you could corrupt the standard sorted set structure by inserting two NaNs).
> Their identity is the bits they consist of, just like identity on references is the bits of the reference.
That's a low-level implementation detail that may not even be true on all platforms. The language semantics should make sense.
> We already have that: AnyRef and AnyVal.
No, those are just implementation details of how they're passed around. Many AnyRef types have value semantics.
> That doesn't make any sense. The case keyword is basically just a compiler built-in macro to generate some code. It is already doing way to much, and overloading it with even more semantics is not the way to go.
Well, what I'd like in an ideal language is: no universal equality, opt-in value equality with derivation for product/coproduct types. As for references... I'm not really convinced there's a legitimate use case for comparing references, especially the implicit invisible references that the language uses to implement user classes. If we need reference comparison at all I'd rather something a bit more explicit - either an opt-in "the identity of this class is meaningful", or a notion of explicit references that were much more visible in the code (something a bit like ActorRef), or both.
> What I'm proposing improves the consistency across value and reference types so that it's always obvious which kind of comparison happens: > - identity: Low-level comparison of the bits at hand. Built into the JVM and not overridable. > - equality: High-level comparison defined by the author of the type.
That's very inconsistent at the language-semantics level; which things are "the bits at hand" are a low-level implementation detail that should probably be left up to the runtime to represent as best suits a particular code path. At the language level, "does 2L + 2L equal 4L?" is the same kind of question as "does "a" + "b" equal "ab"?", and both those questions are quite different from any question to which reference comparison would be the answer.
It makes decisions that break existing code and introduce pointless complexity, while failing to offer any tangible benefits in return.
And it's a perfectly valid question. The real answer is that they wanted strings to have methods, so it had to be a class, because primitives can't have methods; and Java doesn't have value classes, so it had to be a reference type. So the reason is a deficiency in the language.
A better question is, why .NET string is a reference type (with overloaded == to make it behave like a value, even!). It could have easily been a struct. I suspect that this is one of those decisions that were made very early on to basically be like Java because that's what people expect; and then it became impossible to change without major breakage.
The real problem is that most implementations that do reference / value type distinction, mix up semantics and implementation. What we really need is the ability to say that something 1) doesn't have a meaningful identity, and 2) is immutable (in a sense that you can't replace individual components of that object; you still can replace the whole object). When you combine these two, you get something that can be implemented as a value type under the hood, but the programmer doesn't need to think about it. They can just continue treating it as a reference, just as they do in Python. There's no way to tell the difference.
And to avoid issues with is/==, we recommend our students to always use == (except for `is None`). Also related piece:https://blog.rmotr.com/avoiding-being-bitten-by-python-161b0...
[1]> (eq (expt 2 47) (expt 2 47))
T
[2]> (eq (expt 2 48) (expt 2 48))
NIL
re python: this is of course superlame. the compiler can easily detect "is <int>" and forbid the const int table optimization. Only "is None" would make sense.
>>> 42 == 0
True
Equal Rights for Functional Objects or, The More Things Change, The More They Are the Same (1993) by Henry Baker
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.23.9...
b = 1000
a is b
The output is false only if you use the REPL. Run a .py script writing the above code and it returns True. I read about it on SO but can't find the link to it!
It's not necessarily a common thing. Most people aren't running python directly on the internet but being able to screw around with stuff like this is just another important tool in the toolbox when you're attacking something.
> has 4-byte arbitrary global write primitive
> uses it to change arithmetic