PyTorch – Internal Architecture Tour | Hacker News Reader

PyTorch – Internal Architecture Tour

blog.christianperone.com

193 pointsperone22 comments

“For that reason, the statement a = 200; b = 200; a is b will be True, while the statement a = 300; b = 300; a is b will be False.”

This is kinda funny.

It is similar to strings up to some size (I think it was 1024 but I may be way off base). Basically small strings are interned.

Same thing happens in Java - Integer objects are the same between -128 and 127.

Another common trick along those lines is pointer tagging, where you use some of the bits in a pointer/reference to store flags and if those are set, put data in there directly.

E.g. you could on a 32-bit machine reserve the least significant bit to mark an integer, and if it is set the high 31 bits are treated as a 31-bit integer instead of a pointer. Since your objects are likely bigger than a byte, you don't actually need all 32 bits for addresses and thus don't even loose addressable memory space. On 64bit machines you can do it even more, since they are far from being able to use all bits for actual memory, and you might even fit a short string or other more complex type in there.

Clever trick.

Here's an interesting case study by Mike Ash on how the Objective-C runtime does tagged pointers for strings: https://mikeash.com/pyblog/friday-qa-2015-07-31-tagged-point...

lua stores extra data in 64-bit pointers. I feel like I should take advantage of unused space in pointers myself (in C++). x86 uses only 48 of the 64 bits.

Java does string interning too: all constant Strings are stored in an interned cache, as well as Strings that have manually been interned with String.intern() method (which makes teaching "don't use == to compare Strings" in AP Computer Science all the more difficult!)

It's a optimization strategy. Many languages have an configurable integer cache, including python.

Funny, but also efficient.

Funny? I'd say back to the drawing board!

Why? There is no reason to assume that two integers which have the same value should be the same object?

On the other hand if you want to check if two integers are different, why would you check if they are different objects, after all they could still represent the same numerical value.

I mean I get that this is something that probably bites someone at the worst possible moment, but isn't this the case for most optimizations?

The problem is that if you allow too many of these inconsistencies, then you're almost destined to run into one of them. It's better to eliminate them while you still can.

You may not realize that the removal of the inconsistency would be so that

  a = integer, b = same integer
  a is b

is always false.

The inconsistency is that the two separate variables reference the same object just because the values are the same.

In general, you shouldn't care about identity (is), just about value comparison (==).

Let's see. The first mistake is of course that the object identity operator is called "is", and should have been called something like "is-same-object-as".

Then, a is-same-object-as b, for integers could raise an error, as a and b are not objects.

If that's not desired (for some reason), then a is-same-object-as b could indeed return false in all cases, and that would at least be more consistent than returning true for integers smaller than some number and false otherwise.

> is-same-object-as

I don't expect that's a language I'd enjoy. You may prefer Objective-C over Python.

> a is-same-object-as b, for integers could raise an error, as a and b are not objects.

That would be a greater inconsistency than the minor additional optimization for small integers. It's a Java performance optimization that has crept into the way you think about the data, impacting much more of your program than just "is-a-long-name".

  >>> isinstance(1, object)
  True

If you think that's funny, here's a nice overview over some python gotchas: https://github.com/satwikkansal/wtfPython

Great. I am wondering which tools he used to draw object diagrams. It looks neat.

I used Sketch (https://www.sketchapp.com/), the graphs of the objects are actually from a built-in template for iOS notifications lol, but it worked well for that purpose.

Awesome. Off-topic: this reminds me of Stardock's Galactic Civilizations:

I bought a book “Teach Yourself C in 21 days” and “OS/2 2.0 PM programming” Because I couldn’t afford anything more, everything in the game I created, Galactic Civilizations, could be found in those 2 books. [...] Let me give you an example – in GalCiv on OS/2, each star ship is actually a full blown window that is of style SS_ICON. So when you move a ship in the game, I’m just using WinSetWindowPos to move the ship X,Y coordinates. There’s no “graphics” in the game per se, just all icons being moved around. That’s because I couldn’t afford any more books than those two and they didn’t cover graphics programming, just icons and window movement.

Source: https://www.stardock.com/stardock/articles/article_sdos2.htm...

By the way, OmniGraffle is also great for drawing graphs on the Mac.

haha nice, in the end, we squeeze what we know. Didn't know about OmniGraffle, will take a look.

> auto array = (PyArrayObject* )obj;

Why not reinterpret_cast<PyArrayObject * > here?

(extra spaces are so Hacker News doesn't make this italics)

Reply on news.ycombinator.com