Memory use in CPython and MicroPython | Hacker News Reader

Memory use in CPython and MicroPython

lwn.net

164 pointssigna1165 comments

My least favorite part of Python 3 is the new 28 byte int structure which spills over to 32 bytes at 1,073,741,824 (2^30).

    >>> sys.getsizeof(1073741823)
    28
    >>> sys.getsizeof(1073741824)
    32

As a directly result of the complicated implementation of Python 3 ints, we lost some important optimizations.

In Python/ceval.c, BINARY_SUBSCR can no longer easily have a fast path for tuple and list indexing with integers. This makes somelist[0] about as slow as somedict[0]:

    timeit36 -s 's=[0]' 's[0]'   # 0.0320 usec
    timeit36 -s 's={0:0}' 's[0]' # 0.0331 usec

Likewise, the builtin sum() function lost speed on its fast path for summing integers. That path is still there but it now uses the slow PyLong_AsLongAndOverflow() function instead of the PyInt_AS_LONG() fast macro.

    $ timeit27 'sum(xrange(1000000))'  #  7.32 msec
    $ timeit36 'sum(range(1000000))'   # 19.20 msec

IMO, Python 3 int/long unification should have just dropped the trailing "L" from the display of long integers and called it done.

From the user's point of view, that would have unified the two types enough to mostly not care. You would still have two different types for int and long, but otherwise the switch back and forth was already somewhat seamless in Python 2.7.

The range of single cell ints was enormous (up to 9,223,372,036,854,775,807 or 2^63-1). It fulfilled typical use cases without spilling into slower and more complex multi-cell calculations and without impeding useful downstream optimizations for common cases.

    >>> sys.maxint
    9223372036854775807
    >>> sys.getsizeof(sys.maxint)
    24
    >>> sys.getsizeof(sys.maxint + 1)
    36

Once Python 3 was released and the API guaranteed that all integers were the same type regardless of size, that design choice was set in stone. So, now we will have to live with it forever :-(

List indexing is not that fast on Python 2 too:

  (py2) python -m perf timeit -s 's=[0]' 's[0]'
  Mean +- std dev: 31.7 ns +- 0.9 ns
  (py2) python -m perf timeit -s 's={0:0}' 's[0]'
  Mean +- std dev: 44.5 ns +- 1.1 ns

vs. Python 3:

  (py3.6) python -m perf timeit -s 's=[0]' 's[0]'
  Mean +- std dev: 36.7 ns +- 0.8 ns
  (py3.6) python -m perf timeit -s 's={0:0}' 's[0]'
  Mean +- std dev: 41.1 ns +- 1.0 ns

Python's int/long unification was done in Python 2.2 (PEP 237). Which by the way, I think it's a great feature, even if it makes it slower. That is what Python is about after all: simplicity and ease of use over performance. If you really need the max performance possible, don't use Python.

As far as I knew, the only thing that changed in Python 3 was that they renamed the long type to int, and they also removed L suffix. But from your code seems that other things have changed as well and now two structures are used for short integers. Do you know when this happened exactly and why?

> the only thing that changed in Python 3 was that they renamed the long type to int

Python 2.x had two types: int and long. Transitions from int to long were pretty seamless and the two types were somewhat compatible. This is not unification: there were still two separate types.

Python 3 has one type called int that functions mostly like long. You can sort of say (I guess) that "long was renamed to int", but only if you say "and int was removed": really, the types were unified.

When Python 3 broke compatibility, it would be perfect timing to implement tagged pointers, small integers and other performance improvements. At least it would be an easier sell to your boss to rewrite your code base. "We are doing so it runs faster" is much better than "we are doing it for intellectual purity" from a business standpoint.

Agreed. A 'fast' Python, to me, the perfect language. The changes 2 to 3 broke compatibility, but for 'pureness' sake. If speed and the deployment situation were better for python, that would be all I use. I once thought to make a compiler, but don't have the time and honestly knowhow to begin. Nuitka is trying, but also agreeing to live with compatabilty vs performance or type safety. A type safe compiled python is my dream.

Python 3 did not break compatibilty for "pureness" sake. They broke it, because for 6 billion people in the world, for whom English is not the only language, there was a very real pain (bordering on stuff that was impossible).

Most of the world is moving to unicode exactly because of this (I know Sillicon Vally companies are probably moving to unicode, so that their users can use cute emojis).

Most people also need to handle stuff that are in different encodings (I work daily in both Latin1 (north-western European) and UTF8), translating between that was painful/borderline-impossible in the old days (python 2).

The sharp distinction between bytes and strings are essential for any non-english/american/aussie person. Sadly python2 got that wrong, and hence had to die..

Which is probably also why Perl6 choose to go the same route..

If you really need a good reason to go to python 3 other than 6 billion extra potential customers, try "oh look, we have emojis" <sarcasm/>

One might argue that making unicode strings the default instead of a special datatype (which they were in 2.x) is also a "pureness" decision (since you can handle unicode as much in 2.7 as you can in 3.x)

I'm one of those people for whom English is not the only language.

Python 3 is objectively harder and more confusing to use in a multilingual environment. Python 2 didn't pretend that only Unicode exists and Python 2 was encoding-agnostic. This is the correct default for the real world.

Ironically, the effect of Python 3 was the opposite: it made life really hard for those that aren't exclusive anglophones and made life a lot easier for American programmers who now don't have to bother with the whole character encoding mess.

Personally, Python 3's broken support for internationalisation made me drop Python altogether. Python 2 was confusing, but the defaults were at least workable.

Really?

I find it so much easier (I am also not an English language programmer), because it is very explicit about what is the codepoints (i.e. string) and what is the bytes (i.e. on the disk or the network).

The encode/decode functions is basically the only change, but it gives you the reason to explicitly tell it about the encoding you expect..

But as you left python, the point is moot :)

You're right that you can get Python 3 to behave correctly if you jump through some hoops. However, wasn't the point of Python 3 to remove the hoopjumping in the first place?

Anyways, the point is that Python 3's design was broken from the start. Anglophone programmers think that internationalisation means "upgrading" from ASCII to Unicode. In reality, though, the rest of the world's programmers got along just fine long before Unicode was invented and will continue to do fine long after Unicode gets replaced by something else.

True internationalisation means mechanisms to deal with the world's text encodings in a neutral and culture-agnostic way. (And "Unicode only everywhere and no exceptions" is definitely not that.)

The bytes and bytestring types in Python 3 are basically the str type of Python 2 and I don't see that as much of a hoop to jump through. The Unicode default seems like a very sane decision from my point of view. Your argument primarily seems based around an anti-Unicode viewpoint. Am I missing something?

If you see the converting as hoop-jumping then I agree with you.

If you do recognize that there is such a thing called text, that human understand, and for example the text "hello" has five "letters".... The point that that text might be represented in five different (and equally valid) ways with different amounts of bytes is irrelevant to the "human user"...

Hence the sharp distinction between makes sense to most humans.

The danish word "rødgrød" for example is seven letters (i.e. every person would say it is 7 letters long), a computer would call it 7, 7 9 or 16 bytes (in cp865 (nordics) latin1("wester european", utf8 and utf16...)

Unless you work in the ideal "text" mode getting the correct length of the text is not that trivial. Equally, the pair 2-3 and and 7-8 bytes in utf8 must not be split (as either half has not meaning on its own).. hence (in utf8) a function "give me the next letter" will return 1,2,1,1,1,2,1 bytes in its revocations..

Also casefolding (upper/lower case) is hard when working in bytes (some glyphs might not even have one or the other)..

I am unable to say if unicode fits all languages in the world as needed, but it is much better as byte-wrangling if you have multiple possible encodings at once..

EDIT: for reference the bytes of "rødgrød" are

cp865 : b'r\x9bdgr\x9bd'

latin1 : b'r\xf8dgr\xf8d'

utf8 : b'r\xc3\xb8dgr\xc3\xb8d'

utf16le: b'r\x00\xf8\x00d\x00g\x00r\x00\xf8\x00d\x00'

utf16be: b'\x00r\x00\xf8\x00d\x00g\x00r\x00\xf8\x00d'

(and it cannot be written in ASCII ("C"))

Anyone saying that bytes and text are the same are nuts...

> You're right that you can get Python 3 to behave correctly if you jump through some hoops. However, wasn't the point of Python 3 to remove the hoopjumping in the first place?

The hoops of encoding and decoding bytes are not optional, manipulating text and manipulating arbitrary bytes are not the same thing and encoding and decoding is how you translate between the two domains. Python 2 hid this for a subset of the bypes and code was usually broken as a result. Python 3 requires that this split be taken in account in all cases (as do e.g. Java or C#) and is significantly better as a result.

> Anglophone programmers think that internationalisation means "upgrading" from ASCII to Unicode.

Which is a pretty significant upgrade from their previous case of literally not giving a fuck.

> True internationalisation means mechanisms to deal with the world's text encodings in a neutral and culture-agnostic way.

That's completely meaningless word salad.

And just to clarify: it's completely meaningless word salad because

1. encoding and decoding has relatively little to do with internationalisation

2. encoding and decoding is no more culture specific than the source encoding is, if you're dealing with culture-specific encodings aside from not doing anything with the content (not even displaying it) you can't be culture-agnostic until after you've decoded the text

3. you can't "neutrally" deal with "the world's text encoding" (whatever that's supposed to mean in your mind) because most of them are not compatible with one another since they use the same binary space for completely different text mappings

Agreed. I'm in the same position and I'd also love to hear what the pain points were for OP.

Python 2 was also explicit: it had unicode and it had str. I never had a problem managing complex character encoding issues in Python 2.

Objectively os a badly choosen word since as a french i strongly disagreewith you. All my dev and all my new commers training are much much easier in python 3

I've written commercial code to handle decidedly non-english languages in python2 just fine. Have you? Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like.

Yes, I have (for more than a decade in four different languages, not all of those who are latin1)... One project was even both LTR and RTL :)

Byte handling is definately not fun in variable encodings, and even fixed-length multi-byte encodings require a lot of support functionality when working with bytes..

Such a simple thing as "len" is suddenly hard (for the human definition of length and not the byte-length), the latter is also easily attainable by converting the string to bytes and get the length of that in the given encoding.

len isn't a great example because it doesn't do anything remotely useful on python3 strs in the general case. In particular it doesn't tell you how many characters the str has, which is what you probably wanted to know -- possibly as an approximation for figuring out how many columns in fixed-width it would take, in which case of course you are even more likely to be disappointed.

In python2 len(str) will tell you exactly how many bytes some text takes, which is useful.

> Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like.

* Rust has a very strict split between bytes and text

* Go has fucked up bytestrings which depending on validity may break at any point depending on the API you're applying

* Erlang's strings are widely considered awful, and are literally just linked lists of integers making them slow, memory-heavy and prone to pretty-printer whims (as the pretty printer decides on the fly whether a given list looks like a string — and should be printed thus — or does not — and should be printed as a regular list of integers)

> Erlang

What the fuck. And I thought constant-time indexable strings were a bad idea!

FWIW it's way too common in functional languages, Haskell's standard String is a typedef for [Char] (a linked list of Char) hence the proliferation of various other string types.

They are. You want to use iolists in Erlang. Constant time concatenation (which is useful) rather than constant time indexing into codepoints (which is not; and erlang does not have BTW).

Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design and has very nice features that are pretty much unique to Erlang.

Erlang has 3 types of strings:

1. the list of integers thing, which you should mostly avoid

2. binary strings (like python2 str or go)

3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists. So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you?

See http://www.evanmiller.org/elixir-ram-and-the-template-of-doo... for more details.

> Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design

It is nice in a theoretical sense, in the same way Haskell's [Char] is nice, in a practical sense it's awful.

> 2. binary strings (like python2 str or go)

AKA not actually strings, I don't remember Erlang having much if any string-manipulation API for working on bytes.

> 3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists.

This is only for output, so they're not actually ropes.

> So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you?

Most mutable languages will use buffered output streams and possibly some sort of string sequence (even C has iovecs), so while iolists are a good fit for Erlang they make sense in few other languages.

> It is nice in a theoretical sense, in the same way Haskell's [Char] is nice, in a practical sense it's awful.

Haskell's list of Char is awful in both a theoretical and practical sense whereas in erlang's case you can think of it as just a degenerate iolist.

> I don't remember Erlang having much if any string-manipulation API for working on bytes

Elixir is better but erlang certainly has string manipulation functionality for binaries: pattern matching, regular expressions, search, splitting and joining etc.

> This is only for output, so they're not actually ropes.

Not true, lots of non-output functions also accept iolists, e.g. the re module.

BTW, I agree that you want to differentiate between bytes and text, but the underlying representation should be utf-8 bytes, not some constant time indexable utf-32, which is a semantics and performance disaster. It's much better to mess up in not having a clear distinction between bytes and text (annoying, but mostly easy to work around) then to mess up by having constant-time indexable utf-32 as python3 does (not at all easy to work around and even the bits that are easy to work around, like pretending stdin or stdout and sys.argv are unicode on unix are still super-annoying).

> BTW, I agree that you want to differentiate between bytes and text, but the underlying representation should be utf-8 bytes

Sure, but for most languages older than a few years this ship has sadly long sailed and few would be willing to switch to O(n) indexing.

> It's much better to mess up in not having a clear distinction between bytes and text (annoying, but mostly easy to work around)

Except when it's not and the packages you're trying to use simply do not work and fuck up mightily with non-ascii data.

> then to mess up by having constant-time indexable utf-32 as python3 does

It's funny that just above you praise Erlang which does exactly that. Meanwhile Python 3.3 introduced a flexible string representation so depending on the content of the string the internal string encoding may be ISO-8859-1, UCS2 or UCS4 (and yes sadly only these as the designers refused to break the constant-time indexing guarantee).

Where's erlang's constant time indexable utf-32 string?

Erlang's strings are even worse than that since they're non-constant time indexable UTF-32.

Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2?

I do, and I do believe Python 3's approach is superior for strings.

An array of 8-bit integers is not a string. It's an array of 8-bit integers. I want my string type to be a sequence of Unicode codepoints, and I want iterating it to yield codepoints, I want indexing into it to yield the codepoint at the index, and I want the length of it to be the number of codepoints it contains. When I'm ready to send it over a wire or flush it to disk, then I'll think about the best way to represent it as an array of 8-bit integers, but bringing the baggage of encodings (especially the baggage of UTF-8) into any part of my code except the boundary layer is asking for trouble.

What's your use case for iterating or indexing into code points?

My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes.

I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.

But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either

a) low-level mucking around like normalization -- a fringe use

b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases

Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.

I handled a transition of a multilingual/multiencoding Python 2 project to Python 3.

Firstly, you can _still_ work entirely in bytes in Python 3 if you want to. But you can't call print on them... For good reason! You don't know the encoding to use to display them!

When we transitioned, there were a lot of subtle issues in our code due to our almost cargo cult usage of encoded/decode that ended with a lot of redundant transformations. Loads of that disappeared afterwards.

You can still operate in "Python 2 mode" in Python 3. The only exception I can think of is the difficulty in getting a bytestream version of standard output.

Also... Python 2 is decidably not like Go or Rust. The simplest example is how json.dumps would return bytes or a Unicode string depending on the keys on your object. No respect for typing

> The only exception I can think of is the difficulty in getting a bytestream version of standard output.

sys.{stderr, stdout, stdin}.buffer

I am a Chinese and according to my observation Python 2 are still dominant in China. I don't know any companies on Python 3. The idea that a cleaner Unicode handling trumps backwards compatibility for the non-English speaking is disproved by reality. Or at least disproved in China.

Porting from py2 to py3 is a "lot" of work and has a lot of risk. See (was it?) Instagrams keynote at PyCon of how one missing "b" prefix cost them 10% or so percent of runtime performance (can't remember the exact number).

For every project that ports, the risk goes down for the rest (of those who still need to move), as more bugs are uncovered and more lessons/best-practices have been learned.

Hence porting late is an advantage from a cost/risk point..

Also, if there are no strong technical people advocating for the change, most organisation only see porting as a (unnecessary) costly exercise..

I think the judge is still out on whether that is really disproved, but right now you are likely correct :)

> for 6 billion people in the world, for whom English is not the only language, there was a very real pain

Python 3 is different but not necessarily better in this regard. While I hated the encode/decode on .. every .. single .. function call, py3 pretends a lot of bytestreams are instead unicode (file and path names, for example). This can lead to bugs. It's not that it wasn't worth it, but it could have been done a lot better.

> Which is probably also why Perl6 choose to go the same route

Not at all. Perl 5 did unicode right. The difference between character, glyph, code point and byte is clear and uniform throughout the entire stack. Python could have copied this and it would have been awesome.

The issue w.r.t. paths is that paths have different types depending on the operating system. On Linux and most BSDs paths are bytes. On some Unices they are bytes, but must be in a specific encoding (e.g. macOS => UTF-8). On others, they are strings, not bytes (e.g. Windows).

It's implausible to expect Python programmers to correctly handle this, hence fsencode/fsdecode and surrogate escaping.

It's part of the problem of cross platform software. Sometimes you have to go with the least common denominator, in this case byte strings. Do not be smart about it and let the programmer handle the errors. It is unexpected to blow up my software because a user supplied completely valid file name happens to be invalid by Python rules. Yes, fsencode was finally added to work around parts of this problem which is good, but it shouldn't have been there in the first place.

The problem is universal to every time you interface with the operating system. Sometimes it's locale dependent too and you never know beforehand so a program might blow up in the hands of a customer. While the encode/decode process was tiresome at least it gave predictable results.

If you have to work around your runtime in order to give the parameters you want to the underlying syscalls, you are doing it wrong. The premise to move to unicode was a laudable one, but I wish more attention had been paid to how other languages do it, because anything worse than what Perl 5 did more than a decade earlier isn't that great.

Surrogate escaping has been part of Python since 3.2 or so which pretty much solved this.

Python's behaviour is only problematic when all of the below is true:

- Running on Linux and,

- using glibc and,

- glibc's locale files were deleted or the locale is otherwise not configured and,

- you try to read/write non-ASCII file names. (In this scenario Python falls back to the only locale it can process by itself, which is 7-bit ASCII -- the problem is not ASCII, rather the 7-bit restriction. I believe there is a bug open that suggests changing this fallback to UTF-8, which would solve this problem.).

The only time I saw that was with rear which did all of the above on purpose to save like 5 MB on the generated images. In all other cases there is no problem with any kind of file name.

Note that this is a severely broken system configuration where a lot of other stuff falls apart as well.

(Generally speaking I would greatly prefer it if CPython wasn't so reliant on the libc. For example, on a 32-bit system you'll typically find that CPython suffers from y2036 for many date and time operations, including formatting. This would of course add a lot of size to the already big CPython distribution.)

> because for 6 billion people in the world, for whom English is not the only language, there was a very real pain (bordering on stuff that was impossible).

Python2 unicode is not broken, the basic type str() is broken because it defaults to ascii.

I don't understand this. Support was fine: first class Unicode objects and first class tools to encode and decode them to/from byte strings. Not mega intuitively named, but definitely not worth breaking backwards compatibility for an entire language over. Notably the semantics didn't even change; they still have Unicode objects and byte strings in 3, just with different names now. And you still need to encode/decode them. You always will, that's the thing with encodings.

What specific issues do you mean?

If you don't think the sharp divide between bytes and str in py3 is something new, you haven't grasped the value of it..

I am not sure, you are correct that py2-unicode == py3-str and py3-bytes == py2-str ... But even if (and I must admit that it is more then 5 years since I wrote py2 of any length)..

The value is... in py3 "str" is the ideal string, which does things the way humans expect.. When you need to pump in or out bytes, the class is called "bytes" and requires an explicit conversion, where you get prompted to think about encodings.. (but you don't need to think about it all the time as you would with byte-wrangling)

This way, people not well-versed in the horrible mess of encodings are going to be safe by default... If they do something unsupported they get an exception, not something that is wrong... That alone is worth it..

Also, the full stack of python is working correctly with unicode, you don't have to handle two different "string" classes ("str" and "unicode") where one of them is like to do the wrong thing..

One often overlook thing is that py3 defaults to utf8 source code, in the old days integrating stuff from files with non-ascii chars in different encodings was a nightmare...

Sigh, that could have easily been implemented with a "use strict" directive. Or, `from __future__ import strict_unicode`. JavaScript has this strict mode, albeit to solve a different problem, and its success despite its messiness alludes to the notion that gradual improvement is better than sudden breakage.

The problem is that the byte-string confusion is widespread. You can import Unicode literals from future but you still need to update every bit of codec which works with files, network sockets, IPC, etc.

> Sigh, that could have easily been implemented with a "use strict" directive.

You can already remove the implicit conversion in Python 2 (sys.setdefaultencoding(None)), it will break pretty much every non-trivial software as the standard library itself does not have strict separation and many of its "string" API operate on bytes.

> JavaScript has this strict mode, albeit to solve a different problem

That it solves a completely different and almost entirely syntactic issue is a bit of a difference there, you'll note that despite all of its changes the ES committee has refused to come even close to touching strings.

Why would "makes your code run 1% faster" be a better sell than "saves you 1% development time by reducing the number of bugs"?

That tends to be a more difficult sell because the counter-argument from further up is 'But we're not seeing those bugs right now, and won't we find bugs when we do this upgrade?'

It depends on a) the size of your deployment b) whether you are paying SV salaries to your developers c) how sensitive you are to speed.

In many things I'd be happy to double the number of bugs if it meant I could run the code twice as fast. Bugs can be fixed. Slow code is still slow.

Because during the upgrade you would have an explosion of bugs and therefore development time fixing them. The 1% saved by purity wouldn't make a dent.

When Python 3 broke compatibility, the Python team did exactly the thing you're supposed to do with code: first they made it correct (i.e., got all the changes implemented and working as expected), then made it fast.

"We're doing a huge rewrite already, let's pile on more scope" is a recipe for disaster.

It depends. It is really surprising that Python does not use tagged integers, which is a 50 year old technique used by pretty much any other dynamic language in existence.

Using an extension API breakage as the excuse to introduce them would have made sense, since it also breaks the API. Now they cannot happen until Python 4.

And Guido says he won't break backwards compat again in Python 4 as he has learned the lesson. No tagged integers for life.

He learned the wrong lesson then. Which wasn't "don't break backwards compatibility", but:

1) If you break it, break it for compelling reasons.

2) If you provide a tool to convert, don't make it as awful as 2to3

Breaking language backwards-compatibility is one thing; breaking the extension API is less problematic, and breaking the extension ABI is better still.

I believe ABC, one of python's influence, used tagged pointers, but I can't find a quote on this. Not using tagged pointers was a conscientious decision from Guido (see e.g. https://github.com/dropbox/pyston/issues/181#issuecomment-62...)

[edited] Something that people often overlook when talking about optimizing python is that the "simplicity" of the C Python API allowed the rich ecosystem of C extensions. GC, tagged pointers, etc... are great, and often used in modern VMs, but they tend to make integration w/ C code more difficult. Without rich C extensions, no numpy, no scipy, no scikit learn, no tensorflow/theano/etc... I am not sure this is completely incidental.

[edit 2] Ah, found the quote I remembered about tagged pointers in ABC from Guido: https://mail.python.org/pipermail/python-dev/2004-July/04614...

How about "We're doing it so that subtle encoding errors don't shoot us or our customers in the foot."

The comparison is interesting, but CPython's lavish use of memory (which of course has a performance impact by clogging caches, though that's one of the least problems you have with CPython, performance wise) is well known. ¹

This is in part because there are still macros such as PyBytes_GET_SIZE which directly access struct members, and these macros are part of the stable interpreter ABI. That doesn't mean small integer opts and such for length fields aren't possible, it just means they can't happen for Python 3 any more. Tagged pointer would probably break too much code to ever happen.

¹ well known as it may be, people are still surprised that bytes() requires at least 33 bytes (due to the implicit extra NUL byte), an empty string is around 50 bytes and every item in a dict or set takes between 30 and 70 bytes. All this overhead adds up.

Borg works around these problems with a simple hash table (straight out of the text book, with some associated issues). Even though that one is in itself inefficient in how it uses memory, it still only uses a fraction of what the equivalent dict in CPython 3.6 would use. I recently added a similar, pure Python construct in another place (borg mount).

Off topic: Micropython on ESP8266 is a blast.

1. Pick yourself up a < $5 ESP8266 board from ebay (Wemos D1 mini or NodeMCU Lolin).

2. On debian based linux:

- Get the correct binary from http://micropython.org/download

- sudo pip install esptool

- esptool.py --port /dev/ttyUSB0 erase_flash

- esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect 0 /path/to/binary.bin

3. Hack away!

MicroPython is cool, but the incompatibility with CPython's MRO is a deal breaker for me (please correct me if it's been changed). My understanding is that ancestor methods on classes using multiple inheritance are not called in the same as in CPython. This seems like a minefield, especially in codebases using lots of mixins.

Reply on news.ycombinator.com