Reducing Python's startup time
lwn.net
lwn.net
At Guido's behest, I agreed to treat this miniscule component of start-up time as being important and will review and implement some optimizations in the sprints next week. I've agreed to do this even if it comes at the expense of code clarity, complexity, or maintainability. I expect that when I'm done, _source will be a lazily generated attribute and that much of the code currently in the template will be pre-computed. This seems reasonable to me.
FWIW, I've devoted 16 years of my life to optimizing Python and care a great deal about start-up time. I'm not happy about all the Raymond bashing going on in this thread.
Regarding this post and related comments, I would like to say that, as an external observer who does not know the details of CPython development decision-making process, phrases such as "Okay, then Nick and I are overruled" and "At Guido's behest" sound not so good to my ears.
Maybe you and Guido are best friends, and you are half-joking with that tone, or maybe you are responsible for that part of code, so it is your duty to change it according to BDFL last words even if you do not agree with them... I do not know. Anyway I think it would be better, in the open source world, if a code change is developed by someone who thinks it is the right thing to do, or I could not see so much difference from commercial companies and proprietary software.
Hey, just my two cents. Keep rocking! :)
I did not follow the whole thread, but did not see any "Raymond bashing". Most people are well aware of the elegance and beauty you added to Python.
I am sure Python community has much to gain having this arguments then just blindly follow any trend.
(BTW if you need a quick and dirty way to make Python start up faster, see if you can use the -S flag for your application. It skips the site-specific initialization and has a significant impact on startup times.)
This video talks about it: https://youtu.be/sPiWg5jSoZI?t=1h37m18s
You can do some pretty amazing metaprogramming in Python with metaclasses, and the inspect and ast modules give you access to the guts of the language.
import collections
collections.namedtuple = lambda _0, _1, _2=False, _3=False: "Where is your god now?"
import b
import collections
print(collections.namedtuple('Point', ['x', 'y']))
Wat? It's neither of those things.
Edit: I'm talking about namedtuple, the implementation of which is slow because namedtuple() has to construct _source for every new namedtuple type. Which was clear from the article but not from my comment.
Another purpose of _source is so that you can write it out to a module. This lets you pre-compute the named tuple for speed and it makes it possible to easily Cythonize it or to customize it (in particular, people like to take out the error checking).
Since the namedtuple() factory function was already creating the source string as part of its operation, just storing it and exposing the attribute had nearly zero additional cost.
That said, don't worry about it, next week I plan to make this a lazily generated attribute so you all can sleep soundly at night :-)
I'm definitely looking forward to faster startup though.
I also didn't realise you can't pickle namedtuple by default, which probably cost me a few days on a project, don't think I tracked this down as the reason, but it makes sense now.
It is written in a way to pin a blame on someone (Raymond H) and also looking at python-dev (and even on the github issue) the decision was made month earlier (July) to go with the optimization[1], and Raymond complied[2].
[1] https://mail.python.org/pipermail/python-dev/2017-July/14860... [2] https://mail.python.org/pipermail/python-dev/2017-July/14861...
Let's say you have a light-weight object, like a "full name" which can be represented with a "forename" and an "surname". (Yes, there are many names which don't fit this category. This is a toy example.) So you write:
from collections import namedtuple
FullName = namedtuple("FullName", "forename surname")
name = FullName("Edward", "Smith")
Except you also inherit the tuple behavior, because this isn't really light-weight object but a ... let's call it a heavy-weight tuple.
>>> name[0]
'Edward'
>>> name[1]
'Smith'
forename, surname = name
for forename, surname in names:
...
Or, if you decided to change the API, to make it be:
FullName = namedtuple("FullName", "forename middlename surname suffix")
'namedtuple' has one good use - migration from a tuple object to a real object while allowing backwards compatability to the tuple API. For example, the old tuples once returned from os.stat() and time.gmtime().
It's also fine if it will only be used by code where you have full control over its use and can either prevent inappropriate API use like destructuring assignment, or can update all uses if you want to change the API. For example, a light-weight object used in a test-suite or internal helper function.
Otherwise, don't use it. Spend the extra few minutes so in the future you don't have to worry about backwards compatibility issues.
I was referring to publishing a library which uses a namedtuple as part of the public API.
Your argument seems to be that there are good things about namedtuple - which I agree with - so therefore I shouldn't be so dismissive of it. That might be reasonable if there were no other easy way to construct a light-weight object with immutable attributes.
But the attr package exists, as js2 pointed out in a sibling branch. See http://www.attrs.org/en/stable/examples.html#immutability .
From that perspective, I don't see the problem with returning a namedtuple, if you document it and write your changelogs appropriately when/if you add a field.
If someone uses a module where the top-level docstring says "not part of the public API", and which starts with an "_", then that person is not being reasonable, by well-established Python developer standards. I have no objections to breaking compatibility for people who reach into a private API.
It's different if I hand them an object and say "use it". Even if I don't document anything, it's implicit that this is part of the public API.
These are two different things, though yes, there can be ambiguity.
"if you document it and write your changelogs appropriately when/if you add a field."
I tried to point this out earlier. It takes only a few (boring) minutes to write the essentially boilerplate code.
While over the lifetime of the project it will likely take more time to write the supporting documentation and deal with the effects of breaking compatibility.
Personally, I suck it up and write the boring boilerplate, because I prefer to avoid dependencies and most of the things I work on are low-level. But I'm special that way. ;)
One thing that could happen is, when you might expect to see a type error if you take one of these objects x and mistakenly write x[0], it'd just continue on with some field value. Or if you mistakenly try to iterate over x, you can. Or maybe you were generically walking a tree of objects and didn't intend for an instance of one of these classes to have children. The bafflement I ran into was weirder than these; maybe it had to do with the fact that inheritance was mixed in here too? And I think I'd defined `__getitem__` in some of these classes one of those times.
Sorry I can't remember more. To write https://github.com/darius/tailbiter I had to know quite a bit about Python, but it's a big language! It was a mistake to assume I could use namedtuple as just a convenient way to define a struct type without grokking all its details.
In [13]: from collections import namedtuple
In [14]: import json
In [15]: nt = namedtuple("Test", ["field1", "field2"])
In [16]: sample = nt(field1="foo", field2="bar")
In [17]: json.dumps(sample)
Out[17]: '["foo", "bar"]'If the majority of your use case just immutability and named member access, the tupitility of it doesn't matter much.
In that regard, we just need a lightweight way to specify immutable structs, which I would throw a tupleware party to support.
Maybe include a .to_tuple() method on your immutable structs
>>> from collections import namedtuple
>>> FullName = namedtuple("FullName", "forename surname")
>>> print(FullName._source)
from builtins import property as _property, tuple as _tuple
from operator import itemgetter as _itemgetter
from collections import OrderedDict
class FullName(tuple):
'FullName(forename, surname)'
__slots__ = ()
_fields = ('forename', 'surname')
def __new__(_cls, forename, surname):
'Create new instance of FullName(forename, surname)'
return _tuple.__new__(_cls, (forename, surname))
@classmethod
def _make(cls, iterable, new=tuple.__new__, len=len):
'Make a new FullName object from a sequence or iterable'
result = new(cls, iterable)
if len(result) != 2:
raise TypeError('Expected 2 arguments, got %d' % len(result))
return result
def _replace(_self, **kwds):
'Return a new FullName object replacing specified fields with new values'
result = _self._make(map(kwds.pop, ('forename', 'surname'), _self))
if kwds:
raise ValueError('Got unexpected field names: %r' % list(kwds))
return result
def __repr__(self):
'Return a nicely formatted representation string'
return self.__class__.__name__ + '(forename=%r, surname=%r)' % self
def _asdict(self):
'Return a new OrderedDict which maps field names to their values.'
return OrderedDict(zip(self._fields, self))
def __getnewargs__(self):
'Return self as a plain tuple. Used by copy and pickle.'
return tuple(self)
forename = _property(_itemgetter(0), doc='Alias for field number 0')
surname = _property(_itemgetter(1), doc='Alias for field number 1')
I might as well re-write it as:
class FullName(object):
__slots__ = ("_forename", "_surname")
def __init__(self, forename, surname):
self._forename = forename
self._surname = surname
def __repr__(self):
return "%s(forename=%r, surname=%r)" % (
self.__class__.__name__, self._forename, self._surname)
@property
def forename(self):
return self._forename
@property
def surname(self):
return self._surname
It sounds like you might be happier with attrs if you want an easy to construct class that doesn't have tuple behavior:
If what you need is a cheap, quick container, then you want types.SimpleNamespace.
I wrote 'toy example' for a reason. While it does not make domain sense, it's easy to understand and it highlights the problems I wanted to show.
"it should be used for iterable things"
Really? This article concerned the use of namedtuple in
functools.py
21:from collections import namedtuple
403:_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
If a future version adds another attribute, it will break most code which uses destructuring assignment.
(For that matter, I can't see why it make sense to use a namedtuple for this case. Why not just:
class _CacheInfo:
def __init__(self, hits, misses, maxsize, currsize):
self.hits = hits
...
Or for another case, here's a Python function which returns a class derived from a namedtuple:
>>> from urllib import parse
>>> result = parse.urlparse("http://www.python.org/")
>>> for term in result:
... print(result)
...
>>> for i, term in enumerate(result):
... print(i, repr(term))
...
0 'http'
1 'www.python.org'
2 '/'
3 ''
4 ''
5 ''
If the Python standard library gets it wrong, what is the right solution?
"If what you need is a cheap, quick container, then you want types.SimpleNamespace. "
You mean the one where the documentation at https://docs.python.org/3/library/types.html#types.SimpleNam... says "SimpleNamespace may be useful as a replacement for class NS: pass. However, for a structured record type use namedtuple() instead." ?
I do not think you are correct.
The fact is, the use cases you are showing are wrong, and the one I'm talking about are the proper ones. It's ok to make mistakes, we all do a lot of them. And we will have to pay for those.
If you are interested in the subject, last month we had a huge debate on the Python mailing list about a way to have a proper official immutable but non interable structured record to replace named tuple and avoid those abuses. And also to maybe include something like attrs in the stdlib.
My argument is to resist the temptation to use namedtuple for anything except a migration from tuples to named attributes. While there is some overlap with what you wrote, they are not the same. I would like to know what I'm missing.
@property
def coordinates():
x = self.calculateX()
y = self.calculateY()
return x, y
Point = namedtuple('Point', ('x', 'y'))
@property
def coordinates():
x = self.calculateX()
y = self.calculateY()
return Point(x, y)
x, y = stuff.coordinates
point = stuff.coordinates
point.x
But now imagine you have a response from a ldap server returning a list of authorized person. You could manipulate those persons in a dictionary. Some people may want to have a nicer way to access them, with the look up syntax. You can be tempted to use a namedtuple here, but that make no sense, as a person is not iterable. The alternative is to use empty class, but it's very verbose.
But now you can use types.SimpleNameSpace:
>>> class Person(types.SimpleNamespace): pass
>>> p = Person(name='foo', age=12) # no __init__ to write
>>> p # free repr
Person(age=12, name='foo')
>>> p.age # nice access syntax
12
The bad things are:
- you may need something immutable
- you may want some checks done to restrict the attributes used
Which is why currently people abuse namedtuple. We don't have a good story for those. Hence the debate on the mailing list.
Nevertheless, it's good to remember that the Python community has a philosophy of "we are all consenting adults here" and a good track record for following it. immutability and attributes restrictions are not definitive show stoppers in Python, and SimpleNameSpace is a decent solution while waiting a purer to come up.
I said "'namedtuple' has one good use - migration from a tuple object to a real object while allowing backwards compatability to the tuple API. ... It's also fine if it will only be used by code where you have full control over its use and can either prevent inappropriate API use ... Otherwise, don't use it."
You replied "Only if one is misusing namedtuple".
I interpreted this as an objection to my use guidelines. From what I can tell, you are in complete agreement with me.
And I agree with the criticisms: if namedtuple is slow, but so useful that it's all over the place, perhaps we should extend the language to allow these kinds of easy-to-create data types.
Otherwise, use a dict, types.SimpleNamespace, python-attrs, or a custom class.
FWIW, the origin of namedtuple() was that some variation of it had been re-invented many times and this tool brought together the common features of those tools (substitutability for regular tuples, compact storage, usability as a dictionary key, named fields, a clear __repr__, etc.)
However, in practice it's not used that way. This specific LWN article concerns the following code from functools.py:
_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
def decorating_function(user_function):
wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo)
...
def _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo):
...
def cache_info():
"""Report cache statistics"""
with lock:
return _CacheInfo(hits, misses, maxsize, cache_len())
static PyObject *
lru_cache_cache_info(lru_cache_object *self, PyObject *unused)
{
return PyObject_CallFunction(self->cache_info_type, "nnOn",
self->hits, self->misses, self->maxsize_O,
PyDict_GET_SIZE(self->cache));
}
There is no reason for this to expose a tuple API. And yet it does.
>>> @functools.lru_cache(maxsize=None)
... def fib(i):
... if i <= 1: return 1
... return fib(i-1)+fib(i-2)
...
>>> for i in range(10):
... print(i, fib(i))
...
0 1
1 1
2 2
3 3
4 5
5 8
6 13
7 21
8 34
9 55
>>> fib.cache_info()
CacheInfo(hits=17, misses=10, maxsize=None, currsize=10)
>>> for x in fib.cache_info():
... print(x)
...
17
10
None
10
So while I agree with you, it appears that the core developers do not agree with us.
Regardless, through conversation I am confident that at least one of the core devs thinks about namedtuple in that manner. Not as namedtuple's only purpose, but as one major benefit. Further, the core devs are not a single entity, but many people who don't always agree with each other.
Edit: note Raymond's comment elsewhere on this page.
For the specific case under discussion, ncoghlan added cache_info() on 30 Nov 2010 - https://github.com/python/cpython/commit/234515afe594e5f9ca1...
Previously, "Performance statistics stored in f.cache_hits and f.cache_misses." With the change "Significant statistics (maxsize, size, hits, misses) are available through the f.cache_info() named tuple."
There was no time when it was a tuple, much less in a tuple in a public release.
I'll look first at the namedtuples which have been present since at least 2.7.10.
The named tuples in "urllib/parse", "difflib", "inspect", "sched", and "doctest" were originally tuples. The decimal module's uses a DecimalTuple as a return value from to_tuple(), so I didn't check that history.
ssl uses a DefaultVerifyPaths added by tiran on 9 Jun 2013 - https://github.com/python/cpython/commit/6d7ad13a458afdf2cbd... . It did not replace a tuple. I assume _ASN1Object is the same way.
Now for the namedtuples in recent version control:
dis.py:_Instruction was added in https://github.com/python/cpython/commit/b39fd0c9b8dc6683924... . It did not replace a tuple.
aifc.py:_aifc_params replaces a tuple.
crypt.py:_Method replaced a class with a namedtuple in https://github.com/python/cpython/commit/daa5799cb8866785543... .
nntplib.py:GroupInfo and ArticleInfo added in https://github.com/python/cpython/commit/69ab95105f5105b8337... . It did not replace a tuple.
pkgutil.py:ModuleInfo replaces a tuple
platform.py:uname_result replaces a tuple
sndhdr.py:SndHeader replaces a tuple
sunau.py:_sunau_params replaces a tuple
tokenize.py:TokenInfo replaces a tuple
typing.py:nm_tpl ... is above my paygrade to understand. I believe it's required to be a namedtuple since that what it's trying to implement.
wave.py:_wave_params replaces a tuple.
Finally, urllib/robotparser.py appears to contain a bug in the following:
req_rate = collections.namedtuple('req_rate',
'requests seconds')
entry.req_rate = req_rate
entry.req_rate.requests = int(numbers[0])
entry.req_rate.seconds = int(numbers[1])
In conclusion, 8 of the current uses of namedtuple are not to emulate a tuple. 13 of the current uses replace a tuple.
If a given function was in python2.7 then it was easier. I checked if the code used to be a tuple.
FWIW, that is a norm in the standard library that predates named tuples.
For example, sys.version_info and time.localtime() return structseq instances. Those both have a named tuple API. The are tuple subclasses with named fields. Accordingly, they are indexable, iterable, hashable, space-efficient, and have a self-documenting __repr__().
The two examples you gave are examples of that good use.
The time.localtime() used to return a tuple. See https://github.com/python/cpython/blob/770e4042db8a1c277b5d9... . The change to StructTimeType is an example of how to do that sort of migration.
Similarly, sys.version_info() used to return a tuple. See https://github.com/python/cpython/blob/193a3f6d37c1440b049f7... . Again, a good example of how to migrate.
However, the "this" in the "There is no reason for this to expose a tuple API" which you quoted refers specifically to the namedtuple that cache_info() returns. That did not replace a tuple, it replaced two attributes, as you can see at https://github.com/python/cpython/commit/234515afe594e5f9ca1... .
And returning a simple type with a couple of attributes while also (and needlessly) treating it as a tuple was never a norm in Python before namedtuples, if only because it was a lot of work.
Thus the desire for types.SimpleNamespace as others have noted.
Though we might have better design choices today than yesterday, there's also a downside to code churn. Case in point the annoyance of trying to modernize threading from isAlive to is_alive.
Even when writing new code, it's often wiser to stick with a "worse" standard than to use a new design.
I disagree with raymondh's statement that "that is a norm in the standard library that predates named tuples."
Other people can and do use tools incorrectly. I use tools incorrectly. That doesn't mean we shouldn't learn from the mistakes of others.
It's incredibly frustrating, because this leads to external libraries abusing mutable objects everywhere, Django being one of the worse offenders.
Buried imports free the interpreter from doing something until it is actually required; really nice if you just want to run "--help" and not wait 4 seconds for utterly unrelated modules to be found. It also creates this interesting situation where a script can technically be broken (e.g. module not found) but you don't care as long as the part you're using is OK.
Grouped imports are undoubtedly nice for purity and easily seeing dependencies but they may not be smart in a dynamic language. It is still pretty easy to "grep" to find imports if you're trying to track dependencies.
Removing eval from tuple would not solve your issue. You'd need to instead remove eval from pickle, of sorts. The trouble is that pickle tries to look up your namedtuple by name and can't find it in your module.
https://github.com/pypa/setuptools/issues/926
There are hacks to get around it, but it's a deep hole.
:(
https://www.mercurial-scm.org/repo/hg/file/tip/hgdemandimpor...
Basically, it's a lazy loading of all imports. You can write `import foo` but it won't actually be imported until you do `foo.whatever()`.
It's a crutch, and it's true that hg still pays the overall startup cost of Python. However, even with "45 times slower than git", the situation is not so dire:
jordi@chloe:~$ time hg --version
Mercurial Distributed SCM (version 4.2.2)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
real 0m0.096s
user 0m0.084s
sys 0m0.008s
jordi@chloe:~$ time git --version
git version 2.1.4
real 0m0.001s
user 0m0.000s
sys 0m0.000s
Pierre-Yves also has an interesting talk about all of the tricks that have to go into hg in order to make it fast with Python. There's stuff like doing `method = object.method; method()` instead of doing `object.method()` over and over again to avoid paying the cost of method lookup and so forth:
It's bad for many things -- like showing VCS status on your shell prompt or IDE status line, where you repeatedly start hg.
100ms+ delay every time you hit enter in your shell is noticeable (and that's just the version, now doing an actual "dirty" check takes more).
That can also be used for frequent polling like shell status.
PS1="${PS1}\$(sleep 0.1)"
> Concluding, I think we should move on from the original implementation and optimize the heck out of namedtuple.
I remember reading this on the mailing list and thinking "yes, namedtuple deserves optimization because it's an excellent resource".
For my use cases, I don't think I notice Python's startup time. When I write Python code it's usually not performance critical. When I do write performance-critical Python code, I usually care about total runtime and PyPy is usually a win here. Aside: if you use PyPy, you should probably be using namedtuples. IIRC it models those much better than it does dicts. And IMO namedtuples model many common data structures better than dicts do.
Just because I don't feel the pain of Python's startup time doesn't mean we shouldn't try to optimize it. I think I find my way into some esoteric Python/CPython corners but FWIW I've never needed namedtuple's "_source" attr.
NT = namedtuple("NT", ('a', 'b'))
default_NT = NT(42, 'foo')
my_new_NT = default_NT._replace(a=7) NT.new = lambda **kwargs: NT(42, 'foo')._replace(**kwargs)
nt1 = NT.new()
nt2 = NT.new(b='bar')
from typing import NamedTuple
class Point(NamedTuple): x: float y: float = 0.0
and then Point(1.0).y == 0.0.
Man, I don't see many positive opinions of exec out there.
> I think we should move on from the original implementation and optimize the heck out of namedtuple. The original has served us well. The world is constantly changing. Python should adapt to the (happy) fact that it's being used for systems larger than any of us could imagine 15 years ago.
I really wish the BDFL could say the same for CPython itself. This continued insistence of treating python as a teaching language with a teaching implementation is really weird.
It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to teach than Py2 and thef. bad. Crazy.
EDIT: OK, if not primarily teaching, spending a lot of his time teaching it. That's a big part of his motivation + worth knowing that when assessing his arguments.
Why is wanting something to be easy to teach crazy?
It can be a factor: but to place it higher than startup time is prioritising the one-off teaching experience of a few over the repeated development experience of many. For a tool for development. Crazy, no?
The vast majority of software development is not hitting any kind of scaling or performance bottlenecks. So optimizing for simplicity isn't as crazy as you think.
Businesses use Python because it is easy to pick up, not because it's fast
The main "personal feeling" involved is just a sense of responsibility for this code. All of us involved in the optimization business have to make judgments that strike a balance between speed, code clarity, maintainability, and risk of having bugs.
For the most part, we want to focus our optimization efforts on things that have a high payoff. For example, we're always having to decide how much of the library to write in pure Python and how much to write in C. If all we cared about was speed, the entire standard library would be written in C (in fact, there is a suggestion for Stefan Bethel to Cythonize the whole library).
In the case of the proposed patch, my judgement was that we would be net worse-off with the patch than without (in part because namedtuple costs are typically a tiny fraction of start-up time). Guido disagreed, so I will go forward with the patch. That is the way open source works.
This is less of "pride" issue and more of a difference of opinion about whether a particular patch was a net win.
This sentence is very surprising to me. If the best way to implement something is to literally build a string of source code and pass it to the interpreter, to me it means the user is unable to really express what they want in the language.
As a (terrible) example, imagine a language whose number addition operator only works on literals.
1 + 2
=> 3
a = 1
b = 2
a + b
=> ERROR!
foo = read number!
bar = read number!
baz = concatenate x, ' + ', y
quux = eval baz
> The approach of generating source code and eval-ing it, is a cool demonstration of <this language>'s expressive power.
...but it clearly wouldn't make sense. Functions like exec and eval are an escape hatch for when there is no sensible way of expressing something in the language.
That said, this particular use case for eval is basically macros lite: the code being evaluated doesn’t generally depend on input to the resulting program, so it could have been expressed as a transformation to the module source code, i.e. a macro. And a true macro would be able to provide more natural syntax, whereas namedtuple requires stuffing the field names into a string (or using a wrapper based on some metaclass thing, but the syntax for that isn’t great either). Thus, languages that have macro support, both dynamically typed (Lisp) and statically (e.g. Rust), could be called more expressive in this respect.
How tiny do the steps in this thought process need to be? We can go on all night.
But it still has to call into Python to get my mercurial status (we use mercurial at work so I need to have that).
But I doubt that the namedtuple change will noticeably decrease startup time for most applications (after having experience with this problem for 10+ years).
From a comment over 3 years ago:
[Python interpreter startup does] random access I/O (the slowest thing your computer can do) proportional to (large constant factor) x (num imports in program) x (length of PYTHONPATH).
https://news.ycombinator.com/item?id=7842873
I don't think that instantiating a Python parser 100 times for exec() is going to compare to that. I guess the difference is that I'm thinking about the cold cache case -- namedtuple might be noticeable in the warm cache case.
And there are many many command line tools that start slow because they're written in Python, not just Mercurial.
Mercurial is actually one of the best because they care about it (demandimport) and they don't have too many dependencies. IIRC some of the Google Cloud tools take 500-1000+ ms to start because they are crawling tons of dependencies with long PYTHONPATH.
Also: for the same interpreter, the same "*.pyc" files and the same PYTHON PATH, we could dump a cache file and share it across interpreter starts if no dirs in PYTHON PATH have been touched.
If it's not done, then it might be interesting to bring that up to the Python ideas mailing list.
The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right.
Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.)
I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)
For example, I supported a 10 year old set of CGI scripts running on a machine with Lustre filesystem where every filesystem metadata lookup was painfully slow. (See ehiggs' comment at lwn.) I spent time to trim away every import to the bare minimum.
Once you do that, other factors become more significant. (Remove your biggest problem and something else becomes your biggest problem.)
If the view is that a few milliseconds here and a few milliseconds there isn't a problem, well, those milliseconds add up.
'import scipy' (which depends on numpy) takes 3x longer to import than Python itself does. I have replaced dependencies on scipy when the overhead of importing the one function I needed took longer than my program took to run.
There's an old story along these lines, recounted at https://www.folklore.org/StoryView.py?story=Saving_Lives.txt :
> One of the things that bothered Steve Jobs the most was the time that it took to boot when the Mac was first powered on. ... One afternoon, Steve came up with an original way to motivate us to make it faster. ...
> "You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day."
> "Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"
How many people use Python? How many times is it started per day?
I just tried with Python 2.7 and 3.6 and the hello world startup time is 100ms which is awful lot.
On an app I worked on the unit tests took >10 seconds to boot on each run. Among the languages available to Google developers, Go ends up being the lightest weight, despite there being plenty of other lighter languages available outside. (This is not the place to start an argument about language plurality.)
buildout tends to append millions of path to os.path
Annnnd your python project start time requires 1 minute or more because each import requires scan millions of directories.
$ time java &> /dev/null
java &> /dev/null 0,09s user 0,03s system 120% cpu 0,102 total
$ time ruby -e 42
ruby -e 42 0,04s user 0,01s system 99% cpu 0,053 total
$ time python -c 42
python -c 42 0,03s user 0,00s system 98% cpu 0,030 total
$ time perl -e 42
perl -e 42 0,00s user 0,00s system 92% cpu 0,005 total
Go produces compiled binaries and, as such, exhibits no such startup delay. Startup times for Go programs that I tested are in the same ballpark as Perl above, i.e. not reasonably measureable with a millisecond-precision tool like time(1).
$ time python -c ''
real 0m0.071s
user 0m0.064s
sys 0m0.000s
$ time python -c 'import functools'
real 0m0.073s
user 0m0.052s
sys 0m0.016s
$ time python -c 'print(1)'
1
real 0m0.067s
user 0m0.015s
sys 0m0.025s
This is a somewhat contrived example, however. As the article notes, it's worse when you have a lot of imports (and I don't know how that'd differ between versions 2 and 3). I remember talking to some folks at Google who work on the Google Cloud CLI, and they said that Python startup time is a constant problem for their (large) codebase. Especially when it's used for tab completion, because it might run every time you press tab, and the user wants instant results.
Relevant ticket: https://github.com/catkin/catkin_tools/issues/297
$ time python3 -c 'import tensorflow'
real 0m4.522s
user 0m1.712s
sys 0m0.466s