Say “no” to import side‐effects in Python
chrismorgan.info
chrismorgan.info
1. Global lexical changes to Perl. Sometimes this is the whole point of an import (for example Carp::Always, which typically turns any warning or exception string into a full stack dump). I can't imagine doing something like this in Python. However making soemthing like this work properly without breaking too many things requires a heck of a lot of forethought Yes, even Carp::Always may break something.
2. Manipulation of the importing module's symbol table. This is important for lexical extensions to Perl that you don't want to be global (for example Moo, Moose, and PGObject::Util::DBMethod). Among other things this allows MOPs to be added with greater sophistication than the language typically allows. I am not a Python guru but I could imagine metaprogramming side effects to be useful in setting up a consistent and powerful environment.
The problem the author describes is something which is different though, which not only is a side effect issue but also a violation of separation of concerns. There are certain problems you do not want to solve at import time, and connecting with/configuring external components is almost always one of them.
Why? Because integration with external components is almost always something you want the fine-tuning and decision-making to reside with the application developer. That's very different than setting up a consistent lexical programming environment for use (which is what the acceptable side effects do).
This is the approach taken by gevent, which allows you to replace the entire I/O stack. But importing it does not effect the change; you must explicitly call code to do that. This is done thus:
from gevent.monkey import patch_all
patch_all()
The reason for this is that you want to fully attach the function in the space as a native method and this requires some closures to make work sanely. In short you want:
package foo;
use Moose;
has bar => (is => 'rw');
package notfoo;
use foo;
foo::has(bar => (is=>'rw'));
As I understand it you can't do this in Python not because it is a bad idea (it is a very good idea sometimes) but rather because of limitations in the language (no multiline lambdas, which you need in order to do useful stuff with the symbol table in this regard unless there is an equivalent in terms of defining a method the calling class inside a closure in the imported package). But again, my knowledge of Perl is much better than my knowledge of Python, so I could be wrong.
When you get into this sort of metaprogramming, custom symbol table manipulations are not only helpful but downright necessary. It is what allows you to add a sophisticated MOP when such is not included in core.
Instead, in normal Python it is the importing module that must decide who has access to its own namespace, and who must live within a separated one. It has a completely different set of costs and benefits. (I sometimes wonder if Guido used Perl as a counterexample when creating Python - its principles are almost completely opposite.)
from __future__ import print_functionWhich of the following are side effects?
1. Initializing a point of sale printer, checking for errors, and raising exceptions if, say, it is out of paper? Let's say this is a cash drawer driver and the cash drawer connects through the printer, and if the printer is out of paper, the drawer won't open properly (this happens btw). I would call this a side effect as well as a separation of concerns violation btw. However it is not likely to be a visible change to other modules.
2. Check for the presence of a binary and if found, cache the path to it, perhaps instantiating another object to do so? Definitely a side effect there, but not publicly visible.
3. Initializing an external library's environment (as happened in this case)? Done wrong it crashes the system but I suspect the segfault was not intended. Again it isnt clear to me you have a publically visible side effect intended.
On another, it apparently tried to set up an MPI cluster:
*** The MPI_Init() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[hostname:14114] Abort before MPI_INIT completed successfully; not
able to guarantee that all other processes were killed!
Python 2.7.6 (default, Mar 22 2014, 15:40:47)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')
Please wait a moment while I gather a list of all available modules...
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
import gobject._gobject
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot register existing type 'GtkWidget'
g_type = info.get_g_type()
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot add class private field to invalid type '<invalid>'
g_type = info.get_g_type()
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot add private field to invalid (non-instantiatable) type '<invalid>'
g_type = info.get_g_type()
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: g_type_add_interface_static: assertion 'G_TYPE_IS_INSTANTIATABLE (instance_type)' failed
g_type = info.get_g_type()
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot register existing type 'GtkBuildable'
g_type = info.get_g_type()
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: g_type_interface_add_prerequisite: assertion 'G_TYPE_IS_INTERFACE (interface_type)' failed
g_type = info.get_g_type()
/usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: g_once_init_leave: assertion 'result != 0' failed
g_type = info.get_g_type()But even inside a virtualenv I have had such a problem before.
Our steps to always get it working: make sure mysql_config is in your path env variable, make sure the xcode command line tools are probably installed, and make sure the mysql command line client is setup and working correctly locally. Past that is just works for us on Mac OS X 10.7+. I think 10.6 and earlier also work but I'm not sure.
I'm sure there are other packages that are similar: not so easy to get working with pip but not impossible.
Because of that and similar incidents, I've learned to import argparse up front but nothing else unless necessary. Once argument parsing is done, then importing other modules begins.
How is that at all acceptable? I can't believe that a library that phones home would gain any sort of popularity.
I can't say I know much of anything about the Ruby community, but if they've conditioned you to jump through hoops like importing modules at specific times to avoid delays, that is a serious problem. Conditional/delayed imports have their place, but they should be relatively rare.
curl http://foo.com/tBrwn | bash
It's explicitly user space software, and unprivileged user space software at that, so having it require admin interaction (i.e. touching the system package manager) to install seems like a sledgehammer where a flyswatter would do.
They could use a VCS repo of some kind, but that doesn't handle the various folder and script installs that need to happen - and also doesn't help if there isn't that specific VCS on the system.
Upstream security is less of a concern since that hotlink just points to raw code on Github, and over HTTPS no less.
So what are the negatives here? For software like RVM, this seems like the best, most portable solution that works for the most people.
If you look at the actual script it contains a ton of sudo and also additional curl commands. The chain of security is very lacking.
Even spawning objects that are referenced in modules can have some rather unpleasant properties. The __del__ method will likely never get reliably called and other behaviours that work great in scripts break in subtle ways, especially with a KeyboardInterrupt. Threading and multiprocessing will leave processes running using 100% cpu. Trying to debug these things gets insane, as you can often only find them as the interpreter is dying.
I think import side-effects can be tempting because Python is often introduced using a scripting-oriented approach. Combined with Python following the principle of least surprise, most people doing this won't even realise that it's wrong. This does seem to be a rather common anti-pattern.
IMHO, the way to go here instead is dependency injection:
Inject the configuration into the module through a function or class method (e.g. Flask.initialize({config state}). Wrapping all module functionality that depends on configuration in a class is a good idea here since it allows you to use multiple configurations in parallel and makes your code more modular.
As an example, in BlitzDB (a document-oriented database for Python, https://github.com/adewes/blitzdb) there is no global configuration at all, so you can initialize and use multiple backends in parallel as you please without worrying about side effects. SQLAlchemy does it in a similar way btw.
Of course, that doesn't help the mountains of code written for Twisted that expect a singleton reactor, so we're stuck with it for the foreseeable future. Perhaps in the Brave New World of Python 3, where Twisted is just an implementation detail of the asyncio module, life will be better.
Well, to be fair, that's more an issue with Python than with GTK
pygtk setting the default encoding to UTF-8 is concerning—if true, that is certainly bad behaviour.
The problem about documentation seems to be the usual dilemma that as soon as you know enough to implement an API browser reading GIR that outputs the API in your language, you know enough about how the bindings work themselves to just use the original documentation.
I took a quick look at PyGObject[0] (unsure if that is what one would actually use for this), and the most helpful part seems to be [1], giving some hints on extending GObject.Object, which should be translatable to extending other existing GObject type classes.
As for porting, I agree that can be a pain in that case. When bindings move from a manual implementation to inflating it from an external description. You often end up with good tutorials for the first, and good API reference for the second. If I need to understand other kinds of Gtk bindings, I search for examples on custom TreeModels or other potentially messy things like that to get a feel for it.
[0] https://wiki.gnome.org/action/show/Projects/PyGObject?action... [1] http://python-gtk-3-tutorial.readthedocs.org/en/latest/objec...
I'm OK with Java's static-initializers, but that's because they've got a whole bunch of rules around them preventing common kinds of abuse.
Mind you, my library does not connect to a database, or any such crazy thing. It simply creates a singleton which is then useful throughout your application. There would be very few cases where you would not want that singleton and you would want your own instance. If so, you are free to completely ignore the created singleton and make your own instance. The creation process is also idempotent, except the data you put into that singleton; that data is a special case: it is your explicit responsibility to namespace it, which is indeed the whole point of this library. I feel pretty good about this thing.
I've just put myself a task to reverse the behaviour so that lazy loading is the default.
I liked metaprogramming to an extent in C++. I _love_ it in D.
When I put any code in __init__.py, it's usually just to make the import path shorter for the programmer using my library. So instead of this:
from mypackage.models import Foo
...users can do this:
from mypackage import Foo
And all it takes to support that is putting this in mypackage/__init__.py:
from mypackage.models import Foo
A quick Googling of "Should I put code in __init__.py" shows a lot of people doing that same pattern. It doesn't show a consensus of people saying that more substantial code in __init__.py is an anti-pattern.
As an example for discussion, the first time I wrote a Flask-based back-end, I backed myself into a corner almost immediately in the following way.
Firstly, the WSGI file that the web server uses to start the application followed the suggestion in the Flask docs by doing this:
# webserverseesthis.wsgi
from yourapplication import app as application
# yourapplication/__init__.py
app = Flask("yourapplication")
# Do some general application configuration.
app.config.from_pyfile("/path/to/configuration/file")
# Set up some overarching security things that modify application behaviour.
from flaskext.securityplugin import SecurityPlugin
sp = SecurityPlugin(app)
To compound the error, I then used Flask’s decorators to wire up routes from various URLs to the relevant parts of my code. Those decorators work on the application object (sticking with ideas common to many Python web frameworks and avoiding getting into anything more Flask-specific like blueprints) so I was effectively creating circular dependencies from almost everything to that top-level package:
# yourapplication/pages/home.py
from yourapplication import app
@app.route('/')
def home_page():
# Render home page
# After setting up the application object in yourapplication/__init__.py
import yourapplication.pages.home
However, as soon as you want to write tests or otherwise reuse any of the code in a different context, the entire system is a big bowl of spaghetti with all the usual problems. The moment you import any part of the system to run a unit test on something in it, you get much of the rest of the system as well, complete with the side effects of any imports therein.
This was of course all horribly naïve on general programming principles, but the nature of these frameworks tends to push in this direction, and even Flask’s own documentation features various simple examples that follow a similar approach, so I’ll forgive myself for falling into the trap the first time. I’ve since experimented with various techniques to break the cycles and avoid the side effects on imports, with some success, but frankly I’ve never found a satisfying, general strategy for organising larger code bases built around a web framework.
How is everyone else doing this?
def app_factory(config):
app = Flask("yourapplication")
app.config.from_pyfile(config)
# ...
return app
from flask import current_app as app
@app.route('/')
def home_page():
# ...
def app_factory(config):
# ...
app.route('/')(somemodule.home_page)
# ...
from flask import current_app as app
from myfoo import app_factory
import unittest
class MyFooTest(unittest.TestCase):
# ...
if __name__ == '__main__'
test_app = app_factory(test_config)
# The app proxy will point to that inside test cases
unittest.main()
That’s basically what I did on my second iteration. It is an improvement in some respects, particularly breaking the circular dependencies caused by using the decorators on the global application singleton. On the other hand, now you need some variation of God Object that not only imports all your modules that used to have decorators but also knows enough about their internal implementation to set up the routes and things like pre- and post-request logic directly on the application object you get back from the factory.
The next logical step after that then seemed to be having each module/package that contains views or similar logic provide some sort of initialization function that is declared when you import the module and takes an application object as a parameter. Then we can use app.add_url_rule and friends to wire up the various handlers within each package/module but decoupled from any sort of global application object that needs the circular import. This is the tidiest style I’ve found so far, and all my Flask projects in recent years have used something broadly like it. It does only require one import followed by one initialization call for each package/module, which logically seems to be as good as we can get, given that our starting point is a desire to avoid including any initialization implicitly within the import itself and to avoid depending on global singletons.
Somehow, it still doesn’t quite feel right for some reason. I think it’s because even with that general design, I’ve still got a recurring pattern in each of how I create these modules and how I import and then initialize them. My instinct says we ought not to need that extra boilerplate in a highly dynamic language like Python, but I’ve yet to find any alternative that is neater in general. At least in the most simple cases this only adds a couple of extra lines (converting the decorators to an init function in each package/module, and then calling that function at the top level after importing the package/module), which is clearly better than the earlier, more highly connected designs.
Instead, if you really need long-lasting objects which get initiated once, then use an object, and put any initialisation stuff in it's `__init__` method. Then the module can be imported whenever, but your initialisation stuff is only called when the user of your library creates a new instance of that class.
For bonus points, make your classes able to be used with the `with ...` syntax, so then lifetime is kept to a minimum, and errors/whatever are dealt with by default.
If you really really need to monkey around and take control of the whole python interpreter (gevent, twisted, and possibly some GUI frameworks come to mind...) then don't do that at import time, do it with a `run_forever()` or `take_control` type function.
But yes, a virtualenv for every project does help a lot with not accumulation cruft.
In general - do not do anything that might fail, might take a long time (>0.5 seconds), or can't be stopped easily.
Specifically - do not create windows, do not connect to the internet, do not try to create a database in a hardcoded location, do not connect to postgres. Do not do things that will require calls to the operating system other than allocating memory.
The one thing I'm willing to concede would be reading a default configuration file. But only if you're certain you've written it in a way that won't blow up if for whatever reason the default path is not readable for the current user, or in other edge cases.
If your library needs to talk to the OS, it will need to be passed initialization parameters. Just provide a top-level class that the user instantiates and passes all the needed initialization parameters. Resist the urge to do silly things in it like having a module-global instance variable for it that is set when the class is first instantiated and overloading the __new__ method to return that instance if it's not None.
Windows has DllMain, which lets you run code at load of a shared lib and similar things are possible on Linux, using the -init linker flag and friends.
Finally, Haskell has features that allow you to load, compile and run Haskell (through the GHCMonad, if installed), which allows precisely for those shenanigans.
All those lack the habit that the OP is describing (and which broke for him): the use runtime introspection during development time (possibly in a REPL), but that's another pair of shoes.
Is Rust going to allow side effects on initialization?
$ echo 3 > /proc/sys/vm/drop_caches
$ time python2 -c 'help("modules")' >/dev/null
/usr/lib64/python2.7/site-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
import gobject._gobject
python2 -c 'help("modules")' > /dev/null 1.58s user 0.23s system 69% cpu 2.626 total
$ time python3 -c 'help("modules")' > /dev/null
python3 -c 'help("modules")' > /dev/null 2.00s user 0.17s system 74% cpu 2.928 total
$ time python2 -c 'help("modules")' >/dev/null
/usr/lib64/python2.7/site-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
import gobject._gobject
python2 -c 'help("modules")' > /dev/null 1.26s user 0.11s system 99% cpu 1.375 total
$ time python3 -c 'help("modules")' > /dev/null
python3 -c 'help("modules")' > /dev/null 1.75s user 0.09s system 99% cpu 1.852 total
It's interesting; now that I've tried running it a few more times, `help('modules')` on my Python 2.7 is getting down to six or so seconds. (On Python 3 it takes around 0.15 seconds.)
Python3 behaves well, and both succeed...