Be careful with exec and eval in Python | Hacker News Reader

Be careful with exec and eval in Python

lucumr.pocoo.org

45 pointsmattyb33 comments

Is there any reason why you would want to use exec or eval? In any language? I have never seen the need for it, but have seen plenty of very poor uses of eval in PHP and JavaScript. Are there any legitimate ones, aside from writing a debugger of some kind?

I've seen it used in JavaScript for compression before, on the same theory as one of those installers that starts off by unpacking itself. That's not as important now that everything gzips.

I've also used it myself on occasion in quick-and-dirty scripts, but never in production code. It's just too easy to make a small mistake and blow your security wide open.

I guess that is one case I can see it being used: where actual manipulation of the source code is the only way to get the job done.

In Python versions older than 2.6 and on Google appengine it's the only reliable way to generate bytecode. For things like template engines this is important to not add extra overhead over an already slow language.

That seems like a terrible approach to me. If your application's bottleneck is in parsing templates, either everything else is really fast or you have way too many complex templates. Maybe caching is a better strategy in that case. Then again, for a variety of reasons I have decided not to touch GAE if I can help it.

> If your application's bottleneck is in parsing templates, either everything else is really fast or you have way too many complex templates.

There are countless of examples where the template engine is the bottleneck of an application. If you are doubting that, I welcome you to try it yourself. Benchmarking Django's templates and Genshi is an eye opening experience. :)

Proof is in the pudding. Where's the pudding? :)

From a quick google search, they seem pretty comparable. And in fact Django oftentimes wins.

http://genshi.edgewall.org/wiki/GenshiPerformance

http://stackoverflow.com/questions/1324238/what-is-the-faste...

Never trust a benchmark you didn't run yourself. A realistic Genshi template is butt slow. More than a handful function calls per byte emitted is not uncommon. Regarding Django it become a lot slower when they introduced automatic escaping and loop item unpacking a while ago.

I have seen many people switching from Django templates to Jinja2 especially because of the increased performance.

I believe Mitsuhiko was trying to draw comparisons of Django or Genshi engines to those of Jinja2 or Mako. There was a well depicted article, year(s) back, showing (profiling) where Django templating was losing a lot of work, compared to Jinja. Unfortunately, the article appears to have fallen off pocoo.org, but you can find the HN discussion at the following: http://news.ycombinator.com/item?id=726461

I don't see it as so much of a performance issue as a usability issue.

Tornado uses eval for parsing templates. The nice effect of this is that you write Python snippets in your templates rather than learning an entirely new DSL for it (like in Django). It's pretty nice as long you're disciplined enough to not put business logic in the templates.

Eek. Sorry, I disagree. IMHO, templating engines should not allow you to do much more than simple substitution, loops and escaping. I think Django strikes a very nice balance and learning to use it takes minutes. This is like saying that you should be allowed to use ASM in your Python code, as long as you promise never to actually do it.

Say what you will, but I find my Tornado templates much less verbose than in Django, even when respecting the rules of "proper" templating.

Also, I just realized you're my former coworker. Hi Igor, it's Yusuf :)

Small world!

Terrible how? I find frameworks are more than welcome to make use of optimizations that are not going to have ill effects on the user. For example, I've yet to find fault in Jinja2's ability to use bytecode-cached templating.

How do you define complex templates? And as Mitsuhiko has pointed out, it is not just a problem with GAE.

By complex I mean templates that make your templating engine go slow. Unless you are using a terrible engine, which is slow no matter what it's parsing.

Perhaps you need to define slow or run the benchmark(s) yourself. Sorry to be pedantic. I don't really understand what you mean by "templates that make your template engine go slow," as business requirements are going to trump anyones' skewed, dogmatic reality on the [controlled] use of eval, exec, etc.

Where my data is coming from local cache, the template does stick out in the request/response chain.

Pedantic is fine. I mean templates so complex as to become a bottleneck in processing web requests (rendering the template takes more time than any other operation performed). http://en.wikipedia.org/wiki/Bottleneck#Bottlenecks_in_softw.... In your scenario, if you can cache the data, you can likely cache rendered templates, or parts of the rendered templates, or introduce a caching reverse proxy.

It is useful for autogenerated code. Which is a big sledgehammer that should only be swung carefully. But one use case is that you can have a DSL that you convert into code in your target language that you then eval.

Given the ease of going horribly wrong with this strategy, I strongly recommend only using it in cases where it is obviously a huge win. Such cases are not non-existent, but they are rare.

The only good use I've seen for turning a template into a callable bit of code for performance.

Here's a JavaScript example of such a templating function:

http://documentcloud.github.com/underscore/docs/underscore.h...

I've used it in Matlab to get around the lack of first class functions and function pointers.

Heh. Clever, and I suppose nobody would be trying to find security vulnerabilities in Matlab scripts.

I was just about to post the exact same thing. I guess it must be a more common technique than I imagined.

It can be somewhat handy when working in interactive mode, though it has a very informal feel to it.

One (bad) example would go along the lines of:

>>> def f(x): ... if x in dir(scipy): ... print eval("scipy." + x + "(2)") ... >>> f("sin") 0.909297426826

Then you can let the user plot whatever function they want.

Can't you use scipy.__dict__ to look up the function name in this case?

  scipy.__dict__[x](2)

It would be preferable to use getattr(scipy, x) rather than reaching inside the internals of the object (module).

    getattr(scipy, x)(2)

Short answer: no.

Long answer: I have used it as part of a function that generates functions when I was writing an AST-generating portion of a parser. You need to make a bunch of similar functions, so you make a list of tuples containing all the bits that differ between the functions, then write a generator function which converts these strings into real Python functions which you can then eval in global context. Surprisingly enough this actually made the code more readable and easier to maintain. :)

But that is literally the only time I have ever thought it was a good idea -- and that's in code where I completely control the data going to exec(), in code that will never be in an Internet-facing arrangement.

Ruby typically uses it pretty heavily (and to great effect) alongside its metaprogramming capabilities.

How about a REPL?

http://svn.python.org/view/python/tags/r271/Lib/code.py?revi...

I use compile/eval in an IRC bot I wrote (http://github.com/rmmh/skybot) for plugins.

It lets me do hot-reloading of updated files, easily examine the namespace to find functions marked with various decorators, and not litter the plugin directory with unnecessary .pyc files.

The builtin imp module has problems with reloading already-loaded modules, and I might have been able to do some weird hacks with __import__ to make it work like I wanted, but when the core functionality boils down to 3 lines of code, why bother?

exec is useful to create a scripting environment inside a GUI application, without the need for separate processes/interpreters. Not without its issues, but works well enough for relatively simple needs.

A nugget from an old version of WordPress:

  eval('$v_result = '.$p_options[PCLZIP_CB_PRE_EXTRACT].'(PCLZIP_CB_PRE_EXTRACT, $v_local_header);')

Reply on news.ycombinator.com