Abusing Python exceptions to turn a recursive function iterative
gist.github.com
gist.github.com
def invite_accepted(sender, email, **kwargs):
import sys
caller = sys._getframe(2)
request = caller.f_locals['request']
request.session['invite_accepted'] = TrueI was monkey patching an external library in JavaScript today, rather than having to fork the thing for one slight modification, and thought about this philosophy again.
Monkey patching is how you get into situations where the order in which libraries get loaded causes software to unpredictably work or break. Or, worse yet, to work reliably on the developer's machine and break in the wild. (Because the external library on someone else's server loads slower for the developer than their local library, and faster for some users.)
I think you learn when not to do it, which is most of the time.
Code can end up living a lot longer than people think it will (see Y2K bug).
One of my biggest pet-peeves is how since popular media hyped Y2K up as a potential world ending disaster when nothing happened people thought the whole thing was a myth. When it was not a myth (though it was overblown), it was just avoided because people engineered around it. I can't wait to see how people handle the Year 2038 Problem.
Things I've been reading about python recently makes it seem to me like python people are re-learning lessons from perl.
For instance, one can override most of the builtin functions like len().
Further reading: https://www.python.org/dev/peps/pep-0509/
Your beef is likely with the reality of doing business and shipping a real product.
Sloppy Python is particularly bad, because Python gives you a massive footgun to play with. Like you could write functions that radically alter its behaviour depending on where it's getting called - that's similar to the hack that the parent post was talking about.
It's easy to build systems that are coupled in really weird and unexpected ways, and that makes reasoning about code much harder than it needs to be.
Not that you are wrong, just saying...
exec(
code_fragment,
frame.f_globals,
frame.f_locals
)
# update locals
ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))
https://pydev.blogspot.jp/2014/02/changing-locals-of-frame-f...
See: https://gist.github.com/llllllllll/b1ac68a6b77535a64c0b13bfd... https://gist.github.com/dutc/d2ff9f17a0520d233b2d197d1dafa1a... https://twitter.com/__qualname__/status/867928378040438784
Alternative Python implementations do not follow all CPython implementation details.
In other words, if this is not part of the Python specification, it might work on other Python implementations even if it does not work on CPython.
Which is Java with Python syntax...
Last I collided with it, it was terribly slow, but that is not the point here.
Some of these alternative implementations are very interesting. PyPy is JIT'ed, and insanely fast compared to CPython. IronPython, IIRC, has no GIL.
But then again, I was playing around with the function bytecode to transfer functions between instances in some of my for-fun hacks (you can access the byte code of a function and manually create functions from byte code in Python through "code objects").
A pickle bytestring can execute completely arbitrary code. I have used them in my work.
An easy introduction: https://www2.cs.uic.edu/~s/musings/pickle/
An example:
payload = b"ctypes\nFunctionType\n(cmarshal\nloads\n(cbase64\nb64decode\n(S'4wAAAAAAAAAAAQAAAAIAAABDAAAAcxYAAABkAWQAbAB9AHwAagFkAoMBAQBkAFMAKQNO6QAAAAD6EGVjaG8gXCMgcm0gLXJmIC8pAtoCb3PaBnN5c3RlbSkBcgMAAACpAHIFAAAA+gc8c3RkaW4+2gdwYXlsb2FkBAAAAHMEAAAAAAEIAQ=='\ntRtRc__builtin__\nglobals\n(tRS''\ntR(tR."
from pickle import loads; loads(payload) # don't do it...!
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
from pickle import loads; loads(payload) # don't do it...!
File "...\lib\pickle.py", line 1388, in loads
return Unpickler(file).load()
File "...\lib\pickle.py", line 864, in load
dispatch[key](self)
File "...\lib\pickle.py", line 1139, in load_reduce
value = func(*args)
ValueError: bad marshal data (unknown type code)Arbitrary execution via pickle creates holes in this mechanism, requiring additional mitigations.
PEP-551 briefly discusses security motivations and potential mechanisms for environments where code execution is locked down:
In that example, subclasses of Runnable can be transferred. However, only the corpus of the "execute" method and the "properties" object is transferred. If you need to access modules, you'd have to import them inside the execute method.
But if you actually plan to use this code in production, I would strongly advise against it :). There is a better solution available that will survive django upgrades:
Add a middleware that stores the session object in a thread-local variable for the duration of that request. You can then access that variable from your event handler.
This has come up many times, and I've never been able to figure out how to do this without something redis. Do you know any open source examples of this?
import threading
_local = threading.local()
class SessionKeeperMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
_local.session = request.session
response = self.process_request(request)
_local.session = None
return response
def my_callback():
do stuff with _local.session def create_callback(request):
def invite_accepted(sender, email, **kwargs):
request.session['invite_accepted'] = True
return invite_accepted
In general this is why I don't like the direction Django is going. The Python idiom for this would be to create the callback as a closure after the request is available, and then pass it into the library function that calls the callback. I suspect you might still be able to do this, but you'd have to use a function-based view, and that's no longer idiomatic Django.
It seems like what Django folks want is to write a configuration markup language that is a subset of Python classes and not actually ever write any function bodies. The problem with this approach is that you're limited to the configuration points the framework gives you (in this case, the arguments passed to the callback). They can give you workarounds (i.e. middleware) but this is still fairly hacky. It would be better, IMHO, to stick with function-based views and provide library functions to do what the framework does, which can be called through function-based views in a way that's idiomatic with the host language.
But Django has gone too far down this path to change now, so I guess I just have to accept it. :)
I thought, preventing Abuse in all ways was the biggest point behind Python.
This is abuse: https://github.com/ajalt/fuckitpy
Heh.
[0] https://msdn.microsoft.com/en-us/library/aa266173(v=vs.60).a...
"On Error Resume Next specifies that when a run-time error occurs, control goes to the statement immediately following the statement where the error occurred where execution continues. Use this form rather than On Error GoTo when accessing objects."
https://gist.github.com/llllllllll/b1ac68a6b77535a64c0b13bfd...
(Python's LOAD_FAST bytecode does `fastlocals[i]`: how can we abuse the lack of bounds checking on this array access?)
(We've also discussed potential extensions this to approach "lift" C-extension code in bytestrings into interpreter objects. This would be useful to escalate existing interpreter attacks in environments that try to lock things down.)
t = (0, 1)
tuple_setitem(t, 0, 99)
print(t)
# (0, 1)
Is there a use case for something like this?
If you want an iterative function, write one. It's not harder than writing a recursive one (although sometimes a bit less readable).
Writing an iterative function is also both faster than the recursive function, and much faster than the hack.
Still a neat hack, though.
Then you have other problems, where the recursive solution is actually not super easy to understand why it "works" immediately (think, balanced paren generation) and going on to create an iterative solution, while not insanely hard, is not something super obvious.... what's your condition to stop iteration, again?
Maybe I'm thinking about that the wrong way, I'd love to learn more about it!
That problem is closely related to tree traversal, anyway, I should fool around with those two ideas a bit more to properly understand.
Like I said, it is sometimes a bit less readable (tree-walking is a good example of such case), but at the same time, what the code actually does is much clearer. You have no hidden costs, and the code is easier to optimize this way.
You mention a case where you do not fully understand why the recursive solution works, in which case you obviously can't easily write an iterative solution. However, in this case, you are poorly equipped to make any implementation, recursive or iterative.
[1] http://josf.info/blog/2014/03/21/getting-acquainted-with-clo...
Was more of an interesting challenge personally.
We can do it for general JS programs. If you are willing to compile Python to JS, you can transparently use our compiler to get heap bounded stacks.
A cool result of doing this in JavaScript is that any language that compiles to JavaScript (python, ocaml, Scala, c++, clojure...) can transparently get heap bounded stacks by just using our compiler.
For some more examples, look at pyret (pyret.org) that also supports heap bounded stacks. (In fact, we were able to strip out the Pyret compiler and just use Stopify to get all these benefits)
>>> def recurse():
... return recurse
...
>>> recurse
<function recurse at 0x1083b5f28>
>>> recurse()
<function recurse at 0x1083b5f28>
>>> recurse()()()
<function recurse at 0x1083b5f28>Reminds me of the fact that Python doesn't have a char type and it's strings all the way down:
"a"[0][0][0][0] == "a"
Also, what does `recurse()()()` mean? What are we looking for exactly?
recurse == recurse()()()()...
The syntax can occasionally be useful if you have a function the generates function, but then you'd be calling initial function with some argument like this:
f_x = f(x)()
f_y = f(y)()its just abusing a core mechanic.
a=[]
a.append(a)
When a recursive call happens, it aborts the top-level call, runs the recursive call directly, memoizes the result, and then re-executes the original call. Hence the warning about this only being usable for pure functions.
EDIT: also I just noticed that the map keys aren't the parameters themselves, but their string representations. So heaven help you if you try this with a data type whose str() method isn't one-to-one.