Code blocks in Python | Hacker News Reader

Code blocks in Python

mtomassoli.wordpress.com

49 pointsmtomassoli68 comments

This is a terrible idea, and ruby used the wrong implementation. If you want this behaviour, you should just define a higher order function/decorator, and put your "block" in its own function. This is a case where there should be only one way to do it.

I agree that this doesn't fit Python well, but why was it bad in Ruby?

Because of non local returns. People abuse them when instead they should be using an exception, or preprocessing their data better.

Blocks in other languages can help maintain encapsulation, if they can access variables from their local scope. They can also enable programmers to write their own control structures which look like natural first class control structures in the host language. I don't see this as a natural way of doing either.

Nested functions can access variables from local scope in Python, so this isn't a huge issue. It's a bit wonky in python (see discussion of the nonlocal keyword, Python did this wrong from the start) but it solves your encapsulation problem.

    def sorted_by(xs, attr):
        def cmp_attr(a, b):
            return cmp(getattr(a, attr), getattr(b, attr))
        return sorted(xs, cmp_attr)

It takes a little more vertical space and requires you to give a name to something that might otherwise be anonymous, but in the long term it's actually beneficial in my opinion. I've actually taken to using the same pattern in my JavaScript development, giving all of my callbacks names, because it makes maintenance so much easier later.

    var load_into = function (url, elem) {
        var handle_response = function (response) {
            $(elem).html(response);
        };

        $.get(url, handle_response);
    };

That's not bad. I agree named functions are fine for many solutions for maintaining encapsulation. They're not that great for writing your own control structures. How would you implement my ifTrue:ifFalse:ifMaybe: example?

Seems like a good way to make your code harder to read and harder to debug.

Anonymous functions can generally be abused in ways that make code harder to read. Then again, so can recursion. It's a double edged sword.

Can someone give me an example where using a code block is better than using a simple function? And is the added complexity and loss of readability worth it?

If the code block can run in a different thread (like in Grand Central Dispatch) it is very useful to do stuff in the background without blocking the GUI (that executes in a specific thread) and using the current scope variables. A progress bar is a clear example of this.

Doing this without code blocks adds extra code like launching another thread, connecting the scope to the new thread.

Sorry, I'm not following. Why couldn't you use a function here instead of a code block to do the exact same thing?

I'm not sure there is one. They're useful in C# as a way of defining a function within another function (which sometimes makes for more readable code). For example:

    public int CountRelevantItems(IEnumerable<Thing> things)
    {
      // Non-trivial filter that you don't want in a where lamdba
      Func<Thing, bool> isRelevant = (t) =>
      {
        ...
      }
      
      return things.Where(t => isRelevant(t)).Count()
    }

But in Python you can already define functions inside of functions so you have a better solution:

    def CountRelevantThings(things):
      def isRelevant(thing):
        ...
      return len(filter(isRelevant, things))

> They're useful in C# as a way of defining a function within another function (which sometimes makes for more readable code)

Why wouldn't you just create another function in the same class?

Just personal preference really, if your "isRelevant" filter was only ever going to be used by your "CountRelevantItems" method you might consider it more readable to have the logic of isRelevant right there in the CountRelevantItems method rather than elsewhere in the class.

Some people may consider:

    def is_prime(n):
        if n % 2 == 0:
            return False

        sqrt_n = int(math.floor(math.sqrt(n)))
        for i in range(3, sqrt_n + 1, 2):
            if n % i == 0:
                return False
        return True

    def main():
        with concurrent.futures.ProcessPoolExecutor() as executor:
            for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
                print('%d is prime: %s' % (number, prime))

uglier than (borrowing some JavaScript syntax):

    def main():
        with concurrent.futures.ProcessPoolExecutor() as executor:
            for number, prime in zip(PRIMES, executor.map(
                    function(n) {
                        if n % 2 == 0:
                            return False

                        sqrt_n = int(math.floor(math.sqrt(n)))
                        for i in range(3, sqrt_n + 1, 2):
                            if n % i == 0:
                                return False
                        return True
                    }
                    , PRIMES)):

                print('%d is prime: %s' % (number, prime))

I don't, mostly because in the second example, I cannot unit test the anon version of is_prime in isolation. This is a rather trivial case, but I've seen much more complex and enigmatic anon functions in Node applications.

Note: the original comes from http://docs.python.org/dev/library/concurrent.futures.html#p...

"Some people may consider: [example A] uglier than: [example B]."

Do people really think that? I ask in all seriousness. I am a hardware developer by profession and don't often dive very deep in software, though I love exploring software space.

I find example A to be very clear and readable. I like to look at things in bite sized chunks.

I also like to create things in bite sized chunks. In hardware design, specifically digital chip design, I will know what my overall problem is to solve (aka the requirements), but I don't always have a clear view of what pieces I need to get there.

In the cases where I'm not sure about the smaller pieces, I'll write the larger overall structure and when I get to a spot where I'll need a block of smaller functionality, I'll just make up a function name on the spot and pretend it's already written and does the right thing.

Then I take a second pass, implementing those smaller blocks, and keep going down. I guess that's a typical top-down approach? I don't know, but it is what I do.

But I also do bottom up. Even without knowing all the steps between, I'll often have some idea that there are some basic building blocks I'm going to need, and I'll implement those.

So I iterate with some steps top-down, others bottom-up, until I meet in the middle.

All that is a long winded way for me to say that your example A fits both approaches for me. I can start with the overall structure and implement "main" first, or know what my basis will need and start with "is_prime" first.

Whereas example B seems like an all at once approach that is both more difficult to write and read. At least, that is how it seems to me.

edit: wording

> Do people really think that?

People have been investing a lot of time in this. That certainly indicates someone considers it a worthwhile endeavor.

I think that "readability" also depends on what you're used to reading and writing. To me, my code blocks are quite readable, but that's only true because I've seen a lot of them in these last two weeks.

Code blocks let you write

  with myfunc(iter) << 'x, y':
    code which uses x and y

instead of

  def some_func_name(x, y):
    code which uses x and y
  myfunc(iter, some_func_name)

If you prefer the second form, then you don't need code blocks. I think the first form is much better. It's not just a matter of saving one single line: you don't have to define a function which you'll never use again, so the first code is actually more readable.

    "Here is what my DSL can look like with Blocks in Smalltalk"
    (condition)
        ifTrue: [ ... ]
        ifFalse: [ ... ]
        ifMaybe: [ ... ]

You can accomplish the same thing with named functions and function pointers, but it makes the DSL very awkward to use. Imagine if you had to write all of your conditional logic (if-then and if-then-else and switch) using the names of functions defined elsewhere. That would be much less readable than being able to write block of code inline.

Surely that depends on how well you name the functions?

If by that, you mean that the intention revealing name of the function is always as good as being able to read their source code, then I'm a bit dubious. That's a little better than assuming the comments are never outdated or outright lies. The thing is, naming really well is often hard. By requiring so many more names, I think you've added a whole lot of workload to the developer.

In the right environment, that might actually encourage developers to write short methods. In the wrong one, it might encourage longer ones. It would discourage me from writing DSL control structures.

Gedankenexperiment. Imagine this: An IDE that quickly lets you browse the code for a function by pressing a key, or hovering over its name invoked in code. Some smart cookie then comes up with an add-on that provides a switchable view with in-lines for the function code, as if in a code block.

I'd like to see some before-and-after examples of Python code that's been rewritten to take advantage of blocks using this module.

Interesting, but I'm having a hard time seeing how this syntax saves me much in terms of code size and complexity compared to just wrapping the code blocks into named functions and passing those.

The problem with that is that it forces moving the code above where it is passed, which can greatly clutter things.

For example, consider the function "forever" in Haskell:

  forever action = do
    action
    forever action

Now you can use it as a new control structure, e.g:

  forever $ do
    (sock, addr) <- accept listener
    forkIO $ handleClient sock

The $ simply means "apply" and is low-precedence, so it removes the need to put () around the entire argument to be applied.

In Python, you could define:

  def forever(action):
    action()
    forever(action)

but then, to use it, you have to give a name to your function, so:

  def accept_once():
    (sock, addr) = listener.accept()
    fork(partial(handleClient, sock))
  forever(accept_once)

This makes "forever" much less useful as a new control structure/looping primitive.

In this sense, Python makes DSLs less usable. The built-in primitives are first-class, and can have code directly within their use. Library functions are second-class, and can only have code passed by name which must then fully appear before the use.

Another example is callbacks. The reason "twisted" is probably called "twisted", and that people hate callbacks so much, is that it forces writing the code in backwards order, precisely because of this problem. For example:

  def handle_result(result):
    print "Done:", result
  def connection_started(conn):
    conn.request(SomeRequest(), handle_result)
  def start():
    start_connecting(connection_started)

Compare this with Haskell, as an example:

  start = startConnecting $ \conn -> do
    request conn SomeRequest $ \result -> do
      putStrLn $ "Done: " ++ show result

Note, in Haskell, this would actually be worked out to be (by overloading the semicolon):

  start = do
    conn <- startConnecting
    result <- request conn SomeRequest
    putStrLn $ "Done: " ++ show result

But even the former nested representation is better than the backwards ("twisted") representation that makes people hate callbacks so much.

This seems like a solution in search of a problem. There is a nice 'forever' construct. It's called 'while':

    while True:
      action()

I know what you mean about generalising this to other control structures, but it looks like the whole concept starts from "I want to write 'forever' that resembles something from other languages", rather than "I want to write a loop". There are existing ways to write things like that, so is do we really need to force something from other languages into Python? What about some examples which cannot be easily handled - maybe the solution for them is something completely different than porting codeblocks.

You are missing the point. The forever structure is just a code example. What's missing is the ability to write your own control structures generally. (Some of which won't already exist.)

That is exactly what I meant - this is a synthetic example. The article has synthetic examples. People want code blocks, but no one is really showing why. Own control structures are cool, ok, but what are you trying to solve with them? Where are the real, convincing before and after examples that are not easily solvable otherwise?

Here's some Haskell code I really like, and I think it would be far less nice in Python:

  replicateM 10 . forkIO . forever $ do
    ..

Which is basically the same as:

  replicateM 10 (forkIO (forever (do ..)))

replicateM 10 executes its code block argument 10 times. forkIO executes its code block argument in a new thread. forever loops forever.

So this line basically creates a thread pool with 10 threads, all infinitely executing the given code block.

How would you solve this in Python?

> How would you solve this in Python?

The question is how does codeblocks solve this problem in Python, not how does Haskell solve this problem in Haskell.

Hypothetical Python:

  times(10, forkIO(forever(block:
     ...
     )))

That's not what codeblocks does, though. You use a with statement and a new operator (<<). Then use indentation for the blocks. It doesn't look like you're passing a block to a function. The result is quirky code that will be hard to trace and sort out if you don't already know what's happening.

Oh, I thought you meant code blocks in general, not the particular OP "codeblocks"... Sorry.

I'm not sure that codeblocks actually solves the problem.

I can honestly say that the ability to write your own control structures is one of the quickest ways to make me hate developing in a particular language. Any time you use them, your code becomes significantly less readable, and significantly less portable.

It all just becomes a giant clusterfuck way too quickly. Small set of well-defined semantics, please.

Any time you use them, your code becomes significantly less readable, and significantly less portable.

Specific examples?

I've actually often found the opposite to be true, given a good (meta) syntax for handling this.

I've often seen patterns in Smalltalk like:

    myTransaction 
        commit: [
            "Stuff happens here"
        ]
        onRollback: [   
            "handles rollback"
        ].

Semaphores are used for critical sections like so:

    (mySemaphore)
        critical: [
            "critical section code goes here"
        ].

I think these examples show how such a thing can increase readability.

As far as the portability thing goes, I've never heard of that being a terrible problem in Smalltalk. If code from one environment has some custom method like "isNilOrFalse:" or something like "ifNotNilDo:" then it's a simple matter to port that code over another environment, or have it syntactically rewritten by the browser to make a portable library. The sorts of patterns I list above are generally for DSLs that are only used internally in a given library or for code that uses the particular library, so portability isn't a problem. The structures are carried where needed by the library.

It all just becomes a giant clusterfuck way too quickly. Small set of well-defined semantics, please.

Smalltalk does have a small set of well-defined semantics, just not at the level you're used to. I have never heard of modifications that break the control flow standard library. My experience is that good programmers don't change how the "standard" behaviors work, because those changes tend to break the rest of the code base.

But you don't have to write your callbacks in reverse order in python:

  >>> def fight(batman_wins):
  ...     def t(): return "pow"
  ...     def f(): return "oof"
  ...     def b(msg): print msg
  ...     # if_but not defined at this point
  ...     if_but(lambda: batman_wins, t, f, b)
  ...
  >>> def if_but(cond, true, false, but):
  ...     if cond():
  ...       v = true()
  ...     else:
  ...       v = false()
  ...     but(v)
  ...
  >>> fight(True)
  pow
  >>> fight(False)
  oof

Sure, if you want to call if_but outside a function, it has to be defined, but inside a function (as in your example, and in 99% of python development that doesn't occur in a REPL) x() is effectively globals()['x'](). Have I totally misunderstood your point?

This problem is indeed less severe when the names are defined in a non-local namespace (inside a class, or as globals).

But callback-style often appeals to defining very-local callback code. In fact, it would be anonymous in languages that support it, but it is forced to not only be given a name, but also potentially clutter some namespace (or be written in reverse order).

Consider having every for/while loop, or every "if" require giving a name to the code block[s] within it. It sounds insane. The same doesn't sound insane for library functions, but in my opinion that's really just because everybody's used to this limitation.

When I moved from Python to Haskell, one of the many joys was that I could stick an anonymous code block anywhere so easily.

Thanks for the explanation, that makes sense.

I understand anonymous code blocks is something that python doesn't do natively and other languages can do. What I don't understand is why it's worth the bother, given the hoops one must jump through to get it to work in Python specifically (new operators, keywords, and evaluating string literals as code).

If you really need new control structures, Python syntax isn't all that hard to hack.

No string literals are evaluated as code. Everything is rewritten and then evaluated and executed at full speed.

Surely the act of enhancing a function like that, where you don't want the original code to be available, is more commonly dealt with in Python using decorators?

This seem like a whole series of posts on 'how to twist python syntax to save a line of code or two and make it unreadable'.

Adding function literals to Python would be nice. Something to turn "inc = lambda x: x + 1" into "inc = def(x): return x + 1".

But blocks? They're a misfeature of Ruby, badly copied from Smalltalk.

[edit: added "return"]

It looks like you changed "lambda" to "def" and added parentheses. It seems like that feature is already there...

Perhaps my example was crap. They would also allow anything inside them.

The problem with multiline lambdas it that no one knows a good way to indent them. If they weren't butt-ugly python would probably already have them.

"The problem with multiline lambdas in Python is that no one knows a good way to indent them."

How about new keyword(s)? "begin" and "end"? I imagine this has thoroughly been discussed ad-nauseum. Is there a good synopsis?

Basically, there is no scope or block ending token anywhere else in python. And Guido thinks inline multi-line anonymous functions make programs harder to understand.

I'm not sure it's that big a problem. CoffeeScript handles this pretty well using indentation.

CS has ambiguous cases because of optional ( ), but that wouldn't be the case with Python anyway.

I also question the utility and reason behind this. I spend a lot of time fixing memory leaks from lousy developers who use closures and anonymous functions incorrectly in JavaScript. I'm just learning Ruby as well, and see a lot of room for similar black-boxing pitfalls in procs and blocks. It seems to me like you're reducing code size at the expense of readability and debugability. That might be the ruby way, but it seems to me like a bad way to program.

Like I said, I'm just learning... so I'd love counterexamples and reasons why I'm wrong.

This is all well and good, but it doesn't appear to add support for the one thing I've often wanted in Python: proper closure support for functions within functions.

Do you mean 'nonlocal' keyword?

"code blocks" is why they invented functions and procedures half a century ago (and even those differ only in how they return). I think people forget (or don't know) that these are all labels to jump or branch to. So your named functions, functions, lambdas, code blocks, etc, become.. well.. equivalent. Hipsters, don't confuse the young ones with your 30 ways of jumping into a block of code. Cheers!

Not the whole story. Code blocks can be used to protect encapsulation and create DSLs which are first class citizens of their host language.

So your named functions, functions, lambdas, code blocks, etc, become.. well.. equivalent.

They can also become so lexically awkward as to be unusable. For example, one code a toy "fuzzy logic" system in Smalltalk with a handful of methods. The result looks like a 1st class member of the language, just like the control structures that are already there. (Same goes for the loops)

    "Here is the standard if-else"
    (condition)
        ifTrue: [ ... ]
        ifFalse: [ ... ]

    "Here is what my DSL can look like"
    (condition)
        ifTrue: [ ... ]
        ifFalse: [ ... ]
        ifMaybe: [ ... ]

Doing this with named functions is going to scatter code between different functions. It's semantically equivalent, but harder to read. Write anything involved in such a way, and it becomes untenable. Blocks can make such DSLs an order of magnitude more readable.

Hipsters, don't confuse the young ones with your 30 ways of jumping into a block of code. Cheers!

A good heaping fraction of hipsters don't understand what's good about code blocks.

"Code blocks can be used to protect encapsulation". So can functions, which are already in the language.

"Doing this with named functions is going to scatter code between different functions". Please show an example.

"A good heaping fraction of hipsters don't understand what's good about code blocks." That's one way of phrasing it.

The code examples are confusing. First, I think there is a typo in the first two examples; I think < should be <<. But he never actually explains what the semantics of << are until several examples in, and it took me a long time to figure out that the string value 'x, y=3' gets evaled into code. Which is strange, to mix strings as code, when the whole point is to have code be code.

At the very least, the semantics of this need to be explained better. When you give a new code example, you always need to say "And this is the result." Otherwise, I can't close the loop; I have new code with new and unknown semantics, and an unknown result. I need to have a known result to figure out the new semantics. (Much like you can't solve a single equation with two unknowns; you either need to have one unknown, or two equations.)

Just read the docstring of the module. It's very detailed.

As for mixing strings with code, I still need to adhere to Python's syntax in order to avoid syntax errors. Anyway, syntax errors in strings are caught at "rewriting time". Using words instead of operators would also cause some problems.

Should it be "<< 'x'" rather than "<< 2"?

I don't believe it. Let's try to fix it again... thanks.

You're putting the carriage before the horse. People want to evaluate what it is they're getting into before they download it. If they can't understand the code examples, they won't get as far as downloading the module and looking at the docstring.

I understand why you had to use strings as code. I had two concerns about that. One, you didn't explain it - I had to figure it out on my own. And two, if a solution is supposed to make things cleaner, but ends up introducing warts like that... maybe it's not worth it. What you implemented is interesting, no doubt, but I wouldn't want to use it for that reason.

However, I think you would benefit greatly from having improved examples. Even if people don't use your module, they can still build on your ideas if it's well explained.

People want...? codeblocks is free and I don't get paid for it. If someone is so lazy that he won't even have a look at the doc in the code (as suggested in my article), I don't want to have anything to do with him or her.

1 click --> bitbucket

1 click --> source

1 click --> codeblocks.py

It's that too much to ask?

shrug

I'm assuming that you want people to use your work. People need convincing. There are thousands and thousands of ideas and projects out there. The ones that are well explained are the ones that will get attention. I do research for a living. The research itself is only half the job. Presenting the research - convincing people that what I did is important and useful - is the other half.

Basically, what is your objective: do you want to be right, or do you want to be heard? You said you welcome constructive criticism. That is what I'm trying to provide - except it's not about your work itself, but about better ways to present it.

I do research for the heck of it and if other people think that what I'm doing is useless, I'm ok with that. That's the mathematician way. The exploration is the end, not the mean.

I'm not trying to convince anyone that my "project" is useful because that's not my project anymore. Anyone can look at it, dissect it, propose new feature, etc... Now that the thrill of the exploration is almost over, I'm losing interest... but I'll keep an eye on my repository on bitbucket.

By the way, if someone could provide better examples, documentation, etc... I'd be grateful. Let's just say I'm not much into the "convincing thing". If you need convincing, don't look at me. I gave you an object. Now it's your job to figure out what to do with it (or just toss it away) ;)

By the way, I wish that '<' was a simple typo. Unfortunately it's Wordpress: it's driving me mad. Everytime some code has '<' and '>' in it, something unexpected happens.

Adding blocks to Python feels like adding classes to JavaScript... Do we really need them?

No. You can always use "named blocks", but they are called functions.

I did something very similar 3 years ago: http://billmill.org/multi_line_lambdas.html

Oh, I know. My module is your fault! :)

I really like the << x syntax, it's devious :)

Reply on news.ycombinator.com