Python multi-level break and continue
lwn.net
lwn.net
for/while... as label1:
__for/while... as label2:
___break label1
suggestion. It actually seems a good enough idea to implement that. And it's rather concise. I often need to break deep inner loops to outermost and doing it with flags is... strange.
Eg, quoting the text:
> Angelico noted that he has used the Pike programming language, which does have a labeled break. He found that he had used the feature twice in all of the Pike code he has written. Neither of the uses was particularly compelling in his opinion; one was in a quick-and-dirty script and the other is in need of refactoring if he were still working on that project, he said. That was essentially all of the real-world code that appeared in the discussion.
https://grep.app/search?q=break%20%27%5Cw®exp=true&case=t...
A spot check shows that quite a few in your link are used specifically to ensure correct handling of Rust multi-level breaks work syntax, like https://github.com/rust-lang/rust-analyzer/blob/master/crate... , https://github.com/rust-lang/rustfmt/blob/master/tests/sourc... , https://github.com/rust-lang/rust/blob/master/src/tools/rust... , https://github.com/rust-lang/rust/blob/master/src/tools/rust... and likely more.
Another is a translation of BASIC code to Rust, using break as a form of goto. https://github.com/coding-horror/basic-computer-games/blob/e... . The Python version at https://github.com/coding-horror/basic-computer-games/blob/e... doesn't use that approach.
The example at https://github.com/tokio-rs/mio/blob/master/tests/tcp.rs is a nice one
// Wait for our TCP stream to connect
'outer: loop {
poll.poll(&mut events, None).unwrap();
for event in events.iter() {
if event.token() == Token(1) && event.is_writable() {
break 'outer;
}
}
}
def find_writable_event(events):
for event in events:
if event.token() == Token(1) and event.is_writable():
return True
while 1:
poll.poll(events, None)
if find_writable_event(events):
break
There are a ton of Python features (and misfeatures) that I'm sure most Python devs never use. __subclasses__ for example, or sitecustomize.py.
Similarly I'm pretty sure Rust's i128 is almost never used but it would be really weird to omit it.
Hence my "Could you give a real-world example of how it proved useful?"
Convincing examples might change their minds!
While simply saying "here's how to find Rust programs which use that construct" is not informative nor convincing.
With __subclasses__ there are definite real-world examples where they are useful. We can see the motivation in the commit history:
commit 1c45073aba8f4097b4dc74d4fd3b2f5ed5e5ea9b
Author: Guido van Rossum <guido@python.org>
Date: Mon Oct 8 15:18:27 2001 +0000
Keep track of a type's subclasses (subtypes), in tp_subclasses, which
is a list of weak references to types (new-style classes). Make this
accessible to Python as the function __subclasses__ which returns a
list of types -- we don't want Python programmers to be able to
manipulate the raw list.
In order to make this possible, I also had to add weak reference
support to type objects.
This will eventually be used together with a trap on attribute
assignment for dynamic classes for a major speed-up without losing the
dynamic properties of types: when a __foo__ method is added to a
class, the class and all its subclasses will get an appropriate tp_foo
slot function.
Author: Victor Stinner <victor.stinner@gmail.com>
Date: Fri Mar 25 17:36:33 2016 +0100
...
* Use __subclasses__() to get resource classes instead of using an hardcoded
list (2 shutil resources were missinged in the list!)
...
I've never used 128-bit integers, but its seems many people do have real-world cases for it.
What do you think is a convincing real-world example that should motivate its inclusion in Python?
That Rust example is a decent one. Also the various "find something in a nested structure" examples people have posted.
(And the fact that you can achieve the same result in a more awkward way by putting it in a function and using `return` is irrelevant because there are many many language features that are just convenient sugar: +=, lambdas, even while loops!)
The use of a helper-function is given in the lwn article as a reason for not supporting multi-level break:
] The solution to "Python needs a way to jump out of a chunk of code" is usually to put the chunk of code into a function, then return out of it.
More specifically:
] To make this proposal convincing, we need a realistic example of an algorithm that uses it, and that example needs to be significantly more readable and maintainable than the refactorings into functions, or the use of try…except (also a localised goto).
] If you intend to continue to push this idea, I strongly suggest you look at prior art: find languages which have added this capability, and see why they added it.
The multi-level break in that Rust example may be "more readable and maintainable than the refactorings into functions", but it is not "significantly more readable."
> The solution to "Python needs a way to increment variables" is usually to so `a = a + 1`.
> To make this proposal convincing, we need a realistic example of an algorithm that uses it, and that example needs to be significantly more readable and maintainable than `a = a + 1`.
Why does `+=` get into the language but labelled breaks (a reasonably standard feature) don't. This isn't hypothetical - not all languages have +=. Matlab for example requires you to do `a = a + 1`.
> The idea behind augmented assignment in Python is that it isn’t just an easier way to write the common practice of storing the result of a binary operation in its left-hand operand, but also a way for the left-hand operand in question to know that it should operate on itself, rather than creating a modified copy of itself.
Here's an example of how "a += b" is not syntactic sugar for "a = a + b". First, "a = a + b", which rebinds 'a' to a new object while leaving 'b' bound to the original:
>>> a = b = [9,8]
>>> a + b
[9, 8, 9, 8]
>>> a = a + b
>>> a
[9, 8, 9, 8]
>>> b
[9, 8]
>>> a = b = [9,8]
>>> a += b
>>> a
[9, 8, 9, 8]
>>> b
[9, 8, 9, 8]It was meant to show your comment at https://news.ycombinator.com/item?id=32741165 wasn't a relevant parallel, because it ignored how "a+=1" and "a=a+1" are different.
For a more convincing use, consider NumPy arrays, where a+=1 re-use the same (potentially very large) array, while a=a+1 creates a new array.
import numpy
a = b = numpy.array([[1,2], [3, 9]])
a += 1 # modify in-place
a = a + 1 # create a new array
>>> a
array([[ 3, 4],
[ 5, 11]])
>>> b
array([[ 2, 3],
[ 4, 10]])
> To make this proposal convincing, we need a realistic example of an algorithm that uses it, and that example needs to be significantly more readable and maintainable than the refactorings into functions, or the use of try…except (also a localised goto).
How about this:
found = None
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
if search_query.matches(cell.value):
found = (r, c)
break 2
logger.log("Found cell: {}".format(found))
return found def find_cell(table):
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
if search_query.matches(cell.value):
return (r, c)
found = find_cell(table)
logger.log("Found cell: {}".format(found)) def searchRow(row,r):
found = None
for c, cell in enumerate(row):
if cell.value == expected:
found = (r, c)
break
return found
found = None
for r, row in enumerate(self.rows):
found = searchRow(row, r)
if found is not None:
break
logger.log("Found item: {}".format(found))
return found def _find_cell(table, search_query):
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
if search_query.matches(cell.value):
return (r, c)
return None
found = _find_cell(table, search_query)
logger.log("Found cell: {}".format(found))
return foundIt may be worth observing Gouvernathor does not exactly write "factored-out" code like this either: https://github.com/Gouvernathor/renpy-ParliamentDiagram/blob...
> that example needs to be significantly more readable and maintainable
Yes, things can be slightly tidier with labelled breaks in some contexts. But that’s not a particularly high bar. There are tons of language features for which that’s true. Adding everyone’s pet nicety to the language makes it a huge language, which isn’t what they’re aiming for. If that’s what you want, go use Perl or C#, both of which are very fine languages.
I might risk going on a rant here, but if the improvement from break-label is insignificant enough that it wouldn't meet that bar, I'm not sure how a ton of other features already in the language ever met it. Did for-else and while-else clauses (which somehow pretty much every other language on earth gets by without) genuinely meet this bar? How about the (pitfall-ridden!) match-case and walrus operator that they added quite recently? Did assignments-in-conditionals suddenly meet this bar in 2019 but not meet it in the decades prior where they refused to consider it compelling enough? What about annotations—couldn't you already do f = annotate(f)?
You are unimpressed with various marginal features that have been added but still think this other marginal feature is the special one that should be added.
In other words, the "but everyone else jumped off the bridge" argument is not especially compelling.
Assignment expressions passed basically because there are around 2 common places where they clean up code. But unlike labeled break, those cases are relatively common. I dislike the walrus operator and don't think it was worth adding to the language, but I've still run into cases where it's been valuable. That's never been true for labelled break.
Added cruft? I don't think so.
The second version without the break is imho a clear improvement! It's pure functional code (ignoring the logging).
Compared to the imperative code which uses mutation of state defined outside of the function, which makes the inner workings of this algo even harder to follow, it's an obvious improvement.
Line count stays the same. Therefore I would in any case prefer the second, refactored version.
There may be cases where a multi-level break or even a goto would yield a better solution, sure. But those cases are rare and likely not present in Python at all as you do such things usually only to avoid a few machine code instructions; something that is irrelevant in Python as it's anyway slow, no matter what you do, so such optimizations would not make any sense at all.
all_matches = (
(r, c)
for r, row in enumerate(table.rows)
for c, cell in enumerate(row)
if search_query.matches(cell.value)
)
found = next(all_matches, None)
if found:
logger.log("Found cell: {}".format(found))
return found
But if you really need this to work on an 2-d array that is only an Iterable, and somehow isn't Sized or Sequence, this still works:
def flatten(table):
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
yield (r, c, row, cell)
def query(table):
for (r, c, row, cell) in flatten(table):
if search_query.matches(cell.value):
return (r, c)
(2) You probably don't want to call it something generic like "flatten" if it's going to do something so specific like concatenate table.rows. Note that I'm not just nitpicking on the name here. Rather, my bigger point is your factored-out functions aren't really as reusable as your name suggests. In this particular case you can trivially get around it by passing table.rows directly, but that obviously wouldn't generalize if you needed anything else in the table in the process.
> This process is generalizable
(3) I'd question that too. Imagine if you had a per-row operation too. Like maybe:
found = None
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
if search_query.matches(cell.value):
found = (r, c)
break 2
logger.log("Finished row #{}".format(r + 1))
def flatten(table):
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
yield (r, c, row, cell)
logger.log("Finished row #{}".format(r + 1))(2) sure, but this doesn't change anything. Factoring a big ugly function out into two smaller easy functions is...fine, even if each is only used once. The name I picked wasn't to imply generalization, it was because I needed a name.
(3) yes.
But again the whole reason this is unreadable at all is because you've created an intentionally obtuse api. We're returning the row and column, so clearly either were able to index into this thing normally, or it's a temp we've created, in which case we can modify it's api and index into it normally.
Do you never consume APIs you have no control over? And is every sequence you use an indexable list?
And no, there was nothing intentionally obtuse in my comment. It was incredibly straightforward and readable code. We obviously disagree, but regardless, I don't enjoy the bad-faith accusations, so I'll stop here.
I already explained this. You're returning an index. There's no reason to do that unless.you can use the index. Somewhere, the api allowed you to use the index, so in this case, you've got control over the relevant parts of the api that don't!
> And is every sequence you use an indexable list?
Yes, or it's small enough that I can store or transform it into one, or it's big enough that I need to use specialized tooling that doesn't play well with break or continue.
> It was incredibly straightforward and readable code
I agree, there's nothing wrong with the code as written. It's fine. The obtuseness was the example. I'll reiterate: you're returning an index into some table. What are you going to do with that index, as you're claiming the table you pulled that index from cannot be indexed into for some reason? The code is fine, the example is obtuse.
Python has itertools.product and unpacking for this case:
from itertools import product
space = product(enumerate(table.rows), enumerate(row))
for (r, row), (c, cell) in space:
if search_query.matches(cell.value):
logger.info("Found cell: (%s,%s)", r, c)
return (r, c)
I think the Python team is having to do the difficult job of filtering new feature, and it makes sense they don't introduce it in this case. We already have all the facilities in the language to deal with most nested loop cases, and those are rare cases anyway.
For this case, one should use a nested comprehension:
space = ((r, c, cell) for r, row in enumerate(table.rows) for c, cell in enumerate(row))
for (r, c, cell) in space:
if search_query.matches(cell.value):
logger.info("Found cell: (%s,%s)", r, c)
return (r, c)
In all languages, you separate code for preparing your input and code for processing it. In python, it's also a good idea to do that at the loop level:
- prepare your stream of data so that it's normalized using itertools, generators, a function, a numpy array, etc
- do a single loop pass for your processing
This way you will you obtain a single iterable, and will be able to get the most out of the language tooling:
- you can pass it to map/filter, itertools or another comprehension
- you can convert to a list, a tuble, a Counter, etc. And slice it
- you can use "else" in the loop, which is an underused construct, but very useful when you need break
- you can use advanced unpacking
- you can call iter() and/or next(). This is what masklinn does in his example.
The point being, nested loops are rare enough, and we have tool to deal with them in an idiomatic way in Python, and not adding new syntax for a rare case that usually has a good solution makes sense to me.
found = next((
(r, c)
for r, row in enumerate(table.rows):
for c, cell in enumerate(row):
if search_query.matches(cell.value)
), None)
[0] well map + itertools.concat, IIRC the stdlib doesn't have a flatmap per-se, though more_itertools probably does.
found = findIt(search_query, table.rows, logger)
def findIt(query, rows, logger):
for r, row in enumerate(rows):
for c, cell in enumerate(row):
if query.matches(cell.value):
logger.log("Found cell: {}".format(found))
return (r, c)
break 2
return None found = next((
(r, c)
for r, row in enumerate(rows)
for c, cell in enumerate(row)
if search_query.matches(cell.value)
), None)I think early exit constructs improve readability and python already offers a lot of syntax open to abuse at this point. (things you can write with nested list/dict comprehensions with if conditions/expressions scattered..)
To dig deeper into why goto labels are bad and the alternative, what goto labels allow you to do is to create a state machine in procedural code. The more explicit way to do this is to create a dictionary object that tracks the state and a while loop with state-based logic that includes a halting condition. But according to Python Millionaire this would not be “Pythonic” which is basically just a blanket term for saying it is built into the language and therefore designed to be simple and easy to read. However, I would argue that goto statements are inherently un-Pythonic because they encourage spaghetti code. I don’t see how for/else or try/except is not Pythonic, but also I don’t see the aversion to refactoring to separate function which is the accepted cross-language way to refactor and simplify code. It even lets you write a built-in comment for what the function does, the function name. That being said, an easy way to determine if the function failed is to either throw and exception or simply return a result like True/False which would determine whether to break out of the parent loop.
https://lwn.net/Articles/907510/
With something like: For loop... As my_loop: my_loop.continue()
from easytowritelibrary import LoopManager
lm = LoopManager()
for x in lm(a1):
for y in lm(a2):
if cond:
lm.break_(2); break
block()
for x in lm(a1, "a1"):
for y in lm(a2, "a2"):
if cond1:
lm.break_("a1"); break
if cond2:
lm.continue_("a1"); break
block()
for i = 1 to 50
for j = 1 to 50
if i * i + j * j * j = 80036
break 2
end
end
end
print i & " " & jThat seems strictly worse than labelled breaks. What are the upsides?
Genuinely useful feature that is awkward to emulate? Nah.
Let's face it, everyone writes a double or triple nested for loop every now and then.
Python is already expressive enough that unlabeled break and continue are usually the wrong choice.
Citation needed. You can of course write code so they are not needed, but it won't necessarily be clearer, more efficient or "better" in any way.
That sometimes they facilitate spaghetti code is no reason no ban them.
If efficiency truly matters, you should not be using break or continue, you should be using a vectorized processing library like numpy or pandas.
The pattern I describe in https://news.ycombinator.com/item?id=32710822 works for continue, and the same general concept works for break (and note that I've made this as unappealing as possible for my refactor, and it's still not really worse, but could easily be made even better):
for i in x:
if cond(i):
break
return do_a_thing(x)
def find(x):
for i in x:
if cond(i):
return i
do_a_thing(find(x))
Like break and return from a loop are equivalent modulo implicit state, and the same is true for continue and a conditional generator.
Not all loops ate numeric, in fact most aren't (since vectorisable stuff already is vectorised).
Being explicit about which state needs to be passed is almost universally better than having implicit state in a large complex function. If you're dealing with a complicated function tracking a large amount of state, separating the state needed to iterate over the loop, and the state needed to validate whatever condition you're checking is less complex than keeping it all in one larger function. The number of potential interactions grow exponentially in the number of local variables. Less state (fewer locals) means exponentially less complexity.
When dealing with very small numbers of locals, the additional confusion from a second function might win out, but any situation where you're forced to pass state between the loop and the conditional is going to be over the threshold where multiple functions are easier for most people to follow.
And that's just stateful interactions, the functions I've written have precisely one entrance and exit point, and precisely one path through them, while break and continue create opportunities for more complicated control flow.
You're arguing that a function with more local state and more complex control structures should be simpler in some cases, but I don't see that happening as the state and control structures get even more complex. This is of course made obvious by the fact that the first code example I posted contains a bug, while the second avoids it.
I find these things are not universal. Passing state around can be very confusing. Not on trend with everything being "pure" and functional but true pragmatically speaking.
Certainly compared to walrus operator or f strings, labelled break/continue operations are very valid tools, which has different tradeoffs to alternative ways of emulating it, so sometimes will be better.
I'm not arguing anything about your code, I've only provided my own code examples. There's no reason for that to be patronizing.
My point was and is that labelled break and continue are almost always a code smell, and would be banned by any good style guide, much like c++ has many misfeatures that are banned by most style guides. You may not be most people, break and continue may be clearer to you, but that's not a good reason to add a feature to a language. Not every language needs to be perfectly catered to your needs. Certainly not every language should (or needs) to cater to all of my needs, and again there's nothing patronizing about expressing that.
Adding a misfeature doesn't make much sense, and you've done nothing to demonstrate that labelled break and continue are useful or make code clearer. Nor even that unlabelled break and continue are particularly valuable. Instead you've claimed I'm insulting you.
"Break and continue are almost always the wrong choice" is not, and indeed dissing one the oldest and most retained features in programming language history requires substantial justification.
Exactly; at least for me the other features you mention I use daily (or when considering coding hours only, multiple times per hour), whereas I cannot even exactly remember when the last time was I had to break out of a nested loop. I'm not saying it's not useful, just that I don't think that code frequency is a useful argument to make here, especially not in comparison with the features you mention.
This feels like a bunch of people projecting their subjective biases on what code should look like on everyone else. Python 2/3 all over again (in a small teacup).
But for those you've got iterools.concat, or generator comprehensions.
for item in itertools.chain.from_iterable(list_of_lists):
# Do something with itemWhat would you have the inner function do?
for x in a1:
for y in a2:
if test(y):
break break # break both outer loops
an application might be scope management parsing -- I tried to build something like this a while ago, not clear that it's better than parser-generators but it's certainly different
from itertools import product
for x, y in product(a1, a2):
if test(y):
break
Why introduce a new syntax when a library does the job? A new syntax is a huge bar to pass for a language, and nested loops are rare.
Itertools is a gem, and any python dev will benefit greatly from mastering it.
If the loops were written as
for y in a2:
if cond:
break
for x in a1:
# something
The link below describes an example of handling errors which arise when reading a collection of files line by line.
https://gigamonkeys.com/book/beyond-exception-handling-condi...
(loop named loop-a do
(loop named loop-b do
(loop named loop-c do
(return-from loop-b)))
(return-from loop-a 42))