Python 3.6.0 released
mail.python.org
mail.python.org
"My name is {} and I am {} years old".format(name, age)
"My name is %s and I am %s years old" % (name, age)
"My name is {name} and I am {age} years old".format(name=name, age=age)
f"My name is {name} and I am {age} years old."Yes, I know you can do more with format strings than plain interpolation, but that's all that the basic examples show.
(Also: Perl has allowed underscores in number literals for ages).
Python just got one more way of doing string interpolation.
This is a textbook example of string interpolation.
You can look it up on wikipedia. I am not sure what you mean.
The string "My name is {name} and I am {age} years old" is a plain string. By itself, there is no interpolation:
>>> x = "My name is {name} and I am {age} years old"
>>> x
'My name is {name} and I am {age} years old'
Contrast with the proposal for f-strings:
>>> x = f"My name is {name} and I am {age} years old"
# this is an error unless name and age are defined
> It should be noted that an f-string is really an expression evaluated at run time, not a constant value.
But when you do: "{foo} {bar}" % {"foo": "blah", "bar": "blooh"}, this is string interpolation.
Is there a definition, anywhere(even a book would suffice), where it has your strict definition of what interpolation is?
I'm not trying to undermine the usefulness of the interpolated expressions and what you said. I just really want to know if the definition is so strict.
UPDATE: http://rosettacode.org/wiki/String_interpolation_(included)
I think that you are the one who's making the definition so specific and strict. I can't find a single source on the Internet that agrees with your definition.
string interpolation is the process of evaluating a string literal containing
one or more placeholders, yielding a result in which the placeholders are
replaced with their corresponding values.
A few sentences later, the article includes this statement:
Some languages do not offer string interpolation, instead offering a standard
function where one parameter is the printf format string, and other(s) provide
the values for each placeholder.
Also notice how the statement begins by saying that not all languages offer string interpolation. I think that's the final nail in the coffin of the argument. You could use virtually any language to write a function that handles substitutions in template strings. If that qualified as string interpolation then every language would have it. Since some languages do not, I think we can safely say that we're talking about a syntax feature.
See: list comprehensions obsoleting most uses of map() and filter(), then generator expressions obsoleting any uses of list comprehensions that don't need a list object
Here's four ways to do the same thing.
Naïve way:
foo = []
for x in bar:
if meets_some_condition(x):
foo.append(transform(x))
do_something_with(foo)
do_something_with(map(transform, filter(meets_some_condition, bar)))
do_something_with([transform(x) for x in bar if meets_some_condition(x)])
do_something_with(transform(x) for x in bar if meets_some_condition(x))
Before Python 2.0, either map/filter or the naïve way was the right way (TBH, I'm not familiar with what was considered right in the old days). When Python 2.0 came out, list comprehensions became the right way. When Python 2.4 came out, generator expressions became the right way. The right way depends on your version of Python.
Or for that matter, look at old-style classes vs. new-style classes. When new-style classes came out, they immediately became the one and only right way to do classes, but old style classes still stuck around for the rest of the 2.x series.
An implicit coding style that requires the reader to look up the semantics of Python's comprehensions (and also makes it harder to unwind/inject debugging/keep line lengths reasonable) is a negative unless you're gaining something besides brevity.
I understand some claim that comprehensions are faster for various reasons, but this is presumably implementation dependent and shouldn't be assumed unless you know it's the case in your implementation. I also understand that the generator expression uses less memory, but the generator version can be written the long way too:
def test_bar(bar):
for foo in bar:
if meets_some_condition(foo):
yield transform(foo)
do_something_with(test_bar(bar))
But, I agree with you regarding creeping complexity in general - @wrappers/decorators confused me for the longest time, and I often wonder how someone not familiar with the language is supposed to understand what they are doing.
I disagree. For the (common) circumstances where genexp's are appropriate, they are also the most clear and explicit way of expressing intent (explicit loops express the mechanism, not the intent.)
For cases where you need a list object, list comprehension are the most direct expression of intent.
There's cases where explicit looping is the most clear expression of intent; usually, this would be where complex logic is needed within each loop iteration.
--- "Avoid programming languages starting with the letter P except Prolog"
"My name is {name} and I am {age} years old".format(**locals())
Though I think it would be cool if you could have a deferred format string, like so:
template = 'My name is {name} and I am {age} years old. Next year I will be {age + 1}!'
name = 'Amy'
age = 32
formatted = template.apply()
# formatted = 'My name is Amy and I am 32 years old. Next year I will be 33!'
You can kind of do that with [0], but it doesn't evaluate expressions (so the age + 1 thing won't work).
[0] .format(**locals()) # putting it here because HN eats the asterisks if it's not indented def template(name, age):
return f"My name is {name} and I am {age} .."
formatted = template('Amy', 32)
[Edit: I only started learning Python recently, and this is how I'd think of doing it come from JS:
const template = (name, age) => `My name is ${name} and I am ${age}`;
const formatted = template('Amy', 32);
](Conflict between improvement and backward compatibility, but I miss it.)
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
And if you support any earlier version of python, you cannot use f strings. Hence, there is a conflict between improvement and backward compatibility.
Python is no longer inferior to Bash and Perl for short scripts and more concise. Looking across the modern language landscape, this functionality is becoming an industry standard. The format string is also a little more sophisticated that it looks on the surface.
Still, I was a bit down that so many showed up to complain about it, mostly with purity or adversity-builds-character arguments. I'm glad lots of people are enjoying it.
For the example above, this might be fine, but most examples that I see using this kind of thing are things like:
"select * from foo where id = '%s'" or "<input type='hidden' name='foo' value='%s'/>" or "rm -rf /path/to/customer/%s"
I tend to think that in general appending strings is more often than not quite a dangerous operation, and it behooves the caller to need to have a long think about exactly how the strings being substituted in need to be escaped - as it's pretty rare that they don't need to be.
I think you just have to play it by ear. Old formatting styles seem like one of those things you want to leave alone for a really long time, because they're so common that it'd be a major PITA to fix all occurrences, and because it doesn't seem like there's an obvious negative ramification to allowing the old styles to continue to evaluate according to the old methods.
I mean, there’s a precedent for short built-in functions like len(), and apparently 'print "x"' as a statement was terrible so 'print("x")' replaced it. If they wanted such consistency, shouldn’t they have used 'f("{a} {b}")' or something? Or alternately, allow p-strings like 'p"hello, world"' to print?
Other cases such as redirecting to a file, and changing the line ending are handled much more intuitively with the print function.
In fact, I don't think it would be possible to write such a function in the general case, because it's ambiguous what environment you would need to use.
For example:
def call(func, arg):
a = 2
return func(arg)
print(call(f, "{a}"))
Does this print 1 or 2?
One reason might be that f-strings work in expressions, while statements do not. Even with the BDFLs apparent distaste for functional programming constructs, syntax that works within expressions has lots of practical advantages over statements.
>>> 1 + '1'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>> float('inf') < 'hi'
True
I mean that it feels so much nicer to do:
let x = [ 1, 2, 3].map(i => i * 10)
x = list(map((lambda i: i * 10), [ 1, 2, 3 ])) x = [i * 10 for i in [1, 2, 3]] i. let x = [ 1, 2, 3].map(i => i * 10)
ii. x = list(map((lambda i: i * 10), [ 1, 2, 3 ]))
iii. x = [i * 10 for i in [1, 2, 3]]
for i in [1,2,3]:
...It also works for multiple arguments: `_ * _` is equivalent to `(x, y) => x * y`, which is really `(x, y) => x.*(y)`.
In typical perl^Wpython spirit, we now can save a few characters by typing:
f"My name is {name} and I am {age} years old."
"My name is {name} and I am {age} years old.".format(**locals())
Well that's a definite code smell.
The only way `str.format_map(locals())` is a code-smell is if the people working on the code don't actually know Python. Builtin methods of core types, dictionary unpacking, and builtin functions are all extremely common and only complete novices at python would find any issue with them. Compare that to what you need to know when you encounter `f"hello {name}"`:
0) am I working with Python 3.6+?
1) when does replacement happen?
2) what scopes are examined for substitution?
Of course, 2) is a trick question because you EVALUATE ARBITRARY PYTHON EXPRESSIONS IN THE FORMAT STRING![2]
[1] Technically, PEP-0498 states that we're not exposing a full locals() or globals(), but this actually matters very little. The danger here is that both .format() and f"" use object.__format__() (falling back __repr__ IIRC) to figure out the replacement. If your object.__format__() is malicious, it gets access to more data than it needs. However, since python has zero security model, if it's malicious, it pings whatever command and control server it's reporting to and starts up a thread listening for commands anyway (aka you're fucked). Realistically, this means that, you might leak credentials (or other sensitive information) to third party loggers, but only if your method for censoring outgoing log/exception data is a a poorly-built blacklist.
[2] https://www.python.org/dev/peps/pep-0498/#supporting-full-py...
print("My robot's name is {}. It's {} years old.".format(name, years_old))
Yes, both forms are a little longer, but as you state, it's easier to see what's going on. One of Python's biggest attractions, IMO, is that it's batteries included, but not batteries-sealed-under-1000-layers-of-carbonite-locked-by-a-mystical-incantation, like many other dynamic languages. It's usually pretty easy to get in there and see what's happening without having to traverse tons of indirection due to Python's design principle favoring explicit programming styles.
I won't lie and say I've never wished for more conventional style string interpolation, but it certainly wasn't a showstopper, and it's not something we should assume we get for free.
"{} world!".format('hello')
"{greeting} world!".format(greeting='hello')
Second, the parser at compile time pulls out the actual expressions and fills them in to a .format call. It _does not_ use locals!
... and your worries about security are not useful, you could make the same statements about an entire code base.
I meant what I wrote. Someone, in a rush to criticize about a topic on which they are ignorant, didn't consult the Python documentation:
https://docs.python.org/3/library/stdtypes.html#str.format_m...
> they don't show here for some reason.
Asterisks only show in code blocks. They denote italics in HN's markup. Double asterisks outside of a code block begin and end empty italicized text.
** no italics because we're in a code block
I'll quote myself here, since you missed the first part of this sentence when jumping to a conclusion about why security is irrelevant, but we need to blacklist a well-understood and widely used builtin function anyway.
>> Technically, PEP-0498 states that we're not exposing a full locals() or globals(), but this actually matters very little.
But, while we're trying to be pedantic, let's actually be pedantic:
>>> import dis
>>> dis.dis('f"{foo}"')
1 0 LOAD_NAME 0 (foo)
2 FORMAT_VALUE 0
4 RETURN_VALUE
>>> import dis
>>> dis.dis('"{foo}".format("bar")')
1 0 LOAD_CONST 0 ('{foo}')
2 LOAD_ATTR 0 (format)
4 LOAD_CONST 1 ('bar')
6 CALL_FUNCTION 1
8 RETURN_VALUE
It is true that I've never inspected the C code implementation myself, but I took him at his word.
load_name looks like it read foo, then format_value did format('{}', foo).
I didn't realize it before, but it looks like it might have better performance.
I kind of stepped in, rewrote the peepholer, & then initiated the python dev discussion earlier than he wanted to. He wanted to develop the idea & benchmark it in a bubble, whereas I really felt that this should be developed with feedback from core Python developers. Serhiy was very patient with me, there was a lot of feedback it turns out
Fun little issue that thankfully got caught last month: http://bugs.python.org/issue28782 (I had thought about this error case, but at the time I had lasti being -2 & the first byte of the bytecode is never 72. I forgot this constraint when having lasti be -1 still)
The Python interpreter now uses a 16-bit wordcode instead of bytecode which made a number of opcode optimizations possible. (Contributed by Demur Rumed with input and reviews from Serhiy Storchaka and Victor Stinner in issue 26647 and issue 28050.)
Really interesting to see this come to Python. Thanks for mentioning it.
>>> 1_000_000_000_000_000
1000000000000000
>>> 0x_FF_FF_FF_FF
4294967295
>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
>>> f'{"fstring"} and {{deferred}}'.format(deferred=1234)
'fstring and 1234'But surely i18n/deferred interpolation is a corner case, not the norm. If you need that then use what you are currently using, nothings changed.
[1]
Python 3.5.2 (default, Nov 7 2016, 11:31:36)
[GCC 6.2.1 20160830] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>I'm disappointed that there's so many ways to do string formatting/templating, but I think we're all so very painfully aware of how important backward compatibility is. So for the language to gain features like this, I think it should be understood that it will also keep the old ways.
No. Readability counts. One (and only one) point from the Zen of Python does not make an argument against it.
If there's four different ways to something, then that's four different things the reader has to fully understand in order to parse someone else's code.
Sure, which is why .format() kind of sucks.
> "Hi {name}, your age is {age}".format(name=name, age=age)
You have to parse 'name' and 'age' three times to figure out whats going on. The .format() is completely superfluous in most cases. It's busywork and it's harder to parse.
I quite agree with the "one obvious way" to do something, but that should not hinder progress and improvement. f strings are an improvement and are the obvious way to do things, in the same way that '.format' was an improvement over 'str % values', and how 'str % values' is an improvement over manual concatenation.
They all still exist, but lets use f strings now and stop complaining?
Worse still, I will also need to know how they interact with each other - you can do
something = "abc"
x = f"{something}".format(something="def")
So yeah, I would love to "use f strings and stop complaining", but the fact that the other string interpolation methods still exist, and are still supported, is still going to be a problem.
It might help to realize the f-string is just syntactic sugar for a .format() call.
>>> formatted = str.format(template, var1='a', var2=3)
I suspect there is a strong correlation between programmers that want underscores in numeric literals, and programmers that live and breath hex.
> PEP 484 introduced the standard for type annotations of function parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating the types of variables including class variables and instance variables:
primes: List[int] = []
captain: str # Note: no initial value!
class Starship:
stats: Dict[str, int] = {}The difference here is the ability to add type annotations to variables without needing to use a comment to declare it.
I'm also pretty excited for the formatted string literals and secrets module!
https://www.python.org/dev/peps/pep-0484/
> It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.
The crashes I got at runtime aren't because of the typing module preventing code from running due to types, it was just a bug in the runtime part of typing. I'll try to find the github issue to share, but iirc it was related to using Generics with a TypeVar.
captain: str # Note: no initial value!
That is not quite true, the first assignment is an implicit declaration, and will set up a slot in the enclosing code object, which is why you get an UnboundLocalError if you use a name before it's first assigned to:
>>> def foo():
... a
... a = 3
...
>>> foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in foo
UnboundLocalError: local variable 'a' referenced before assignment
>>> def foo():
... a
...
>>> foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in foo
NameError: name 'a' is not defined
Typing does associate type with the names as well, which isn't necessary for legal Python (but is often cleaner). For example, mypy will yell at you if you do something like this:
def add_week_to_day(d: str) -> str:
# d arrives as an ISO string, like '2016-12-23'
d = datetime.strptime(d, '%Y-%m-%d')
d = d + timedelta(days=7)
return d.strftime('%Y-%m-%d')
Like, I make it a point to run all my stuff through pylint, and it typically catches a few things that will technically run but would cause problems.
Issue 12345 - https://bugs.python.org/issue12345
What this is doing is painting in some of the missing corners of the syntax. Anything you can do with a list comprehension can be done with a for loop, less tersely, often but not always less readably (choose wisely). This new syntax allows you to write comprehensions with asynchronous behavior that in 3.5 and earlier requires you to write a loop.
Progress is in the eye of the beholder. "Code is read more often than it is written." Please chose the most readable form.
Or this is no longer an issue?
I realize it, but (1) even after large libraries support 3 there will still be a long tail of libraries that aren't ported (and will never be), and (2) I see very little value in porting our very large code bases to Python 3, with its breaking changes and subtle behavioral differences that need to be thoroughly tested. There's no must-have feature for us to justify that effort, so we won't be making the switch for the foreseeable future.
IMO Python should have put a lot more effort into backwards compatibility when releasing Python 3. I'm genuinely concerned that this release has killed Python, and I'm not alone.
That's too bad. We've started writing all new code in 3, and now I can't stand working with 2. It feels clunky and old in comparison. A year ago I never would have believed I could feel that way, but I do: I will not willingly (that is, absent strong business reasons) write Python 2 code again.
That should be a non-issue unless you require one of those libraries.
> There's no must-have feature for us to justify that effort, so we won't be making the switch for the foreseeable future.
There's no reason for making the switch if you're going to ditch your code base in the foreseeable future. Otherwise, you'll have to justify your decision in the unforeseeable future. And thereafter.
That's kind of a truism, isn't it? I don't understand the dismissiveness - there's certainly a sizable portion of users that use libraries other than the top 50 most popular.
In the end, you should ask yourself some questions:
Am I starting a new project or continuing an older one? If the former one, Python 3 should be preferred unless some library I need is only available for Python 2. If the latter, I should ask if the cost of migrating my codebase makes sense. If not, Python 2 is still a viable choice until 2020 at least.
Eventually, don't you think that you will start using a more modern version of Python?
You may have also missed the other features from 3.0 to 3.6. Cumulatively they are substantial.
Of course I pray every night to Eris that such a fiasko will never happen to Elixir.
Most certainly use Python3 for any new projects.
Here's a quick overview of the most popular PyPI packages and their support for Python 3: http://py3readiness.org/
As a counterpoint, this JetBrains survey [1] shows only 40% of Python users ever use Python 3. Considering it has been eight years since the initial release, that's a very low share.
[1]
I don't think it's using "Most certainly" in the sense of "Most developers using Python use Python 3 for any new projects.", like you appear to be thinking it says.
Rather, it appears to be saying, "Without any doubt, if you're using Python use Python 3 for any new projects."
It appears to be a statement regarding the suitability of Python 3 for new projects, rather than some quantitative claim regarding the number of users of Python 3.
After it was released, it'd be only the early adopters for a couple years, then the major libraries would follow a few years after. After 5ish years, the big libraries were ported and the second wave of early adopters could shift over. Now we'll start seeing major enterprises make the move over the next few years. Facebook already made the change, but they're tech-forward.
How many people use old versions of RHEL? Of Windows?
supervisord is one of the most useful tools to run production systems.. and is more relevant when we look at the Docker world.
A port of supervisord is something we would gladly donate to :(
Unfortunately, I think the maintainers are handicapped by py3 expertise - https://github.com/Supervisor/supervisor/labels/python%203
Supervisor has some major issues on Python 3. Many of these issues are encoding related as well so merging this one patch doesn't move the needle much. We need someone who has strong experience in Python 2/3 porting and is willing to spend a non-trivial amount of time looking at these bytes/strings issues together.
https://github.com/Supervisor/supervisor/pull/471#issuecomme...
IMHO, what kills the Python ecosystem is people attempting to stay on a 6+ year old major version when they've been told repeatedly that it's officially dead. Upgrading to Python 3 isn't exactly like porting from Java to OCaml, and writing new code in Python 2 today seems flat-out negligent to me.
(I maintain https://github.com/rcarmo/alpine-python, which saves me hours and hours of deployment time)
https://www.reddit.com/r/Python/comments/5jwf41/python_360_h...
Well-organized virtual environments AND python versions. Simply amazing!
"brew install pyenv-virtualenv" has me up and running with everything I ever wanted in one line