I am super excited this is finally accepted. I started working on PEP 501 4 years ago.
As Nick mentioned, PEP 750 had a long and winding road to its final acceptance; as the process wore on, and the complexities of the earliest cuts of the PEPs were reconsidered, the two converged.
[0] The very first announcement: https://discuss.python.org/t/pep-750-tag-strings-for-writing...
[1] Much later in the PEP process: https://discuss.python.org/t/pep750-template-strings-new-upd...
There have been processes put into place in recent years to try to curb the difficulty of things. One of those is that all new PEPs have to include a "how can you teach this to beginers" section, as seen here on this pep: https://peps.python.org/pep-0750/#how-to-teach-this
Other than a more broad "how is the language as a whole faring?" test, which might be done through surveys or other product-style research, I think this is just plainly a hard problem to approach, just by the nature that it's largely about user experience.
"Some hammers are just shaped weird, oh well, just make do with it."
For example, some people that I interview does not "get" why you have to initialize the dict before doing dict[k] += 1. They know that they have to do some ritual of checking for k in dict and dict[k] = 0. But they don't get that += desugars into dict[k] = dict[k] + 1.
Example, if you can go through (I'm not sure you can) and trivially replace all your fs with ts, and then have some minor fixups where the final product is used, I don't think a migration from one to the other would be terribly painful. Time-consuming, yes.
https://peps.python.org/pep-0750/#no-template-str-implementa...
My main motivation as an author of 501 was to ensure user input is properly escaped when inserting into sql, which you cant enforce with f-strings.
I used to wish for that and got it in JS with template strings and libs around it. For what it’s worth (you got a whole PEP done, you have more credibility than I do) I ended up changing my mind, I think it’s a mistake.
It’s _nice_ from a syntax perspective. But it obscures the reality of sql query/parameter segregation, it builds an abstraction on top of sql that’s leaky and doesn’t even look like an abstraction.
And more importantly, it looks _way too close_ to the wrong thing. If the difference between the safe way to do sql and the unsafe way is one character and a non-trivial understanding of string formatting in python… bad things will happen. In a one-person project it’s manageable, in a bigger one where people have different experiences and seniority it will go wrong.
It’s certainly cute. I don’t thing it’s a good thing for sql queries.
One statement the PEP could put front and center in the abstract could be "t-strings are not strings".
t-string is an unfortunate name for something that is not a string.
"foo %s" % "bar"
"foo {}".format("bar")
bar = "bar"; f"foo {bar}"
bar = "bar"; t"foo {bar}" # has extra functionality!It does suck for beginners who end up having to know about all variations until their usage drops off.
I see (4) being about the flexibility of (2) and readability of (3). Maybe it'll eventually grow to dominate one or both, but it's also fine if it doesn't. I don't see (1) going away at all since the curly collision still exists in (4).
The other ones at least are based on the same format string syntax.
"foo {}".format("bar") would be an obvious "just use f-string" case, except when the formatting happens far off. But in that case you could "just" use t-strings? Except in cases where you're (for example) reading a format string from a file. Remember, t- and f- strings are syntactic elements, so dynamism prevents usage of it!
So you have the following use cases:
- printf-style formatting: some C-style string formatting is needed
- .format: You can't use an f- string because of non-locality in data to format, and you can't use a t- string due to dynamism in
- f-string: you have the template and the data in the same spot lexicographically, and you just want string concatenation (very common!)
- t-string: you have the template and the data in the same spot lexicogrpahically, but want to use special logic to actually build up your resulting value (which might not even be a string!)
The last two additions being syntax makes it hard to use them to cover all use cases of the first two.
But in a specific use case? It's very likely that there is an exact best answer amongst these 4.
It’s also the only one which is anything near safe for being user provided.
printf-style... does not support any of that. It can only format the objects passed in.
import string
t = string.Template("foo $bar")
t.substitute(bar="bar")log.error("foo happend %s", reason)
There are also loads of people basically defaulting to "no" on new features, because they understand that there is a cost of supporting things. I will often disagree about the evaluation of that cost, but it's hard to say there is no cost.
Nobody wants a system that is unusable, slow, hard to implement for, or hard to understand. People sometimes just have different weights on each of these properties. And some people are in a very awkward position of overestimating costs due to overestimating implementation effort. So you end up in discussions like "this is hard to understand!" "No it isn't!"
Hard to move beyond, but the existence of these kinds of conversations serve, in a way, as proof that people aren't jumping on every new feature. Python is still a language that is conservative in what it adds.
This should actually inspire more confidence in people that features added to Python are _useful_, because there are many people who are defaulting to not adding new features. Recent additions to Python speeding up is more an indicator of the process improving and identifying the good stuff rather than a lowering of the bar.
[0]: I often think that these discussions often get fairly intense. Understandability is definitely a core Python value, but I Think sometimes discussions confuse "understandability" with "amount of things in the system". You don't have to fully understand pervasive hashing to understand Python's pervasive value equality semantics! A complex system is needed to support a simple one!
> they are better for most use cases of string templating where what you really want, is just a string.
I think use cases where you want to unconditionally bash a string together are rare. I'd bet that in > 80% of cases the "just a string" really is just a terrible representation for what really is either some tree (html, sql, python, ...) structure or at least requires lazy processing (logging, where you only want to pay for the expensive string formatting and generation if you run at the log level or higher that the relevant logging line is meant to operate).
"just a string" where the string really is just a string is really, really common though. Python is the go-to language for a lot of people (including me) if they just want to bash together a one-time-use script that will never see input from other people, and f-strings are incredibly useful in such cases for generating filenames, debugging messages, etc.
`some template {someVar}` was f-strings and someFunction`some template {someVar}` was more like what these t-strings provide to Python. t-strings return an object (called Template) with the template and the things that go into the "holes", versus tagged templates are a function calling pattern, but t-strings are still basically the other, richer half of ES2015+ template strings.
I do this myself. I basically always use the subtl wrong log.warning(f"Unexpected {response=} encountered") and not the correct, and depending on the loglevel cheaper log.warning("Unexpected respone=%s encountered", repsonse). The extra visual noise is typically not worth the extra correctness and performance (I'd obviously not do this in some publically exposed service receiving untrusted inputs).
I'd argue these use cases are in fact more prevalent then the bashing unstructured text use case.
Encouraging people to write injection vulnerabilities or performance and correcness bugs isn't great language design.
https://lucumr.pocoo.org/2016/12/29/careful-with-str-format/
So, right now, you have two options to log:
1. `logger.debug(f'Processing {x}')` - looks great, but evaluates anyway, even if logging level > `logging.DEBUG`;
2. `logger.debug('Processing %s', x)` - won't evaluate till necessary.
What would be the approach with t-strings in this case? Would we get any benefits?
For a logger t-strings are mostly just a more pleasant and less bug-prone syntax for #2
Also: were prompt templates for LLM prompt chaining a use case that influenced the design in any way (examples being LangChain and dozens of other libraries with similar functionlity)?
The main reason for non having deferred evaluation was that it over-complicated the feature quite a bit and introduces a rune. Deferred evaluation also has the potential to dramatically increase complexity for beginners in the language, as it can be confusing to follow if you dont know what is going on. Which means "deferred by default" wasnt going to be accepted.
As for LLM's, it was not the main consideration, as the PEP process here started before LLM's were popular.
Maybe not directly, but the Python community is full of LLM users and so I think there's a general awareness of the issues.
Yes, you can do that, but a t-string is much more ergonomic, and IMO, more readable .
t-strings allow a library to perform transformations on the values, such as escaping them, or passing them as separate values to a parameterized query. Escaping html and parameterizing sql queries were the first two example use cases given in the PEP.
And I disagree that such use cases are niche. In my experience, needing to sanitize user input is an extremely common thing to need to do, and having the language and library make it as easy as possible to do it correctly is a very good thing.
[1]: I do wish they hadn't called these Templates, because it's not really a template so much as an intermediate representation of an interpolated value.
Pre-compilation means that you first compile the template, then you supply the template with values when you render it multiple times. This is not possible with t-strings since the values are bound when the t-string is created.
With respect to compilation, that is basically is how t-strings work, but it is the python interpreter that does the compilation. When it parses the t-string, it compiles it to (byte) code to generate a Template object from from the expressions in scope when it is evaluated, which may happen more than once. And if you really want a template that is a separate object that is passed the values separately, you can just wrap a t-string in a function that takes the parameters as arguments.
> two dozen templating libraries that offer much more comprehensive safe and fast text-generation solutions than what t-strings do
But t-strings allow those libraries to be safer (users are less likely to accidentally interpolate values in an f-string, if a t-string is required) and possibly faster (since the python interpreter does the hard work of splitting up the string for you. t-strings don't replace those libraries, it allows them to be better.
> And if you really want a template that is a separate object that is passed the values separately, you can just wrap a t-string in a function that takes the parameters as arguments.
No, you can't do that: "Template strings are evaluated eagerly from left to right, just like f-strings. This means that interpolations are evaluated immediately when the template string is processed, not deferred or wrapped in lambdas." Every function evaluation creates a new Template object, it does not reuse a precompiled one.
> and possibly faster
Possibly not, since precompilation is not supported.
I don't know what you mean by `executable` streams, but besides databases as I've already mentioned, a common thing that shows up in non-web applications is invoking a shell command that includes a user-supplied file name as part of it. Currently doing so safely means you need to call `shlex.quote` or similar on the filename, but with t-strings you could have something like: `shell(t"some-command {filename} 2> somefile | other-command")`.
And that is just one specific example. There are other cases it might be useful as well, like say generating an XML configuration file from a template that includes user-supplied input.
> No, you can't do that... Every function evaluation creates a new Template object, it does not reuse a precompiled one.
The code that generates that Template object is pre-compiled though.
If you define a function like:
def my_template(a, b,c):
return t"a={a} b={b} c={c}"
def my_template(a, b,c):
return Template("a=", Interpolation(a, ...), " b=", Interpolation(b, ...), " c=", Interpolation(c,...))