PEP 0501 – Translation ready string interpolation
python.org
python.org
That said, I think it would be much improved if it went the way of Scala and JavaScript (2015+) and made `r`, `u`, `b`, and `f` all interpolation providers. That way libraries could provide interpolators without having to extend the language to create a new one.
For example, in Scala or JavaScript, you can create a string context / template tag for composing JSON strings, resulting in a nice syntax like this (scala-like):
json"""{
"my": $data,
"anArray": $someArray
}"""
The `data` and `someArray` values serialized to JSON by the `json` interpolation provider.At that point, internationalization is just creating your own interpolation provider and importing it as `i`, which is far less machinery than this PEP requires.
Both are equally explicit, though f"" is perhaps marginally more clear because it is somewhat mnemonic.
https://www.reddit.com/r/Python/comments/3gfaji/pep_501_tran...
I think this and pep 498 are a pretty horrible proposals and does not fit well with the rest of the language and it's spirit.
1. Decrements readability. When variables are hidden inside strings, you need to check for presence of a variable in two contexts. In the code and inside the strings. With format and % functions, you will be able to spot the variable in the argument list, which is part of the code itself.
2. Security concerns regarding automatic evaluation of expressions inside the strings.
Also, why was format and % made in such a way as to require an explicit list of variables? What was the reasoning behind it? Is that reason no longer relevant?
Sounds like "Explicit is better than implicit" from the Tao of Python.
That is not always true, e.g.:
>>> x = 2
>>> print("x = {x}".format(**vars()))
x = 2For 2, can you expand upon this? Not a challenge, but I'm not able to imagine how this would be a security concern, and this might be a failure of my understanding here.
It is more of a matter of breaking a general principle, which is don't mix executable stuff (expressions) with non-executable stuff (strings) implicitly. This has been a frequent cause of various security issues in the past eg: Sql injection, xss, various remote code execution vulnerabilities.
Strings should be dumb and without any ambiguity. On top of breaking this general principle of not mixing data and code, this pep also want to introduce numerous rules regarding various edge cases, which further makes it harder to reason about it, which increases the chances of slipping a vulnerability. For example, right now, you see a long string, you only need to look at the argument list to see what possible things it can do. Because the string itself is dumb, it cannot lie, and whatever execution that is required to produce that string, happens in the context of code itself. With this pep, this changes. Now a string is not dumb. It is smart. It can do 'stuff' on its own. It can 'hide' things, it can lie and masquerade as something innocent. I think this is bad.
Take an example of PHP. You can embedd php code in what ever content. In fact you can embedd php code in a valid image file in the metadata fields, and pass it as an Image. if you can call the image file as a script, then the embedded php code will execute to do your bidding. This has to be one of the most commonly used technique to hack sites running php. Here is one from last day [1].
Those are some of my reasons for the concerns..I know I am not still giving anything specific. But if we could easily think specific cases about how something could be exploited, we wouldn't be implementing that in the first place. So sometimes we must rely on general principles learned from the past while assessing the issues associated with something. Hence my concerns.
[1] https://www.reddit.com/r/PHP/comments/3gq3mh/my_site_was_hac...
You do have a good argument concerning not mixing of data and execution as a general principle, though, and I think may be you gave the PHP example as how this mixing can lead to certain unintended issues (not necessarily in python with this PEP implemented, if so, I apologize for misunderstanding). In my mind, this helps with short strings, with simple expressions and makes them more readable--in my opinion, as I said...it's just a convenience. Still, allowing a simple expression allows anything really to be executed, so I guess you have an argument there.
A good language protects the experienced developers from making a stupid mistake, and forbids the novices from them to trigger a learning response and guide them to better ways.
Example: In python if you use a normal map to implement an algorithm that depends on the elements being ordered in some way, then you will get different results for each run. If I remember correctly, I think python will randomize the order of the map every time it is iterated over. You see, developers could have got away with blaming the user for using an ordinary map when they should have used ordered map. But instead, they implementation map in such a way that it would be impossible to use it incorrectly.
> I think may be you gave the PHP example as how this mixing can lead to certain unintended issues (not necessarily in python with this PEP..
Yes. correct.
>it's just a convenience..
I come from a land filled with such 'conveniences'. it is called PHP. You can compare
var_dump("228" == "0xe4"); -> True
See? a convenience. You can compare hex and decimal strings with out explicit conversion. And what is the result? Countless exploits and security holes that leverage things like these, all that can be possible blamed on the programmer. So was this worth it? Where would you draw the line? How much correctness are you willing to trade off for a minor convenience?I may be over-reacting. But I think this pep is a step in the wrong direction and should be aggressively corrected.
No, a normal non-ordered dict in python is guaranteed to have the same iteration order as long as no keys are removed or inserted (i.e., if the only changes are changing the values associated with keys.)
1. `"Value: %i" % value`
2. `"Value: {}".format(value)`
3. `string.Template('Value: $value').substitute(value=value)`
However, the Zen of Python (https://www.python.org/dev/peps/pep-0020/ or run `import this` in a Python interpreter) says "There should be one-- and preferably only one --obvious way to do it.".
How are there security concerns? Its a kind of string literal that is expanded into a calls to builtin formatting functions at compile time, so there is no more security concern with evaluating expressions in it then there is with evaluating expressions in the non-string parts of your code.
> Also, why was format and % made in such a way as to require an explicit list of variables? What was the reasoning behind it? Is that reason no longer relevant?
You can use those without an explicit list of variables, its optional to use explicit variables for clarity (though in many cases the natural naming will also make it redundant in simple cases.)
It is more about breaking a general principle of not mixing executable stuff and non-executable stuff. I have written more about it here.
https://news.ycombinator.com/item?id=10063637
>its optional to use explicit variables for clarity
I don't think it is optional. If it was optional, then the behavior would be to interpolate from local variables by default, and do them selectively if an argument list is provided.
So someone, in the past, thought about this and decided that it is best to always require an explicit list of variables for interpolation.
This pep is trying to undo that decision. Why? What was the reason to always require an explicit list of variables? Didn't they think that it would be convenient to automatically interpolate from locals? What was it that made them decide against it? Isn't that reason still relevant?
Without answering these questions, I think it is not wise to continue with this pep.
1. Internationalization is, as far as I know, an unsolved problem. This solution doesn't stand out as obviously better than the alternatives. Unsolved problems don't belong at the language level. I'd rather wait until a real solution to this problem is discovered.
2. What happened to explicit is better than implicit? Isn't that a core value of Python? Overriding implementations of builtin functions is all too magical, allowing spooky action at a distance. I'm extremely skeptical of solutions that don't allow me to grep my code for where a function is implemented. `gettext`, at least is clearly calling a function.
Overall, I see there being two major styles for building strings. Since I use C++, I can't do literal interpolation, but I can get close with variadic templates. I hope you guys won't me talking about a statically typed language here, but it should be on topic to string interpolation at least.
//interpolation
print("My name is ", name, " and next year, I will be ", age + 1, " years old.\n");
//arguments
print("My name is {0}, and next year, I will be {1} years old.\n", name, age + 1);
//arguments with hinting (:label is ignored)
print("My name is {0:name}, and next year, I will be {1:age} years old."\n", fetch(), *values + 1);
print(hex(size, 4), " KiB"); //interpolation formatting
print("{0,4x} KiB", size); //argument formatting (more covenient, but more cryptic and limited)
//without the print component:
string s{"My name is: ", name, "\n"}; //interpolation
string s{"My name is: {0}\n", name}; //arguments
The latter is certainly vastly superior for localization. Because in another language, it may be more grammatically correct to reverse the parameters, eg: "来年の歳は{1}と名前は{0}"; which is something C printf doesn't support, shy of unofficial extensions for argument reordering.But I find that as the string gets longer and longer, the former is much more readable. (You'll have to imagine the syntax highlighting, which helps a lot.) Eg when you have 17 arguments, it can be annoying to count to find the 11th parameter, whereas with interpolation, it's right there where it's used.
Ultimately, I decided on the former being the default, and added a format class for the latter, which is also really handy for building your argument lists somewhere else. Plus the default won't clobber your escape characters (in my case, braces.)
//inline format
print("I am {0}, and I'm {1} years old.\n", format{name, age});
//external format
format f;
f.append(fetch()); //person's age
f.append(*values + 1); //person's name
... //you could fetch the string from a translation table below, too
print("I am {0}, and I'm almost {1} years old.\n", f);
Whenever string::append(format) is invoked, it will use all of the values inside the format array to parse {#} entries in the string. Of course, it's also safe and will simply skip over a reference to {2} in a format array with only one value.Curious to hear others' opinions on the above, and any suggestions for improvement.