Format Strings in Rust 1.58
rustnote.com
rustnote.com
I wish they don't implement f-strings and s-strings, at least for now. Even if they are more ergonomic than the `format!` and `String::from`, they hide a memory allocation which is not really indicated in a language like Rust and would be really weird in a context without an allocator. The only solution to this would be `const` evaluation of this, but that would restrict their use to `const` environments, so mostly unusable.
"C++11/14 at Scale: What Have We Learned?"
While using C++, adding something like gsl to the toolbox is worth gold, and also enabling bounds checking even in release (really, most of the time it hardly matters to the application users).
f-strings on other hand would be pretty weird. Would `f"5"` return a &'static str or a String?
In your example I would return a String and generate a warning, because the f of the string isn't used.
str (and thus &str) is part of the language, it's a built-in primitive type like i64 or bool, but String is just a struct the alloc crate brings into existence and so it may not be available.
Though could this work by the language feature just working like a Macro? Code without alloc using f strings just wouldn't compile then
Only the macro knows the name of the variable, so you can't do this inside the formatter itself.
The Rust's implementation apparently only looks up variables from the enclosing scope. By contrast, Python's f-strings allow arbitrary expressions, and could potentially fall for the LDAP trick, or something similar. Same with ES6's backtick-strings.
I hope Rust will keep it simple and reliable, and won't allow calling functions in format strings.
> I hope Rust will keep it simple and reliable, and won't allow calling functions in format strings.
Yes, I think people are wary of allowing full expressions in format strings as in Python. That said, I might like to see a small extension to the current rules, so that in addition to identifiers you could also access struct members. I agree that format string captures shouldn't present the chance to run arbitrary code, so I wouldn't even extend this to array indexing (which is overloadable via the Index trait).
The scary part about the vulnerability was that string >inputs< could dynamically delegate it's content to some magic information fetching system which by default allows accessing remote content in a way which by design can lead to remote code execution.
I have no idea who though non static format strings are a good idea.
Or an information fetching system which can trigger remote code execution.
Or that this system doesn't require stric whitelisting.
Additionally, with unsafe it’s also possible. This is in response to “the only possible” statement.
Of course, doing so is not advisable.
Doesn't that open a big security hole where someone could inject a special string to probe for variables' content?
You can't really pass untrusted input as a format string because they have to be available at compile time.
Also, FTA: “Remember that Rust doesn't use any localization, so these outputs will always look the same.”
So, what do rust programs do for localization, e.g. to print the thousands separators users expect? Is there a library that gives you similar string interpolation, taking locale into account?
It’s a tough call between catering for computers by ignoring locale and for humans by applying it, but I think I would have chosen for this to be locale-sensitive, with support for forcing a standard locale.
edit: There is also the issue that on some platforms (Linux) locale has some soundness issues (not a fault of rust) and yet on others it can be hard to use locales properly (Windows can introduce some hard locale weirdness)
{a} foo bar {b}
Baz {b} {a} quux
- As pointed out by GP, `format!` is a macro and only works for literal format strings available at compile time. This allows the compiler to convert the format string to code _at compile time_.
- What you are talking about is a general string template/formatting engine to execute at runtime. Such a feature can easily be provided by external crates, because it would work at runtime and not require any particular interactions with the compiler.
"Things breaking because of unexpected locale issues" is pretty high up on the most common bugs list.
Rust already used curlies to indicate places where values would be inserted in format! (including print!/println!) strings, so you couldn't use them from config files with unescaped curlies already, AFAIK.
[0]: https://blog.rust-lang.org/2022/01/13/Rust-1.58.0.html#captu...
That said, Rust isn't a company, it's a volunteer organization, and features advance at the speed of enthusiasm. If there's a feature that someone wants to see, there's no use waiting around for it, someone's got to be the one to push it forward. :)
So that would be "Hello {{x}}!".
I imagine this is going to break a lot of existing code. There doesn't even seem to be a way to have {x} mean the same thing in 1.57 and 1.58
But how would you print {x} in 1.57? I doubt you would escape it with {{ in that release. Maybe \{ then? Does that still work in 1.58?
This still smells fishy to me, but I need to dig deeper.
You would. Brackets were still escaping in that release because you would of course still do `println!("{}", x);`
pub fn main() {
let y = 4;
println!("{x}", x=y);
}
pub fn main() {
let x = 4; // unused
println!("{{x}}");
} pub fn main() {
let x = 4; // unused
println!("{x}");
}
error: there is no argument named `x`
--> <source>:3:15
|
3 | println!("{x}");
| ^^^
Additionally, new releases are frequently tested with a tool called Crater, which basically builds and runs the tests for all publicly available Rust code. This helps massively in upholding backwards compatibility guarantees and in evaluating if a compatibility break is worth it.
print!("{}", hello_string)
print!("Hello {{x}}!")
print!("Hello {x}!", x="{x}")
let x = "{x}"
print!("Hello {x}!")
Apologies for going meta, but I agree.
To the OP:
The guidelines for Show HN [1] say: "Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread." Blog posts are specifically off topic.
You seem to have posted this link ten hours ago and then re-posted the same link as a Show HN two hours ago - perhaps to get around the duplicate link detection. Please don't do that.
I'd prefer something that would maybe be less concise but easier to read and maintain, using the host language instead of a mini script using in-band signaling and its weird syntax.
Meanwhile, can't complain all that much if zero-cost features can be added to Rust to make it easier to market it to scripting folk. I think that can only be a good thing, even if the feature design is far from ideal.
"Hello {username}!"
"Hello $username!"
"Hello " . username . "!"
"Hello " + username + "!"
"Hello " << username << "!"
Of those, I find the first and second one by far the easiest to read, and the first one is easier to extend (as rust has done, e.g. "USD{total:>6}" for a left-padded number).
It doesn't add additional syntax (and therefore complexity, compiler steps, build time, etc) to the language; the only convenience applied here is varargs for the arguments besides the first.
A lot of languages started with basic (s)printf and added string interpolation later on, but that brings its own headaches. I'm thinking of PHP, where on the one hand you can do "hello $username", but if it's a property in an object you need to add additional syntax already - "hello {$user->name}".
But really I didn't really mention it because printf syntax seems to me like exactly the "mini script using in-band signaling and its weird syntax" that GP complained about.
In contrast, the old-school stdio sprintf (and relatives) will interpret the format string at run time and then read a varargs list to do the interpolation which can lead to run-time errors and buffer-overflow vulnerabilities and so forth.
[1]: https://owasp.org/www-project-top-ten/
[2]: https://www.softwaretestinghelp.com/sans-top-20-security-vul...
As someone noted in another comment, this with Rust is effectively opt in with the println! and format! macros. Does it converted to machine code at compile time?
let f = "{:.3}";
println!(f, 3.145);
Personally I dislike the whole idea of embedding sub-languages in strings inside host languages, but this is a lost cause frankly, and if one must do it, this is a pretty good way.
As a result, if you take arbitrary input, and then run it as an SQL query who knows what will happen, despite it being a memory safe language.
On the other hand, (safe) Rust has strong type checking, so you can make types named SafeHTML, ValidSQLString, XMLelement and so on, with the properties you desire enforced. This does not prevent the same idiots who try to make an SQL query using format!() from doing so, as they won't use ValidSQLString anyway.
If all strings fill you with such fear, probably General Purpose Programming languages aren't for you, maybe you will feel safer in WUFFS. It looks at first glance as though WUFFS has strings, but it actually doesn't, they're just a human readable label for WUFFS non-OK statuses (e.g. errors like "#bad Huffman code"). You can rest assured that your WUFFS code for processing a PDF can't have SQL injection for two reasons: 1. WUFFS doesn't have any strings to inject SQL into, and 2. WUFFS can't talk to an SQL database at all. In exchange for this safety you give up the ability to do anything outside the tiny sphere of interest of the language.
I think the number of developers who would not have insecurely built a SQL string from user input but for the language adding format strings is approximately zero.
By analogy, to me this argument seems like saying, "If a standard library contains a StringBuilder class that minimizes unnecessary allocations when constructing a String, it encourages developers to construct SQL strings manually and therefore makes them more likely to fall victim to SQL injection attacks, therefore we must not provide StringBuilder classes."
I don't imagine there are very many developers who are properly using parameterized queries only because it is inconvenient to concatenate strings.
I'm not saying that format strings necessarily make injection vulnerabilities more common, but rather that, when added nowadays, they should be designed in a way that makes such vulnerabilities less common. If you're adding a feature that makes a potentially dangerous operation more convenient, you should also make it safer so that the convenience will help pull developers toward the safe option.
let result = Query::select().field("id").from("sometable").exec()?;
Maybe, we should create a specialized `format!()` macro, for example: `formatValidHtml!()`, `formatSafeHtml!()`, `formatAccessibleHtml!()`, or just a `formatRestricted!(ValidHtml + SafeHtml + AccessibleHtml, "<h1 role=\"banner\">{safeTitle}</h1>");`
IMHO generating code of any sort through string manipulation is a code smell, even if you ignore the security issues. There are better options like parameterized queries for SQL, macros for HTML, serde_json::json! for JSON, etc.
That is the problem, and why security experts wouldn't let us add string interpolation to Java until we had a way to require the format string to say how it will be used and enforce proper validation. [1]
Even if programmers should be more careful, the fact remains that this is the #1 security vulnerability caused by language features. The language should, if possible, make it easier for developers to avoid the mistake rather than make it easier for them to make it.
I wonder how this would handle something like HTML injection, where the desired encoding can change within the same string depending on whether you’re in an attribute or a text block?
Yes. That's what Java's upcoming templated strings feature does. The library provides a templating policy as the way to create the appropriate type it requires.
> I wonder how this would handle something like HTML injection, where the desired encoding can change within the same string depending on whether you’re in an attribute or a text block?
The client library decides how to interpret and format the string in its policy.
But then format strings don't help you much as you can't use them to create the sanitised types. You can't sanitise the string after it's been constructed.
SQL."SELECT * FROM orders WHERE \{mcol} LIKE '%' || \{qry} || '%'"
SQL."SELECT * FROM orders WHERE \{mcol:sql_id} LIKE '%' || \{qry:sql_likestr} || '%'"
"SELECT * FROM orders WHERE " + mcol + " LIKE '%" + qry + "%'"
Any statically typed language can do the first part, the second part is a tad less trivial (and arguably a footgun in some scenarios, but that's another discussion.)
let query = format!("SELECT name FROM users WHERE id = {}", user_id);
let mut query = String::from("SELECT name FROM users WHERE id = ");
query += user_id;
I know, thank you.
> What, are you also going to forbid simple string operations?
Dude, chill. Why do you feel attacked? I'm just stating facts. Those facts are not a critique of you (or of Rust, a language I love).
Furthermore, within the Rust ecosystem, the tools to build SQL queries, or serialize/deserialize JSON generally provide interfaces that are more ergonomic than manually using format strings, so a programmer has little incentive to do so as it would constitute more work on their part.
It's obviously still possible to implement the Display trait for a type in a way which makes it susceptible to code execution attacks, but that doesn't really have anything to do with string formatting.
I don't know enough about rust but I would imagine it is hard to println into a SQL statement. Hence it is not 'natural' to use this feature to build commands to be executed elsewhere.
let query = sql_query!("SELECT * FROM users WHERE id = {}", user_id);
query.execute();
let query = sql_query("SELECT * FROM users WHERE id = @PLACEHOLDER@", user_id);
query.execute();
In that case, you should ask yourself what's the point of format strings at all? The answer is that they're more attractive because using them is more convenient. You want to make the more attractive feature the safer feature to draw people away from danger, not toward it.
There's an entire world of software which does something else than creating SQL queries or HTML pages. Rust is a general purpose language, not some niche DSL. I use format strings all the time in my Rust code, yet I've never been in a situation where those strings should or could have been sanitized.
> Constructing SQL queries or JSON expressions with templates is convenient, but is at risk for injection attacks. Improving mechanisms for constructing composite strings without similarly improving or enabling safer mechanisms for constructing queries would surely widen the attack surface.
The fact that in many languages (and ecosystem actually, because it's not directly a language issue) building insecure queries (or HTML, or anything) is the simple way, and doing thing right requires specific thoughts from the developer[1] is what leads to so many injection attacks in the wild.
I think this is mostly a cultural thing, and having being developed recently, way after the injections attacks have become ubiquitous, the Rust ecosystem has been focusing on providing better developer experience for the safe path than the vulnerable one. Diesel and SQLx use prepared statement by default, Serde serialize JSON without exposing strings at all to the developer, HTML templating libraries have sanitization built-in, etc.
[1]: this example is also taken from your link:
String query = "SELECT * FROM Person p where p.last_name = '$name'";
ResultSet rs = connection.createStatement().executeQuery(query);
PreparedStatement ps = connection.prepareStatement("SELECT \* FROM Person p where p.last_name = ?");
ps.setString(1, name);
ResultSet resultSet = preparedStatement.executeQuery()But let me push on it a little more. I think that what you're seeing isn't so much a culture thing but a small ecosystem thing. Imagine that Rust takes off and in ten years there are 1M professional Rust programmers who use the language because that's the one chosen by their employer. You'll not have one JSON library (or whatever other format will be used then) but 50 and so on, and most programmers will not be experienced ones but relative novices (to programming in general). How likely would it be for them to generate JSON with format! ? So this feature provides a better user experience for the less safe path. A feature should be designed with the next 20 years in mind.
So in terms of weight pulled by a feature it is not competing with say, the add-and-assign operator +=, or even with the AddAssign operator overload trait (which is a langitem), but only with some library feature like euclidean remainder on integers, which I hope you will agree is unlikely to be more commonly used than format interpolation.
I don't know why you think that serde_json isn't good enough and so 49 other JSON libraries will spring up, but I also don't know why you're sure a programmer will decide they ought to write format!("\"{json_string}\"") but you're confident they would never write format!("\"{}\"", json_string). People determined to shoot themselves in the foot are going to do it, we provide much better, simpler, clearer ways to do what they wanted to do, but in general purposes languages it will always be possible for them to point the gun at their feet, dismiss the warning "CAUTION! Do not shoot yourself in the foot", click the safety and pull the trigger.
Finally, Rust doesn't have to design all its features with 20 years of unknowable future implausibly considered in advance, because it has Editions. If you're correct and we regret providing format!() the Rust 2040 edition needn't provide this, and old code still works.
In this current form yes, but f-strings is a language feature, and with this announcement, there's a lot of people hoping for f-string to arrive too.
Because I have some decades of experience.
> I also don't know why you're sure a programmer will decide they ought to write format!("\"{json_string}\"") but you're confident they would never write format!("\"{}\"", json_string).
That's not my argument. I am not saying string formatting will make injection vulnerabilities more likely, but that it's a missed opportunity to make them less likely. You add a new feature because it's more convenient and attractive, and so you expect people to use it. If you know that feature touches on what's known to be one of the most dangerous aspects of programming, you might as well make it more convenient and safer, so that you attract programmers away from the less safe options and toward the safer ones.
Is this decades of experience with Rust (from 2015) or decades of experience with JSON (from 2001) ? Or just decades of experience making implausible predictions?
Here's how you make a JSON string in serde_rust here in 2022:
let s = Value::String(myString);
let s = format!("\"{myString}\"");
let s = Value::String(myString);You insist it should "validate" the strings but it's a general purpose formatter, there isn't anything to validate that isn't already mandatory in the language.
Yes, if I take the SQL formatter and I use it to make email addresses that's more likely to incur dangerous vulnerabilities. This is not a defect in the SQL formatter, I am using the wrong tools.
> Are you saying that you're not worried about that 0.1% because you can be certain even one programmer in a thousand won't make such mistakes, or because you don't expect Rust to be popular enough for that number to matter?
I can't do anything about the fact that people will make grave logical errors when programming, beyond advocate for testing and code review which might catch those errors. I suspect your 0.1% figure is pulled out of your backside, but, sure, somebody will get it wrong.
General purpose languages shouldn't be riddled with foot guns, but there's a difference between a language not having foot guns and not having any guns at all out of fear that somebody might shoot themselves despite the fact the gun was locked away, the ammo was locked away, and they had taken training in "How to use guns safely" before being given the keys.
Again, if you fear strings you can use special-purpose languages which don't have any strings so that you can't possibly make this mistake. WUFFS is not a language for babies, with training wheels, it's a language for experts who know they don't need stuff like strings in the domain they're working on.
Yes. In my very first comment I posted a link to Java's upcoming feature, which uses the type system to ensure proper validation in a general-purpose string templating mechanism. Here it is again: https://openjdk.java.net/jeps/8273943. As I also said in my first comment, our security experts were so concerned about this problem, which was empirically found to be one of the most dangerous programming operations in recent years (famously so among those interested in security), that they wouldn't let us add it to Java without a solution to the security problem
> This is not a defect in the SQL formatter, I am using the wrong tools.
But 1. people do use the wrong tools, which is why it's such a common and dangerous vulnerability, and 2. a typed language certainly can prevent that, as in my link.
> I can't do anything about the fact that people will make grave logical errors when programming
That's a strange position from a Rust user, especially as in this case there certainly is something the language can do.
> I suspect your 0.1% figure is pulled out of your backside, but, sure, somebody will get it wrong.
Pretty much all top security vulnerabilities lists list this problem near the top, and some languages are so popular (many millions of professional devs) that even 0.1% of their users are sufficient to make this problem a common one, deserving of its spot. So I figure that 0.1% is about the right number to make this a very common problem.
> Again, if you fear strings you can use special-purpose languages which don't have any strings so that you can't possibly make this mistake.
Or use Java's upcoming string templates. Or, if you prefer less popular languages, Scala, which uses a similar technique.
The exact same programmer who you insist will write
format!("\"{myString}\""); // in Rust
will also write:
CONCAT."\"\{myString}\""; // in Java with this JEP
The JEP argues that it'll be all OK when Steiner attacks^W^W^W so long as you only use the potentially tainted "strings" via an API which doesn't allow general string objects. But, that's also the exact situation you've dismissed as unable to prevent abuse in Rust.
Java can't stop you writing your supposedly "JSON" data made with CONCAT to a file or over a network socket, and Rust can't stop you writing some made with format! either, it's just data, who knows why you wanted to write this or that gibberish?
According to you we should expect 0.1% of the "safe" Java using these templates to be vulnerable.
So while preventing someone from constructing JSON (which can be output as a string) is not always as fool-proof as preventing someone doing the same with SQL (as the driver API will simply not offer an option that takes a string) the reason such a feature is added in the first place is because it is easier, and that is what makes it more attractive. A more convenient mechanism attracts people to use it.
Some things, like SQL and JSON, are often convenient to create using templates. Using something like CONCAT."""{"x": "\{x}", "y": "\{y}"}""" is certainly no more convenient, and so no more attractive, than writing JSON."{x: \{x}, y: \{y}}", but it is more convenient in some situations than using an API that requires defining or generating a type for the object in the source language. So while safe JSON libraries in Java and Rust exist today without a built-in template mechanism, being able to offer them with that mechanism will make unsafe options less tempting by comparison.
That is why experienced security experts recommend that languages do not add a string templating mechanism that is more convenient, and so potentially more attractive, than safe options for security-sensitive uses. Their general rule is, "when possible, don't make programmers jump through more hoops to do something secure than something insecure." You're free to find this advice misguided, but I wonder if the people who added this feature to Rust consulted with security experts before adding it. I don't think I would have been aware how sensitive this feature is if it weren't for the advice of security experts.
json!({"x": x, "y": y})
That's much easier than the complicated dance envision in the JEP and which you insist will be "convenient" and dissuade Java programmers from choosing the easy option, yet of course you always get the intended JSON out, even if say x is a malicious input intended to trip up naive JSON encoding.
The entire point of my comment was that, surprising as it might seem to some, templating is now known to be a particularly dangerous area -- quite possibly the most sensitive aspect of language and API design after buffer overflows -- and that's why templating features require a security review.
If Rust's designers' answer is that their security analysis has led them to the conclusion that the right stance against code injection is for template APIs to role their own templating macros from scratch rather than use a higher-level templating mechanism -- then they're doing what I suggested, and macros are their mechanism that corresponds to Java's pluggable templating design. If they did not consult with security experts on their string formatting feature, I suggest they do so. Perhaps all that's needed for Rust is to include components in the standard library that would help library authors write correct and secure templating macros.
Yes exactly.
> which raises the question of why add a feature with so little utility
It's indeed not the most important feature ever, but after running a quick `rg "format!" | wc -l` in my Rust directories (for both work and hobby projects), it found 2797 occurrences, which isn't nothing.
> when it could have much greater utility? There's a missed opportunity here.
I bet most people in the Rust team are simply not aware of the recent Java developments on that front. Hopefully having an openjdk developer sharing insight and documentation on that topic on public forums can foster cross-pollination on that topic :).
Of course culture matters. It isn’t just a side-effect of popularity. Would Scala have the same culture as Java if it became simiarily popular? I doubt it.
Rust has a culture—and it comes from a wider culture of the same ilk—where such messy shortcuts are not taken; instead “fancy language features” (much to some people’s chagrin…) like compile-time evaluation are used to make safer user interfaces for programmers. And that makes for less catastrophically buggy software.
This means you can construct some kind of prepared statement or similar object with a string format, and it will force the correct handling of potential injections.
Of course you can still just format a string and shove it in, but I'd argue that a good SQL library should simply not accept pure string queries and require the format that guarantees injection-safety.
The log4j fiasco didn’t happen because of Java (wink) but because of a plain invulnerable feature. I don’t think Rust makes that more or less likely to happen.
Because all format strings are in macro context, where the macro has full control over what to do with all substituted parameters, Rust already has sanitized string interpolation. In terms of that JEP, the macro invocation is the policy object.