The need for memory safety standards
security.googleblog.com
security.googleblog.com
Also, hilariously, Fil-C is faster than CHERI today (the fastest CHERI silicon available today will be slower than Fil-C running on my laptop, probably by an order of magnitude). And Fil-C is safer.
Which sort of brings up another issue - this could be a case where Google is trying to angle for regulations that support their favorite strategy while excluding competition they don't like.
Fil-C seems to be a C variant that adds capabilities and garbage collection. See https://github.com/pizlonator/llvm-project-deluge/blob/delug...
The speed comparison with a laptop is just disingenuous. Is a device with Cheri integrated slower than one of the same class without?
Here's the difference with Fil-C: It's totally memory safe and fanatically compatible with C/C++, no B.S. The other approaches tend to either create a different language (not compatible), or tend to create a superset of C/C++ with a safe subset (not totally memory safe, only compatible in the sense that unsafe code continues to be unsafe), or just don't fully square the circle (for example SoftBound could have gotten to full compatibility and safety, but the implementation just didn't).
> The speed comparison with a laptop is just disingenuous. Is a device with Cheri integrated slower than one of the same class without?
Not disingenuous at all.
The issue is that:
- high volume silicon tends to outperform low volume silicon. Fil-C runs on x86_64 (and could run on ARM64 if I had the resources to regularly test on it). So, Fil-C runs on the high volume stuff that gets all of the best optimizations.
- silicon optimization is best done incrementally on top of an already fast chip, where the software being optimized for already runs on that chip. Then it's a matter of collecting traces on that software and tuning, rather than having to think through how to optimize a fast chip from first principles. CHERI means a new register file and new instructions so it doesn't lend itself well to the incremental optimization.
So, I suspect Fil-C will always be faster than CHERI. This is especially true if you consider that there are lots of possible optimizations to Fil-C that I just haven't had a chance to land yet.
That's exactly the kind of thing that the boosters of all those previous efforts said. But somehow it never quite worked out.
Language implementations get faster over time and young ones tend to be slow. The Fil-C implementation is young. So were all of the previous attempts at memory-safe C - usually an implementation that had years of at most a few person years of investment (because it was done in an academic setting). Young implementations tend to be slow because the optimization investment hasn't happened in anger. So, "past academic attempts were slow" is not a great reason to avoid investigating memory safe C.
Performance focus is not the reason why all of the world's C/C++ code gets written. Maybe that's even a minority reason. Lots of stuff uses C/C++ because of reasons like:
- It started out in C/C++ so it continues to be in C/C++. So many huge projects are in this boat.
- You're critically relying on a library whose only bindings are in C/C++, or the C/C++ bindings are the most mature, or the most easy to use.
- You're doing low-level systems stuff, and having pointers that you can pass to syscalls is a core part of your logic.
- You want to play nice with the dynamic linking situation on the OS you're targeting. (C/C++ get dynamic linking right in a way other languages don't.)
I'd guess less than half of the C/C++ code that's being written today is being written because the programmer was thinking "oh man, this'll be too slow in any other language".
Finally, Fil-C is already faster than a lot of memory safe languages. It's just not as fast as Yolo-C, but I don't think you can safely bet that this will be true forever.
No, they really didn't. Let's review some of the big cases.
- SafeC: not based on a mainstream C compiler, so can't handle gcc/clang extensions (Fil-C can). Had no story for threads or shared memory (Fil-C does). Hence, not really compatible.
- CCured: incompatible (cannot compile C code with it without making changes, or running their tool that tries to automate the changes - but even then, common C idioms like unions don't quite work). Didn't use a major C compiler,
- SoftBound: not totally memory safe (no attempt to provide safety for linking or function calls). But at least it's highly compatible.
I can list more examples. Fil-C is the first to get both compatibility and safety right.
Has any impartial third party reached that conclusion? Because honestly the way I remember it everyone says this kind of thing when it's their own project, a lot of the people behind these previous efforts were just as confident as you are.
I don't think this is true. - D, Swift, Rust, Zig: different languages, and while they do have FFI, using it means you're only as safe as your C code - CHERI: requires hardware support to be practical - Checked C, CCured, ?SAFECode IIRC?: too expensive - AddrSan@runtime, ARM MTE, SoftBound: mitigations with too many holes
I don't know of many (to be honest, can't think of any) other serious attempts at making a system that tries to cover all three of
A) lets you write normal C
B) covers all the gaps
C) doesn't kill performance
Maybe 3/3 depending on your workload and definition of "killing performance". It's less than 2x slower for some stuff.
The good news is Fil-C is getting faster all the time, and there are still so many obvious optimizations that I haven't gotten around to.
Here's one: even just switching from gcc or msvc to clang, in projects that really want to, takes years.
Here's another one: the Fil-C compiler is really young, so it almost certainly still has bugs. Those compilers that folks actually use in anger tend to get qualified on ~billions of lines of code before anyone other than the compiler devs touches them. The Fil-C compiler is too young to have that level of qualification.
So "immediately everywhere" isn't going to happen. At best it'll be "over a period of time and incrementally".
I’ll summarize: language implementations get faster over time. Young ones tend to be slow. Fil-C is a young implementation that still has lots of unoptimized things. Also, Fil-C being 2x slower than C means it’s already faster than many safe languages. And, for a lot of C use cases perf doesn’t matter as much as the hype suggests.
The fact that young implementations are slow is something that’s worth understanding even if you don’t care about fil-C. It suggests, for example, that if someone invents a new language and their initial implementation is slow, then you can’t use that fact to assume that it’ll be slow forever. I think that’s generally a useful lesson.
I care about performance a lot and Fil-C has gotten about 100x faster since the first prototype. It’ll keep getting faster.
Getting these things to mass deployment, ticking all those little boxes, is a lot of effort. Porting a distribution to a new CPU architecture is likely easier, especially after the early stages (toolchain bringup).
Apart from that, there could be technical issues with the proposed approach. Perhaps the memory overhead? Or it might turn out that the desired performance characteristics basically require a JIT that specializes code so that typed pointers can be used where the types are known to be correct.
> It's not widely known, and it seems to be still in the research stage.
Yes on both counts.
> A lot of things in this area never really get out of that.
I hope that doesn't happen to Fil-C, but it could!
> Getting these things to mass deployment, ticking all those little boxes, is a lot of effort.
100%
I think this is one area where I'm trying to make Fil-C different than what came before it. I'm trying to tick all those little boxes. It's a lot of work!
> Porting a distribution to a new CPU architecture is likely easier, especially after the early stages (toolchain bringup).
Not sure about this. It might be true today because Fil-C hasn't yet ticked all the boxes, but the aim is definitely to be close to the cost of porting to a new CPU. It's already like that for a lot of code.
> Perhaps the memory overhead?
Heh yeah. The invisicaps cost memory. And GC costs memory.
> Or it might turn out that the desired performance characteristics basically require a JIT that specializes code so that typed pointers can be used where the types are known to be correct.
I've thought about how a JIT might help. I don't think it would. (Most of my compiler experience is writing JITs and I wrote JavaScriptCore's JITs, so I'm biased towards seeing JIT opt opportunities - and I don't see any in Fil-C right now.)
https://docs.oracle.com/en/operating-systems/solaris/oracle-...
> Here's the difference with Fil-C: It's totally memory safe and fanatically compatible with C/C++, no B.S.
Is this true on both counts? If I'm reading your docs right, you're essentially adding hidden capabilities to pointers. This is a great technique that gives you almost perfect machine-level compatibility by default, but it comes with the standard caveats:
1. Your type safety/confusion guards are essentially tied to pointer "color," and colors are finite. In other words, in a sufficiently large program, an attacker can still perform type confusion by finding types with overlapping colors. Not an issue in small codebases, but maybe in browser- or kernel-sized ones.
2. In terms of compatibility, I'm pretty sure this doesn't allow a handful of pretty common pointer-integer roundtrip operations, at least not without having the user/programmer reassign the capability to the pointer that's been created out of "thin air." You could argue correctly that this is a bad thing that programmers shouldn't be doing, but it's well-defined and common enough IME.
(You also cited my blog's writeup of `totally_safe_transmute` as an example of something that Fil-C would prevent, but I'm not sure I agree: the I/O effect in that example means that the program could thwart the runtime checks themselves. Of course, it's fair to say that /proc/self/mem is a stunt anyways.)
Fil-C totally allows pointer to integer round tripping in many cases, if the compiler can see it’s safe.
I’m citing unsafe transmute as something that Fil-C doesn’t prevent. It’s not something that memory safety prevents.
I'm having trouble seeing where the type confusion protection properties come from, then. I read through your earlier (I think?) design that involved isoheaps and it made sense in that context, but the newer stuff (in `gimso_semantics.md` and `invisicap.txt`) seems to mostly be around bounds checking instead. Apologies if I'm missing something obvious.
> I’m citing unsafe transmute as something that Fil-C doesn’t prevent. It’s not something that memory safety prevents.
I think the phrasing is confusing, because this is what the manifesto says:
> No program accepted by the Fil-C compiler can possibly go on to escape out of the Fil-C type system.
This to me suggests that Fil-C's type system detects I/O effects, but I take it that wasn't the intended suggestion.
Long answer: let's assume 64-bit (8 byte pointers) without loss of generality. Each capability knows, for each 8 bytes in its allocation, whether those 8 bytes are a pointer, and if so, what that pointer's capability is.
Example:
char* p = malloc(64);
Then if you do:
*(int**)(p + 8) = malloc(sizeof(int));
Hence, each capability is dynamically tracking where the pointers are. So it's not a static type but rather something that can change over time.
There's a bunch of engineering that goes into this being safe under races (it is) and for supporting pointer atomics (they just work).
https://github.com/pizlonator/llvm-project-deluge/blob/delug...
One more thing: I assume the memory allocator has a somewhat efficient way to recover the pointer to the start of the allocation from a pointer inside it, for interoperability with the legacy C world. For recompiled code, this is not used because the capability is either tracked in registers, or loaded from the shadow memory.
The capability is the start of the allocation most of the time. It might have a bit in it saying that it's an aligned allocation (indicating alignment greater than 16 bytes), in which case there's an alignment hole between the start of what the GC thinks is the allocation and where the capability starts. Also capabilities might be global, meaning that there is no need to mark them. So, to mark a capability (which the runtime calls an "object"), the algorithm is something like:
if (object->is_global)
return;
void* thing_to_mark;
if (UNLIKELY(object->is_aligned))
thing_to_mark = object & -object->alignment;
else
thing_to_mark = object;
mark(thing_to_mark); int deref(uintptr_t p)
{
return *(int*)p;
}
edit: I found zptrtable and this construct:
https://github.com/pizlonator/pizlonated-quickjs/commit/258a...
The latter seems to indicate that:
(int*)((uintptr_t)p + 1)
The pair of conversions is guaranreed to always produce a pointer with a capability. That’s how I implemented it and it’s low-tech enough that it could be specified.
(int*)(f((uintptr_t)p))
Could someone implement Rust’s with_addr as:
(int*)((uintptr_t)p, addr)
Not sure about the semantics of with_addr. But note that you can do this in Fil-C:
char* p = ...;
uintptr_t i = ...;
p -= (uintptr_t)p; // now p is NULL but still has its original capability
p += i; // now p points to whatever i's address was, but has p's original capability
https://doc.rust-lang.org/std/primitive.pointer.html#method....
As I understand it, Rust added this in part for experiments with CHERI but mostly for miri.
Interestingly, the implementation of with_addr is very similar to your code.
How do you handle cases where there are multiple possible sources of the capability? For example:
int *a = …, *b = …;
(int*)(((uintptr_t)a + (uintptr_t)b) << 2)
struct node { struct node *p; };
struct node *next(struct node *here, struct node *prev)
{
return (struct node *)((uintptr_t)here ^ (uintptr_t)prev);
}
Rust’s with_addr wins points for being explicit and unambiguous. It obviously loses points for not being C. And Rust benefits here from all of this being in the standard library and from some of the widely-available tooling (miri) getting mad if code doesn’t use it. I can imagine a future Fil-Rust project doing essentially the same thing as Fil-C except starting with Rust code. It might be interesting to see how the GC part would interact with the rest of the language.
If you want to be explicit about where the capability comes from, use `zmkptr`.
It is true that memory-safe C compilers have existed for decades and have seen minimal adoption.
However, improvements to clang/llvm could yield wider impact and benefit than previous efforts, since they may be supported in a widely used C toolchain.
-fbounds-safety is another change that may see more adoption if it makes it into mainline clang/llvm
For example, asan will totally let you access out of bounds of an object. Say buf[index] is an out-of-bounds access that ends up inside of another object. Asan will allow that. Fil-C won't. That's kind of a key spacial safety protection.
Asan is for finding bugs, at best. Fil-C is for actually making your code memory safe.
I am pretty sure you cannot go much lower than 1.2 here on the best cases. In contrast, CHERI on good hardware will easily be as close to current performance as possible.
Google doesn't come out with anything useful. Could you elaborate?
Except for those GC pauses...
For CHERI to be fully safe, it basically needs a GC. They just call it something else. They need it to clean up capabilities to things that were freed, which is the same thing that Fil-C uses GC for.
And as industry took the lead in language innovation, government removed its Ada requirement.
If memory safety was "just the right regulations" easy, it would have already been solved. Every competent developer loves getting things right.
I can imagine a lot more "compliance" than success may be the result of any "progress" with that approach.
The basic problem is challenging, but makes it hard-hard is the addition of a mountain of incidental complexity. Memory safety as a retrofit on languages, tools and code bases is a much bigger challenge than starting with something simple and memory safe, and then working back up to something with all the bells and whistles that mature tool ecosystems provide for squeaking out that last bit of efficiency. Programs get judged 100% on efficiency (how fast can you get this working? how fast does it run? how much is our energy/hardware/cloud bill?), and only 99% or so on safety.
If the world decided it could get by on a big drop in software/computer performance for a few years while we restarted with safer/simpler tools, change would be quick. But the economics would favor every defector so much that ... that approach is completely unrealistic.
It is going to get solved. The payoff is too high, and the pain is too great, for it not to. But not based on a concept of a plan or regulation.
Memory safety is already a solved problem in regulated industries. It's not a hard problem as such. People just don't want to solve it and don't have any incentive to: companies aren't penalised for writing buggy software, and individual engineers are if anything rewarded for it.
> Every competent developer loves getting things right.
Unfortunately a lot of developers care more about being able to claim mastery of something hard. No-one gets cred for just writing the thing in Java and not worrying about memory issues, even though that's been a better technical choice for decades for the overwhelming majority of cases.
I am pretty sure that the parent is talking about hardware memory safety which doesn't require any "human rewriting the source".
It does though. The hardware might catch an error (or an "error") and halt the program, but you still need a human to fix it.
The same thing can be said about a Rust vector OOB panic or any other bug in any safe language. Bugs happen which is why programmers are employed in the first place!
Sure, the point is you're going to need the programmer either way, so "hardware security lets us detect the problem without rewriting the code" isn't really a compelling advantage for that approach.
Maybe. Depends what the fallback for the business that was using it is when that program doesn't run.
> Humans need to fix bugs, but that is nothing new. A memory bug with such features would be hardly more significant than any other bug
Perhaps. But it seems to me that the changes that you'd need to make to fix such a bug are much the same changes that you'd need to make to port the code to Rust or what have you, since ultimately in either case you have to prove that the memory access is correct. Indeed I'd argue that an approach that lets you find these bugs at compile time rather than run time has a distinct advantage.
No, you wouldn't need to prove that the memory access is correct if you relied on hardware features. Or I should say, that proof will be mostly done by compiler and library writers who implement the low level stuff like array allocations. The net lines of code changed would definitely be less than a complete rewrite, and would not require rediscovery of specifications that normally has to happen in the course of a rewrite.
>Indeed I'd argue that an approach that lets you find these bugs at compile time rather than run time has a distinct advantage.
It is an advantage but it's not free. Every compilation takes longer in a more restrictive language. The benefits would rapidly diminish with the number of instances of the program that run tests, which is incidentally one metric that correlates positively with how significant bugs actually are. You could think of it as free unit tests, almost. The extra hardware does have a cost but that cost is WAAAY lower than the cost of a wholesale rewrite.
I don't see how the hardware features make this part any easier than a Rust-style borrow checker or avoid requiring the same rediscovery of specifications. Checking at runtime has some advantages (it means that if there are codepaths that are never actually run, you can skip getting those correct - although it's sometimes hard to tell the difference between a codepath that's never run and a codepath that's rarely run), but for every memory access that does happen, your compiler/runtime/hardware is answering the same question either way - "why is this memory access legitimate?" - and that's going to require the same amount of logic (and potentially involve arbitrarily complex aspects of the rest of the code) to answer in either setting.
Of course, but compare it with rewriting it to a completely different language.
It's not hard, no, but it is expensive, because those regulations have a battery of tests run by a thirdy party that you will pay money to each time you want to recertify.
I've worked in two regulated industries; the recertification is the expensive part, not the memory errors.
Most famously in Rust. Even there it takes work.
The problem is a practical coding work efficiency (and quality) one. You are right that there are no intractable memory problems even in the unsafest least helpful languages.
It seems to me that Rust programmers, unhappy with the pace of adoption of Rust, seek to make other languages illegal because they do things different from Rust.
You are not rewarded for:
1) Formal proofs or careful programming. No one cares if a piece of software works quietly for years.
2) Preventing others from ruining a working piece of software. To the contrary, you will be called a gatekeeper and worse things.
You are rewarded for:
1) Wild ideas, quickly and badly implemented with the proper amount of marketing.
2) Churn, "social" coding, and LGTM.
3) Ironically, if you are a security researcher, finding exploits can help, too. As above, preventing exploits in the first place is regarded as a waste of time.
All of the above is true at Google. But of course they have a technical solution to a social problem. Which might catch one category of bugs at best.
"in C# or Java" ?
where do you even base these claims? Do you know what C# and Java threads have that Rust doesn't? data races. And don't get me stated on the biggest paradigm failure that is OOP.
I beg to differ unless all your logic is in the database with strong isolation guarantees.
Speaking of C# for backends that are using EF actively, I bet there are bugs in pretty much all of them caused by incorrect applications of optimistic concurrency.
Where have you been the past 5 years? Rust developers are insanely productive.
Can we put this myth to rest already? Rust being an "unproductive language" is thoroughly dis-proven.
I can give you an example:
var number = 0;
var delay = Task.Delay(1000);
for (var i = 0; i < 10; i++)
{
Task.Run(() =>
{
while (!delay.IsCompleted)
{
Interlocked.Increment(ref number);
}
});
}
await delay;
To avoid misunderstanding, I think Rust is a great language and even if you are a C# developer who does not plan to actively use it, learning Rust is of great benefit still because it forces you to tackle the concepts that implicitly underpin C#/F# in an explicit way.
Channels and selects. It's trivial.
fn main() { let num = Arc::new(AtomicUsize::new(0)); let finished = Arc::new(AtomicBool::new(false));
for _ in 0..10 {
std::thread::spawn({
let num = num.clone();
let finished = finished.clone();
move || {
while !finished.load(Ordering::SeqCst) {
num.fetch_add(1, Ordering::SeqCst);
}
}
});
}
std::thread::sleep(Duration::from_millis(1000));
finished.store(true, Ordering::SeqCst);
}If you're not comfortable in a language, then sure you ponder and pontificate and wonder about what the right approach is, but if you're experienced and familiar then you just do it plain and simple.
What you're describing is not at all a language issue, it's an issue of familiarity and competency.
Obviously if you don't like the manual-ness of Rust, just use something else. For what you described I'd reach for Elixir or Golang.
I was disagreeing with you that it's not easy or too difficult. Rust just takes a bit of effort and ramping up to get good at. I hated it as well to begin with.
But again -- niches. Rust serves its niche extremely well. For other niches there are other languages and runtimes.
One-liners work best in comedy, dude.
- Whereas in .net you have the best native interop imaginable for a high level language with a vast SDK. I understood that Java has improved on JNI, but I am not sure how well that compares.
- Programming languages are like a religion, highly inflammable, so I can imagine you would not be swayed by some rando on the internet. I would already be happy if you choose Go over Python, as with the former you win some type safety (but still have a weak type system) and have a good package manager and deployment story.
- Go was designed for Google, to prevent their college grads from implementing bad abstractions. But good abstractions are valuable. A weak type system isn't a great idea (opinion, but reasonable opinion). Back then .net was not really open source (I believe) and not as slim and fast as it is now, and even then, I think Google wants to have control about their own language for their own internal needs.
- Therefore, if you are not Google, Go should likely not be your top pick. Limited area of application, regrettable decisions, tailored for Google.
(Sorry for a reply all over the place.)
---
> if you do not need the absolute bare metal control C or Rust provides, you are better of with either .net or Java.
That, like your next point, is a relatively fair statement but it's prone to filter bubble bias as I am sure you are aware. I for example have extracted much more value out of Golang... and I had 9 years with Java, back in the EJB years.
Java and .NET are both VM-based and have noticeable startup time. As such, they are best suited for servers and not for tooling. Golang and Rust (and Zig, and D, V and many other compiled languages) are much better in those areas.
> Programming languages are like a religion, highly inflammable, so I can imagine you would not be swayed by some rando on the internet
Those years are long past me. I form my own opinions and I have enough experience to be able to make fairly accurate assessments with minimum information.
> I would already be happy if you choose Go over Python, as with the former you win some type safety (but still have a weak type system) and have a good package manager and deployment story.
I do exactly that. In fact I am a prominent Python hater. Its fans have gone to admirable heights in their striving to fill the gaps but I wonder will they one day realize this is unnecessary and just go where those gaps don't exist. Maybe never?
And yeah I use Golang for my own purposes. Even thinking of authoring my own bespoke sync and backup solution stepping on Syncthing, GIT and SSH+rsync and package it in Golang. Shell scripts become unpredictable from one scale and on.
> Go was designed for Google, to prevent their college grads from implementing bad abstractions. But good abstractions are valuable. A weak type system isn't a great idea (opinion, but reasonable opinion). Back then .net was not really open source (I believe) and not as slim and fast as it is now, and even then, I think Google wants to have control about their own language for their own internal needs.
That and your next point I fully agree with. That being said, Golang is good enough and I believe many of us here preach "don't let perfect be the enemy of good". And Golang symbolizes exactly that IMO; it's good enough but when your requirements start shooting up then it becomes mediocre. I have stumbled upon Golang's limitations, fairly quickly sadly, that's why I am confining it to certain kinds of projects only (and personal tinkering).
> I recommend reading the comment history of @neonsunset.
I don't mind doing that (per se) but I find appeals to authority and credentialism a bit irritating, I admit.
Plus, his reply was in my eyes a fairly low-effort snark.
For JIT-based deployments, it is measured in 100-500ms depending on the size of application, sometimes below. .NET has first-party support for NativeAOT deployment mode for a variety workloads: web servers, CLI tools, GUI applications and more.
Go is a VM-based language, where VM provides facilities such as virtual threading with goroutines (which is higher level of abstraction than .NET's execution model), GC, reflection, special handling for FFI. Identical to what .NET does. I don't think the cost and performance profile of BEAM needs additional commentary :)
Go also has weaker GC and compiler implementations and, on optimized code, cannot reach the performance grade of C++ and Rust, something C# can do.
> Those years are long past me. I form my own opinions and I have enough experience to be able to make fairly accurate assessments with minimum information.
The comments under your profile seem to suggest the opposite. Perhaps "minimum information" is impeding fair judgement?
> I don't mind doing that (per se) but I find appeals to authority and credentialism a bit irritating, I admit.
Is there a comment you have in mind which you think is engaging in credentialism?
The other guy who told me to inspect your profile. Not you.
> The comments under your profile seem to suggest the opposite.
Sigh. You seem to act in bad faith which immediately loses my interest.
You'll have to take your high and mighty attitude to somebody else. Seems like even HN is not immune from... certain tropes, shall we call them, generously.
Disengaging.
> that's why I am confining it to certain kinds of projects only (and personal tinkering).
You have a fair view of Go, I think. I could see that it makes sense to use it as a replacement for bash scripts, especially if you know the language well. Personally I am wanting to dive into using F# for my shell scripting needs. The language leans well into those kind of applications with the pipe operator.
If you ever have the appetite, you should take a look at it as it can be run in interpreted/REPL mode too, which is a nice bonus for quick one-of scripts.
First, the main controller may panic and die, leaving all those tasks still running; while they run, they still access the two local variables, `number` and `delay`, which are now out of scope. My best understanding is that this doesn't result in undefined behavior in C#, but it's going to be some sort of crash with unpredictable results.
I think the expectation is that tasks use all cores, so the tasks also have to be essentially Send + 'static, which kinda complicates everything in Rust. Some sort of scoped spawning would help, but that doesn't seem to be part of core Tokio.
In C#, the number variable is a simple integer, and while updating it is done safely, there's nothing that forces the programmer to use Interlocked.Read or anything like that. So the value is going to be potentially stale. In Rust, it has to be declared atomic at the start.
Despite the `await delay`, there's nothing that awaits the tasks to finish; that counter is going to continue incrementing for a while even after `await delay`, and if its value is fetched multiple times in the main task, it's going to give different results.
In C#, the increment is done in Acquire-Release mode. Given nothing waits for tasks to complete, perhaps I'd be happy with Relaxed increments and reads.
So in conclusion: I agree, but I think you're arguing against Async Rust, rather than Rust. If so, that's fair. It's pretty much universally agreed that Async Rust is difficult and not very ergonomic right now.
On the other hand, I'm happy Rust forced me to go through the issues, and now I understand the potential pitfalls and performance implications a C#-like solution would have.
Does this lead to the decision fatigue you mention in another sub-thread? It seems like it would, so I'll give you that.
let number = Arc::new(AtomicUsize::new(0));
let finished = Arc::new(AtomicBool::new(false));
let finished_clone = Arc::clone(&finished);
let delay = task::spawn(async move {
sleep(Duration::from_secs(1)).await;
finished_clone.store(true, Ordering::Release);
});
for _ in 0..10 {
let number_clone = Arc::clone(&number);
let finished_clone = Arc::clone(&finished);
task::spawn(async move {
while !finished_clone.load(Ordering::Acquire) {
number_clone.fetch_add(1, Ordering::SeqCst);
task::yield_now().await;
}
});
}
delay.await.unwrap();
use rayon::prelude::*;
use std::time::{Instant, Duration};
use std::sync::atomic::{AtomicUsize, Ordering};
fn main() {
let finish_at = Instant::now() + Duration::from_secs(1);
let number = AtomicUsize::new(0);
(0..10).into_par_iter().for_each(|_| {
while finish_at > Instant::now() {
number.fetch_add(1, Ordering::Relaxed);
}
});You can make it even simpler if you would use Mutex instead of atomics. Atomics are more performant though.
If your baseline is a language that is missing some features that were in standard ML, sure. If you were already using OCaml or F#, Rust doesn't make you any more productive. If you were already using Haskell or Scala, Rust's lack of HKT will actively slow you down.
We need the BEAM VM's guarantees, not yesterday but like 20 years ago, everywhere. The language itself does not matter. But we need that runtime's guarantees!
Just to start with.
We already have some false positives. Happily the team is very motivated and is grinding away at them, for which we the community are forever grateful.
Modern LISP dialect authors still believe threads are a super clever idea which is just... /facepalm.
No massive churn, quite performant and the code i wrote 20 years ago runs without modification.
Can other blub languages claim this?
Even a site called HN if you happen to know it.
Let us say I have a large C++ codebase which I am unwilling to rewrite in Rust. But I:
* Enable STL bounds checking using appropriate flags (like `-DGLIBCXX_ASSERTIONS`).
* Enable mitigations like CFI and shadow stacks.
How much less safe is "C++ w/ mitigations" than Rust? How much of the "70% CVE" statistic is relevant to my codebase?
https://securitycryptographywhatever.com/2024/10/15/a-little...
* CFI
* isoheaps or type-stable allocators
* Shadow stacks
(There is just a single hit of "C++"...)
Ignoring the appeal to authority, I have a hard time believing that incrementally rewriting my C++ code in Rust or just writing new code in Rust ("vulnerabilities exponentially decay" and all that) is going to give me more actual security than the mitigations stated above. Most, if not all, high-profile exploits stem from out-of-bounds accesses and type confusions, which these mitigations prevent at very low cost.
Thanks for replying, though.
> They prevent but do not entirely mitigate.
Ignoring the semantic difference between "prevent" and "mitigate", if at the end of the day, the security provided by the two different approaches are quite similar, I don't get the problem.
If you have an example of a successful widespread exploit that would have happened even with these mitigations, please share.
Note that I am not very knowledgeable in security, and I am really willing to be educated but it feels like most of the replies to my comments are just trying to prove me wrong.
I can't really speak to your comments as you haven't posted many of them. I'm not here to prove you wrong but just to share my views on these mitigations. I'm not an expert by any means but I do get to think about these for work so I can usually do at least a quick once-over to try to figure out how effective they might be.
> Overwriting function pointers gives code execution under shadow stacks
As you are probably aware, there are two kinds of CFI - forward-edge and backward-edge. Forward-edge CFI prevents tampered function pointers, vtables and such from being invoked. Whereas backward-edge CFI protects does the same for return addresses. Clang's and MSVC's (CFG) implementations of CFI only provide forward-edge protection, hence the need for shadow stacks. Without hardware support, shadow stacks can not be prevented from tampered, which is why Intel (CET) and AMD added shadow stacks.
> Mobile platforms are very memory sensitive
Agreed. I'd guess this applies to embedded too. But all things considered, I do hold the opinion that language-level memory safety is being overplayed a lot.
I think we are eventually going to end up with some sort of tagged memory with what this is for (such as credentials) and rules about who is allowed to touch it and where it's allowed to go. Instead of writing yet another tool that won't let fields called "password" or "token" or "key" be printed into the logs, but misses "pk", it's going to be no printing any memory block in this arena, period.
I also think we aren't doing enough basic things with backend systems like keeping just a user ID in the main database schema and putting all of the PII in a completely different cluster of machines, that has 1/10 to 1/100th of the sudoer entries on it of any other service in the company. I know these systems are out there, my complaint is we should be talking about them all the time, to push the Recency Effect and/or Primacy Effect hard.
If you're still using one database in 2025, even if you wouldn't touch microservices with a ten foot pole, then you've got some problems.
OLAP, KV, cache heirarchies, you aren't running a singular database, except maybe for a standalone app.
If enabled (through a command line switch), all data coming in from the outside (sockets, STDIN etc.) are "tainted", and if you e.g. concatenate a non-tainted and a tainted string, it becomes tainted. Certain sensitive operations, like system() calls or open(), raise an error when used with tainted data.
If you match tainted data with a regex, the match groups are automatically untainted.
It's not perfect, but it demonstrates that such data tagging is possible, and quite feasible if integrated early enough in the language.
But they are Boolean values and they need to be an enumeration or likely a bitfield. Even just for web I’ve already identified four in this thread. HTML unsafe, url unsafe, PII unsafe, credentials unsafe. I hesitate to add SQL unsafe because the only solution to sql injection is NO STRING CONCATENATION. But so many SQL libraries use concatenation even for prepared statements that maybe it should be. Only allow string constants for sql queries.
I am 100% in favor of industry standards to enforce safety. It should go way past just memory safety, though. Engineering standards should include practices and minimum requirements to prevent safety issues as a whole.
And add slice and buffer typedefs to the standard library. Especially since they added counted_by to the language.
Add a way to define a slice the points to a c string.
Do you have a source for this? I thought it was use after free.
Out of bounds write and read are more prevalent then UAF. There are multiple types of bugs that can produce OOB read or write though.
It's so easy that thousands of developers trying for 40+ years haven't been able to do it yet.
Unfortunately, easily corrected it is not. Yes, probably >95% of arrays have a nearby easily-accessible length parameter that indicates their maximum legal length (excluding the security disaster of null-terminate strings). But the problem is there's no consistent way that people do this. Sometimes people put pointer first and size second, sometimes it's the other way around. Sometimes the size is a size_t, sometimes an unsigned, sometimes an int. Or sometimes it's not a pointer-and-size, but pointer and one-past-the-end pointer pair. Sometimes multiple arrays share the same size parameter.
So instead of an easy solution getting you 90% for effectively free, you get like 30% with the easy solution, and have to make it more complicated to handle the existing diversity to push it back up to that 90%.
https://news.ycombinator.com/item?id=42962020 - It is time to standardize principles and practices for software memory safety (2025-02-06, 100 comments)
They don't really lead to exploitable conditions, except maybe DoS if you have poor error handling and your application dies with null pointer exceptions.
The site clearly mentions folks to use something else, if they want to write safe code today.
> we suggest establishing a common framework
:(
- Buffer overflow: somebody didn't sanitize the input (length of buffer).
- Stack smashing: somebody didn't sanitize the input (length of input).
- Format string vulnerability: somebody didn't sanitize the format string data.
- Integer converstion vulnerability: somebody didn't sanitize the integer input.
- SQL injection: somebody didn't sanitize the input before querying a database.
- Cross-site scripting: somebody didn't sanitize input and it got submitted and executed on behalf of the user.
- Remote file inclusion / Directory traversal: somebody didn't sanitize an input variable leading to a file path.
...and on, and on, and on. If people were as obsessed with input sanitization as they are with memory, I'll bet you a much larger percentage of attacks would be stopped. Too bad input sanitization isn't sexy.
Prepared statements and safe HTML construction APIs plus some linters that scream at you when you use the unsafe APIs works like magic.
You should simply discern between HTML elements (code) and HTML text nodes (data). Same with prepared statements: Clear distinction between SQL code vs SQL data.
You just need to ensure that your data is never interpreted as code.
That's sanitization. Many different languages implement this. The old-school method is "tainting" data so it can't be used as part of execution without an explicit function call to "untaint" it. Same is used for "secret" data in various programs where you don't want it leaked.
- modern languages prevent you from having to think about it at all. You shouldn’t have to sanitize, it should work
- there are a load of issues that have nothing to do with sanitization but everything to do with memory. Race conditions, type confusion, UAF to name a couple. If we focus on sanitization, we still need memory safety to fix those
Also sanitization is non-trivial for complex inputs
> - Cross-site scripting: somebody didn't sanitize input and it got submitted and executed on behalf of the user.
To be technical about it, this is generally a failure of escaping rather than sanitizing.
You're supposed to be able to put anything into a database field and not have it affect the query. You ought to be able to paste JavaScript into a field and have it be displayed as JavaScript, not executed. The inputs remain as they are -- no sanitization -- they just have to be escaped properly.
That being said, I'm 100% on board about the importance of sanitation/validation. To the extent I think it ought to be part of the design of languages just like types. I.e. if a function parameter is only allowed to be three string values, or a string between 0 and 10 bytes, or a string limited to lowercase ASCII, these should be expressable.
Sanitizing inputs won't protect you against all bugs. For example, you may store a string in a 256-byte buffer, so you check that the string is no longer than 256 characters, but you forget the zero-terminator, and you have a buffer overflow. Or maybe you properly limited the string to 255 characters, but along the way, you added support for multibyte characters, and you get another buffer overflow.
Bound checking would have caught that.
Injection can happen at any point. You may sanitize user input to avoid SQL injection, but at some later point it may get out of the database and in a format string, but it was sanitized for SQL, not for format strings, leading to a potential injection.
Actually it is a simple and effective way to prevent general bugs.
If you have an input field called birthday, you can inject it directly into your database. Doing that could cause an SQL injection exploit, so people use prepared statements.
But even if you use prepared statements, you'll still end up with a database column with all kinds of birthday formatting (M-D-Y, Y-M-D, slashes, spaces, colons, etc etc). These different, non-standard formats will eventually cause a bug.
Input sanitization forces you to standardize on one birthday format, and inject it into your database in one format (let's say "YYYY-MM-DD", no other characters allowed). Then all the code expects - and gets - one format. This reduces bugs from unexpected formats.
It has the added side-effect of also eliminating the SQL injection bug, regardless of prepared statement.
That's not sanitisation as the word is normally used. That's parsing and canonicalisation. That's a good path to actual security - it leads you to the "make invalid states unrepresentable" style and using decent type systems.
IIRC there's some way that a Python C extension can accidentally disable the NX bit for the whole process.. https://news.ycombinator.com/item?id=40474510#40486181 :
>>> IIRC, with CPython the NX bit doesn't work when any imported C extension has nested functions / trampolines
>> How should CPython support the mseal() syscall? [which was merged in Linux kernel 6.10]
> We are collaborating with industry and academic partners to develop potential standards, and our joint authorship of the recent CACM call-to-action marks an important first step in this process. In addition, as outlined in our Secure by Design whitepaper and in our memory safety strategy, we are deeply committed to building security into the foundation of our products and services.
> That's why we're also investing in techniques to improve the safety of our existing C++ codebase by design, such as deploying hardened libc++.
Secureblue; https://github.com/secureblue/Trivalent has hardened_malloc.
Memory safety notes and Wikipedia concept URIs: https://news.ycombinator.com/item?id=33563857
...
A graded memory safety standard is one aspect of security.
> Tailor memory safety requirements based on need: The framework should establish different levels of safety assurance, akin to SLSA levels, recognizing that different applications have different security needs and cost constraints. Similarly, we likely need distinct guidance for developing new systems and improving existing codebases. For instance, we probably do not need every single piece of code to be formally proven. This allows for tailored security, ensuring appropriate levels of memory safety for various contexts.
> Enable objective assessment: The framework should define clear criteria and potentially metrics for assessing memory safety and compliance with a given level of assurance. The goal would be to objectively compare the memory safety assurance of different software components or systems, much like we assess energy efficiency today. This will move us beyond subjective claims and towards objective and comparable security properties across products.
It is just that those who build consumer-grade hardware don't want to spend 12% more on RAM for slightly less performance. Among them are essentially all ARM devices, including smartphones and Apple silicon Macs.
What's preventing Microsoft, or Apple, or the coagulate Linux kernel team, or any other kernel team, from adopting memsafe technology or practice by themselves for themselves?
The last thing we need are what are evidently incompetent organizations that can't take care of their own products making standards, or useless academics making standards to try to force other people to follow rules because they know better than everyone else.
If the team that designed and implemented KeyKos, or that designed Erlang, were pushing for standardized definitions or levels of memory safety, it would be less ridiculous.
At the same time, consciousness of security issues and memory safety has been growing quickly, and memory safety in programming languages has literally exploded in importance. It's treated in every new PL I've seen.
Putting pressure on big companies to fix their awful products is fine. No pressure needs to be applied to the rest of the industry, because it's already outpacing all of the producing entities that are calling for standards.
It seems like decades-old giant code bases are precisely the ones hardest to migrate to memory safety. That's where coercion and enforcement is needed most. You and I don't need to be told to start a new project in not-C++ do we? Nearly every trained programmer has been brainwashed (in a good way) with formal methods, type systems, bounds checking, and security concerns. Now those same people who champion this stuff say it isn't enough, and therefore we need to do more of the same but with coercion. That's a failure to understand the problem.
Google saw "the percentage of memory safety vulnerabilities in Android dropped from 76% to 24% over 6 years as development shifted to memory safe languages" - which I'd say is a positive example.
It's not that they've already fully succeeded (I don't think anyone has on codebases of this size), but neither is it that they tried and failed - it's an ongoing effort.
> You and I don't need to be told to start a new project in not-C++ do we?
Don't need to be told because we all already avoid C++, or don't need to be told because it doesn't really matter if we do use C++?
I'd disagree with both. There are still many new projects (or new components of larger systems) being written in C++, and it's new code that tends to have the most vulnerabilities.