A Note on Fil-C
graydon2.dreamwidth.org
graydon2.dreamwidth.org
The patch itself was maintained for many years, well into the mid 2000s, out of tree (actually by another person in the end), but as it never went upstream it was hard to keep doing that maintenance.
There were several problems in hindsight: C programmers at the time absolutely weren't willing to accept a large slow-down in order to get bounds checking. But also we didn't optimize our changes well (or very much at all) and I'm sure we could have got the delta down a bit if we'd put the work in. The main work that dominated performance was the lookup that you have to do from the raw pointer to the fat struct that stores the pointer bounds (and you have to do this on every pointer operation). We used a splay tree for this which was clever but not very fast. A plain hash or some other data structure could have been much faster.
The provenance model for C is very recent (and still a TS, not part of the standard). Prior to that, there was a vague notion that the C abstract machine has quasi-segmented memory (you aren't really allowed to do arithmetic on a pointer to an "object" to reach a different "object") but this was not clearly stated in usable terms.
Here’s a rough timeline:
- 2004-2018: I had ideas of how to do it but I thought the whole premise (memory safe C) was idiotic.
- 2018-2023: I no longer thought the premise was idiotic but I couldn’t find a way to do it that would result in fanatical compatibility.
- 2023-2024: early Fil-C versions that were much less compatible and much less performant
- end of 2024: InvisiCaps breakthrough that gives current fanatical compatibility and “ok” performance.
It’s a hard problem. Lots of folks have tried to find a way to do it. I’ve tried many approaches before finding the current one.
I'm interested in implementing a safe low-level language with less static information around than C has (e.g. no static pointer-int distinction), but I'd rather keep around the ability to restrict capabilities to only refer to subobjects than have the same compatibility guarantees Invisicaps provide, so I was hoping to look into Monocaps (or maybe another design, if there's one that might fit better).
If a hypothetical time machine allowed you to send the InvisiCaps idea back to your 2004-era self, do you think the approach would have been feasible back then as well?
The observation that the C variants used on GPUs are simplistic takes on memory safe C
Long long ago, in 2009, Graydon was my official on-boarding mentor when I joined the Mozilla Javascript team. Rust already existed then but, as he notes, was quite different then. For one thing, it was GC'd, like Fil-C. Which I like -- I write a lot of my C/C++ code using Boehm GC, have my own libraries designed knowing GC is there, etc.
This has obviously been 'rust'ling some feathers, as it challenges some of the arguments laid past; but once the dust settles, it is a major net benefit to the community.
I hope you get financed and can support other platforms than linux again.
I'm a Rust user and a fan. But memory safe C is actually an exciting prospect. I was hoping that the rise of Rust would encourage others to prioritize memory safety and come up with approaches that are much more ergonomic to the developers.
> as it challenges some of the arguments laid past
Genuinely curious. What are the assumptions you have in mind that Fil-C challenges? (This isn't a rhetorical question. I'm just trying to understand memory safety concepts better.)
> but once the dust settles, it is a major net benefit to the community.
Agreed, this is big! If Fil-C can fulfill its promise to make old C code memory safe, it will be a massive benefit to the world. God knows how many high-consequnce bugs and vulnerabilities hide in those.
However, Rust has been quite successful making more developers think about less known type systems, besides affine types, there is also linear types, effects, dependent types, prof systems.
And we as industry aren't going to throw away the millions and millions of stuff that was written in C, C++ and less extent Objective-C, thus efforts like Fil-C are quite welcomed.
That's the end-goal right? I don't write Rust code myself, but I'm glad its existence means there's safer code out there now, and like you I have been looking forward to seeing shifts in safety expectations. I'm not surprised that it's happening so slowly though.
This is a very odd statement. Mature C programs written by professional coders (Redis is a good example) basically never crash in the experience of users. Crashing, in such programs, is a rare occurrence mostly obtained by attackers on purpose, looking for code paths that generate a memory error that - if the program is used as it should - are never reached.
This does not mean that C code never segfaults: it happens, especially when developed without care and the right amount of testing. But the code that is the most security sensitive, like C Unix servers, is high quality and crashes are mostly a security problem and a lot less a stability problem.
I think if you understand the meaning of "crash" to include any kind of unhandled state that causes the program to terminate execution then it includes things like unwrapping a None value in Rust or any kind of uncaught exception in Python.
That interpretation makes sense to me in terms of the point he's making: Fil-C replaces memory unsafety with program termination, which is strictly worse than e.g. (safe) Rust which replaces memory unsafety with a compile error. But it's also true that most programs (irrespective of language, and including Rust) have some codepaths in which programs can terminate where the assumed variants aren't upheld, so in practice that's often an acceptable behaviour, as long as the defect rate is low enough.
Of course there is also a class of programs for which that behaviour is not acceptable, and in those cases Fil-C (along with most other languages, including Rust absent significant additional tooling) isn't appropriate.
Rust uses panics for out-of-bounds access protection.
The benefit of dynamic safety checking is that it's more precise. There's a large class of valid programs that are not unsafe that will run fine in Fil-C but won't compile in Rust.
The paragraph refers to detecting such bugs during compilation versus crashing at runtime. The "almost all programs have paths that crash" means all programs have a few bugs that can cause crashes, and that's true. Professional coders do not attempt to write 100% bug-free code, as that wouldn't be efficient use of the time. Now the question is, should professional coders convert the (existing) C code to eg. Rust (where likely the compiler detects the bug), or should he use Fil-C, and so safe the time to convert the code?
Don't worry, it's totally sound.
Unfortunately, security hysteria also treats any crash as "an expensive and urgent CVE ticket". See, for instance, ReDoS, where auditors will force you to update a dependency even if there's no way for a user to provide the vulnerable input (for instance, it's fixed in the configuration file).
I think "perhaps the density of crashes will be tolerable" means something like "we can reasonably hope that the crashes from Fil-C's memory checks will only be of the same sort, that aren't reached when the program is used as it should be".
It's provably safer than rust, e.g.
Most software written does not serve a serious nation level user base but caters to so a relatively small set of users. The effort spent eradicating errors needs to be justified by the effort of workarounds, remediation work and customer impact. Will not be fixed can a rationale decision.
My programs can’t do anything about that situation, so let it crash.
Same logic for:
* The server in the config file doesn’t exist.
* The given output file has bad permissions.
* The hard drive is full.
Etc. And again, that’s completely deliberate. There’s nothing I can do in code to fix those issues, so it’s better to fail with enough info that the user can diagnose and fix the problem.
That was in Python. I do the same in Rust, again, deliberately. While of course we all handle the weird cases we’re prepared to handle, I definitely write most database calls like “foo = db.exec(query)?” because if PostgreSQL can’t execute the query, the safest option is to panic instead of trying foolhardily to get back the last known safe state.
And of course that’s different for different use cases. If you’re writing a GUI app, it makes much more sense to pop up a dialog and make the user go fix the issue before retrying.
"Cannot open data file: Not found" and that's it - no more context (such as a filename). Even for a user with no coding experience, this is absolutely useless, you cannot find good explanation for it on the Google. A backtrace might look ugly, but at least would have a much higher chance to point to a useful forum post. And now with AI advances, AI can analyze backtraces and sometimes give an explanation (not very often, but there is no alternatives...)
So by all means, add a nice, human-readable error message for a few common cases that user likely to encounter, such as "internet down" or "wrong type of input file"... but leave backtraces on for all other unexpected cases, like "server returned nonsense", "out of disk space", etc....
I did specify "succinctly as possible with all the information necessary for a human to be able to solve the issue". An error that doesn't have "all the information necessary" is a bad error. It can be worse than a backtrace. That doesn't mean a backtrace is good.
That is a very difficult assertion to validate. It might well be true! But so many conversations about memory safety and C/C++ devolve to assertions with “get gud” at one extreme and “change platforms to one that avoids certain errors” at the other.
Without data, even iffy data, those groups talk past each other. Are memory-error CVE counts on C projects the data we need here? Is there some other quantitative measure of real world failures that occur due to memory unsafety?
This is all by way of saying that I’d love to see some numbers there. That’s not on you, or meant to question your claim. As you implied, errors in code don’t always translate to errors in behavior for users.
It just always sucks to talk about this because broad-spectrum quantitative data on software error rates and their causes is lacking.
Keep in mind he's limited his assertion to UX. That narrow point is almost certainly true in the case of his C codebase.
But read the rest-- he literally wrote how security researchers find memory safety errors in C codebases!
Dollars to donuts he came up with this UX-on-accident vs. security-researcher-on-purpose bug dichotomy in his head as a response to some actual CVE in his own C codebase.
In short, he's agreeing with the research that led to programming languages like Rust in the first place. And even though he's found an odd way to agree, there's no assertion to validate here (at least wrt security).
Edit: clarifications
This is very much not the case for programs that are much newer, even if they are written in Rust they still need years of maturation before they reach the quality of older C programs, as Rust programs suffer from non-memory safety issues just as much. That's why just rewriting things in Rust isn't a panacea.
The perfect example of this the Rust coreutils drama that has been going on.
Wrong, dereferencing a NULL pointer is UB.
Suppose we assume that many C applications aren’t performance sensitive and can easily take a 2-4x performance hit without noticing. Browsers and OS internals being obvious exceptions. The ideal candidates are like the ones djb writes, and he’s already a convert to Fil-C. sudo, sshd, curl - all seem like promising candidates.
But as far as I can tell, Fil-C doesn’t work for C libraries that can be called from elsewhere. Even if it could be made to work, the reason other languages like Python or Node use C libraries is for speed. If they were ok with it being 2-4x slower, they would just write ordinary Python or Javascript.
C (and C++) are fundamentally important because of their use in performance sensitive contexts like operating systems, browsers and libraries. If we’re restricting Fil-C to pure C/C++ applications that aren’t performance sensitive, that might still be very important and useful, but it’s a small slice of the large C/C++ pie.
Also, it’s a great tool for an existing C application, certainly. A performance hit in exchange for security is a reasonable trade off while making a battle hardened application work. But for a new application, would someone choose Fil-C over other performant GC languages like Go or Java or C#? I’d be keen to hear why.
Still, I want to stress - this is a great project and it’ll generate a lot of value.
Python and JavaScript are much more than 4x slower than C/C++ for workloads that are git-like (significant amount of compute, not just I/O bound)
> C (and C++) are fundamentally important because of their use in performance sensitive contexts like operating systems, browsers and libraries
That's a fun thing to say but it isn't really true. C/C++ are fundamentally important for lots of reasons. In many cases, folks choose C and C++ because that's the only way to get access to the APIs you need to get the job done.
You’re the guy actually building this, so you would have talked to potential customers more than me. You’re more likely to be correct, but I would be interested to know what these applications are.
Also, do you hope people pick Fil-C for a new code base over other performant GC languages like Go, Java and C#? These have less than a 2x overhead for CPU bound tasks.
An example: GitHub’s entire business revolves around calling libgit2 (C) from Ruby. Are they more likely to slow down libgit2 and make it substantially more complex by running 2 GCs side by side, or are they going to risk accept any potential unsafety in regular C? It’s 100% the latter, I’ll bet on that.
For a strictly time-limited interaction (like what's involved in a FFI call) it's not that bad. Everything that GC2 might directly access is temporarily promoted to a root for GC1, and vice versa.
The cost of all the additional hardware is just not worth it. If it was a choice between higher hardware costs, higher request latency, greater operational complexity of a new technology and rewriting libgit2 in a different language without all those tradeoffs, GitHub definitely chooses the latter.
But it’s never going to reach that point because they’ll continue using libgit2 compiled by clang forever.
But all existing programming languages seem to have some disadvange: C is fast but unsafe. Fil-C is C compatible but requires GC, more memory, and is slower. Rust is fast, uses little memory, but us verbose and hard to use (borrow checker). Python, Java, C# etc are easy to use, concise, but, like Fil-C, require tracing GC and so more memory, and are slow.
I think the 'perfect' language would be as concise as Python, statically typed, not require tracing GC like Swift (use reference counting), support some kind of borrow checker like Rust (for the most performance critical sections). And leverage the C ecosystem, by transpiling to C. And so would run on almost all existing hardware, and could even be used in the kernel.
These might all be slower than well written C or rust, but they're not nearly the same magnitude of slow. Java is often within a magnitude of C/C++ in practice, and threading is less of a pain. Python can easily be 100x slower, and until very recently, threading wasn't even an option for more CPU due to the GIL so you needed extra complexity to deal with that
There's also Golang, which is in the same ballpark as java and c
Yes, Python is specially slow, but I think it's probably more because it's dynamically typed, and not not compiled. I found PyPy is quite fast.
pypy is fast compared to plain python, but it's not remotely in the same ballpark as C, Java, Golang
Surprisingly, Java is right behind manual memory managed languages in terms of energy use, due to its GC being so efficient. It turns out that if your GC can "sprint very fast", you can postpone running it till the last second, and memory drains the same amount no matter what kind of garbage it holds. Also, just "booking" that this region is now garbage without doing any work is also cheaper than calling potentially a chain of destructors or incrementing/decrementing counters.
fwiw benchmarksgame uses benchexec
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Presumably the benchmark game doesn't allow "I wrote this code in C" as a Python submission, but it would allow unsafe C# tricks ?
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Note: Here are naive un-optimised single-thread programs transliterated line-by-line literal style into different programming languages from the same original.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
I love how people assume that the GC is the reason for Fil-C being slower than C and that somehow, if it didn't have a GC, it wouldn't be slower.
Fil-C is slower than C because of InvisiCaps. https://fil-c.org/invisicaps
The GC is is crazy fast and fully concurrent/parallel. https://fil-c.org/fugc
Removing the GC is likely to make Fil-C slower, not faster.
I understand raw speed (of the main thread) of Fil-C can be faster with tracing GC than Fil-C without. But I think there's a limit on how fast and memory efficient Fil-C can get, given it necessarily has to do a lot of things at runtime, versus compile time. Energy usage, and memory usage or a programming language that uses a tracing GC is higher than one without. At least, if memory management logic can be done at compile time.
For Fil-C, a lot of the memory management logic, and checks, necessarily needs to happen at runtime. Unless if the code is annotated somehow, but then it wouldn't be pure C any longer.
That then may allow for some of the uses to be statically optimised away, i.e. by annotating pointers upon which arithmetic is not allowed.
The Fil-C capability mechanisms for trapping double-free, and use-after free would probably have to be retained, but maybe it could optimise some uses?
Yes, they might lose the meaningless benchmarks game that gets thrown around, what matters is are they fast enough for the problem that is being solved.
If everyone actually cared about performance above anything else, we wouldn't have an Electron crap crisis.
https://microsoft.github.io/react-native-windows/docs/new-ar...
Have fun following the discussions and amount of bugs,
https://github.com/microsoft/microsoft-ui-xaml
That C++ support that WinUI team marketing keeps talking about relies on a framework that is no longer being developed.
> The reason the issues page only lets you create a bug report is because cppwinrt is in maintenance mode and no longer receiving new feature work. cppwinrt serves an important and specific role, but further feature development risks destabilizing the project. Additional helpers are regularly contributed to complimentary projects such as https://github.com/microsoft/wil/.
From https://github.com/microsoft/cppwinrt/issues/1289#issuecomme...
People don't like leaving performance on the table. It feels stupid and it lets competitors have an easy advantage.
The Electron situation is not because people don't care about performance; it's because they care more about some other things (e.g. not having to do 4x the work to get native apps).
And yes, caring more about other things is why performance isn't the top number one item, and most applications have long stopped being written in pure C or C++ since the early 2000's.
We go even further in several abstraction layers, nowadays with the ongoing uptake of LLMs and agentic workflows in iPaaS low code tools.
Personally at work I haven't written a pure 100% C or C++ application since 1999, always a mix of Tcl, Perl, Python, C# alongside C or C++, private projects is another matter.
What Java had going for it was the massive scale of Sun's marketing, and the JDK being available as free beer, however until Eclipse came to be, all IDEs were commercial, and everyone was coding in Emacs, vi (no vim yet), nano, and so on.
However it only became viable after Java 1.3, when Hotspot became part of Java's runtime.
I agree with the spirit of your comment though, and I also think that the blow given by Java to C and C++ wasn't bigger, only because AOT tools were only available under high commercial prices.
Many folks use C and C++, not due to their systems programming features, rather they are the only AOT compiled languages that they know.
I heavily doubt that this would work on arbitrary C compilers reliably as the interpretation of the standard gets really wonky and certain constructs that should work might not even compile. Typically such things target GCC because it has such a large backend of supported architectures. But LLVM supports a large overlapping number too - thats why it’s supported to build the Linux kernel under clang and why Rust can support so many microcontrollers. For Rust, that’s why there’s the rust codegen gcc effort which uses GCC as the backend instead of LLVM to flush out the supported architectures further. But generally transpiration is used as a stopgap for anything in this space, not an ultimate target for lots of reasons, not least of which that there’s optimizations that aren’t legal in C that are in another language that transpilation would inhibit.
> Rust is fast, uses little memory, but us verbose and hard to use (borrow checker).
It’s weird to me that my experience is that it was as hard to pick up the borrow checker as the first time I came upon list comprehension. In essence it’s something new I’d never seen before but once I got it it went into the background noise and is trivial to do most of the time, especially since the compiler infers most lifetimes anyway. Resistance to learning is different than being difficult to learn.
Rust borrow checker: the problem I see is not so much that it's hard to learn, but requires constant effort. In Rust, you are basically forced to use it, even if the code is not performance critical. Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python. The main disadvantage of Rust, in my view, is that it's verbose. (Also, there is a tendency to add too many features, similar to C++, but that's a secondary concern).
I think there's space for Rust to become more ergonomic, but its goals limit just how far it can go. At the same time I think there's space to take Rust and make a Rust# that goes further on the Swift/Scala end of the spectrum, where things like auto-cloning of references are implemented first, that can consume Rust libraries. From the organizational point of you, you can see it as a mix between nightly and editions. From a user's point of view you can look at it as a mode to make refactoring faster, onbiarding easier and a test bed for language evolution. Not being Rust itself it would also allow for different stability guarantees (you can have breaking changes every year), which also means you can be holder on tryin things out knowing you're not permanently stuck with them. People who care about performance, correctness and reuse can still use Rust. People who would be well served by Swift/Scala, have access to Rust's libraries and toolchain.
> (Also, there is a tendency to add too many features, similar to C++, but that's a secondary concern).
These two quoted sentiments seem contradictory: making Rust less verbose to interact with reference counted values would indeed be adding a feature.
If that's what you're looking for, you can use Swift. The latest release has memory safety by default, just like Rust.
Unless you are writing formal proofs nothing is completely safe, GC languages had found a sweet spot until increased concurrency started uncovering thread safety problems. Rust seems to have found a sweet spot that is usable despite the grumbling. It could probably be made a bit easier. The compiler already knows when something needs to be send or synch, and it could just do that invisibly, but that would lead people to code in a way that had lots of locking which is slow and generates deadlocks too often. This way the wordiness of shared mutable state steers you towards avoiding it except when a functional design pattern wouldn't be performant. If you have to use mutex a lot in rust stop fighting the borrow checker and listen to what it is saying.
They have not in the past 10 years.
[1] https://github.com/thomasmueller/bau-lang/blob/main/doc/perf...
Isn't it just enforcing something you should be doing in every language anyway, i.e. thinking about ownership of data.
It's kinda annoying when you run into those. I think I've also ran into a situation where the borrow checker itself wasn't the issue, but rather the way references were created in a pattern match causing the borrow checker to reject the program. That was also annoying.
Source? I’m not familiar with official efforts here. I see one in the community for Lua but nothing for Go. It’s rare for languages to use this as anything other than a stopgap or a neat community poc. But my point was precisely this - if you’re only targeting GCC/LLVM, you can just use their backend directly rather than transpiling to C which only buys you some development velocity at the beginning (as in easier to generate that from your frontend vs the intermediate representation) at the cost of a worse binary output (since you have to encode the language semantics on top of the C virtual machine which isn’t necessarily free). Specifically this is why transpile to C makes no sense for Rust - it’s already got all the infrastructure to call the compiler internals directly without having to go through the C frontend.
> Rust borrow checker: the problem I see is not so much that it's hard to learn, but requires constant effort. In Rust, you are basically forced to use it, even if the code is not performance critical
Your only forced to use it when you’re storing references within a struct. In like 99% of all other cases the compiler will correctly infer the lifetimes for you. Not sure when the last time was you tried to write rust code.
> Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python.
Any language targeting the performance envelope rust does needs GC to be opt in. And I’m not sure how much extra verbosity there is to wrap the type with RC/Arc unless you’re referring to the need to throw in a RefCell/Mutex to support in place mutation as well, but that goes back to there not being an alternative easy way to simultaneously have safety and no runtime overhead.
> The main disadvantage of Rust, in my view, is that it's verbose.
Sure, but compared to what? It’s actually a lot more concise than C/C++ if you consider how much boilerplate dancing there is with header files and compilation units. And if you start factoring in that few people actually seem to actually know what the rule of 0 is and how to write exception safe code, there’s drastically less verbosity and the verbosity is impossible to use incorrectly. Compared to Python sure, but then go use something like otterlang [1] which gives you close to Rust performance with a syntax closer to Python. But again, it’s a different point on the Pareto frontier - there’s no one language that could rule them all because they’re orthogonal design criteria that conflict with each other. And no one has figured out how to have a cohesive GC that transparently and progressively lets you go between no GC, ref GC and tracing GC despite foundational research a few years back showing that ref GC and tracing GC are part of the same spectrum and high performing implementations in both the to converge on the same set of techniques.
> transpiling to C (even Go and Lua)
Go: I'm sorry, I thought TinyGo internally converts to C, but it turns out that's not true (any more?). That leaves https://github.com/opd-ai/go2c which uses TinyGo and then converts the LLVM IR to C. So, I'm mistaken, sorry.
Lua: One is https://github.com/davidm/lua2c but I thought eLua also converts to C.
> Your only forced to use it when you’re storing references within a struct.
Well, that's quite often, in my view.
> Not sure when the last time was you tried to write rust code.
I'm not a regular user, that's true [2]. But I do have some knowledge in quite many languages now [3] and so I think I have a reasonable understanding of the advantages and disadvantages of Rust as well.
> Any language targeting the performance envelope rust does needs GC to be opt in.
Yes, I fully agree. I just think that Rust has the wrong default: it uses single ownership / borrowing by _default_, and RC/Arc is more like an exception. I think most programs could use RC/Arc by default, and only use ownership / borrowing where performance is critical.
> The main disadvantage of Rust, in my view, is that it's verbose. >> Sure, but compared to what?
Compared to most languages, actually [4]. Rust is similar to Java and Zig in this regard. Sure, we can argue the use case of Rust is different than eg. Python.
[1] https://github.com/thomasmueller/bau-lang [2] https://github.com/thomasmueller/lz4_simple [3] https://github.com/thomasmueller/bau-lang/tree/main/src/test... [4] https://github.com/thomasmueller/bau-lang/blob/main/doc/conc...
That is skewing your perception. The problem is that how you write code just changes after a while and both things happen: you know how to write things to leverage the compiler inferred lifetimes better and the lifetimes fade into the noise. It only seems really annoying, difficult and verbose at first which is what can skew your perception if you don’t actually commit to writing a lot of code and reading others’ code so that you become familiar with it better.
> Compared to most languages, actually [4]. Rust is similar to Java and Zig in this regard. Sure, we can argue the use case of Rust is different than eg. Python.
That these are the languages you’re comparing of is a point in Rust’s favor - it’s targeting a significantly lower level and higher performance of language. So Java is not comparable at all. Zig however nice is fundamentally not a safe language (more like C with fewer razor blades) and is inappropriate from that perspective. Like I said - it fits a completely different Pareto frontier - it’s strictly better than C/C++ on every front (even with the borrow checker it’s faster and less painful development) and people are considering it in the same breath as Go (also unsafe and not as fast), Java (safe but not as fast) and Python (very concise but super slow and code is often low quality historically).
Top optimization opportunities:
- InvisiCaps 2.0. While implementing the current capability model, when I was about 3/4 of the way done with the rewrite, I realized that if I had done it differently I would have avoided two branch+compares on every pointer load. That's huge! I just haven't had the appetite for doing yet another rewrite recently. But I'll do it eventually.
- ABI. Right now, Fil-C uses a binary interface that relies on lowering to what ELF is capable of. This introduces a bunch of overhead on every global variable access and every function call. All of this goes away if Fil-C gets its own object file format. That's a lot of work, but it will happen in Fil-C gets more adoption.
- Better abstract interpreter. Fil-C already has an abstract interpreter in the compiler, but it's not nearly as smart as it could be. For example, it doesn't have octagon domain yet. Giving it octagon domain will dramatically improve the performance of loops.
- More intrinsics. Right now, a lot of libc functions that are totally memory safe but are implemented in assembly are implemented in plain Fil-C instead right now, just because of how the libc ports happened to work out. Like, say you call some <math.h> function that takes doubles and returns doubles - it's going to be slower in Fil-C today because you'll end up in the generic C code version compiled with Fil-C. No good reason for this! It's just grunt work to fix!
- The calling convention itself is trash right now - it involves passing things through a thread-local buffer. It's less trashy than the calling convention I started out with (that allocated everything in the heap lmao), but still. There's nothing fundamentally preventing a Fil-C register-based calling convention, but it would take a decent amount of work to implement.
There are probably other perf optimization opportunities that I'm either forgetting right now or that haven't been found yet. It's still early days!
I've always been firmly in the 'let it crash' camp for bugs, the sooner and the closer to the offending piece of code you can generate a crash the better. Maybe it would be possible to embed Fil-C in a test-suite combined with a fuzzing like tool that varies input to try really hard to get a program to trigger an abend. As long as it is possible to fuzz your way to a crash in Fil-C that would be a sign that there is more work to do.
That way 'passes Fil-C' would be a bit like running code under valgrind and move the penalty to the development phase rather than the runtime. Is this feasible or am I woolgathering, and is Fil-C only ever going to work by using it to compile the production code?
All the quick fixes for C that don't require code rewrites boil down to crashing. They don't make your C code less reliable, they just make the unreliability more visible.
To me, Fil-C is most suited to be used during development and testing. In production you can use other sandboxing/hardening solutions that have lower overhead, after hopefully shaking out most of the bugs with Fil-C.
The whole point of Fil-C is having C compatibility. If you're going to treat it as a deployment target on its own, it's a waste: you get overhead of a GC language, but with clunkiness and tedium of C, instead of nicer language features that ground-up GC languages have.
For example, Fil-C lifts all escaping locals to the heap, but doesn't free them.
> Maybe it would be possible to embed Fil-C in a test-suite
A lot of remarkably unusual stuff has been shoved into the format without breaking the tooling, so wondering what the restrictions are.
graydon points in that direction, but since you're here: how feasible is a hypothetical Fil-Unsafe-Rust? would you need to compile the whole program in Fil-Rust to get the benefits of Fil-Unsafe-Rust?
So that implies just running all of Rust through the Fil-C transformation
It can be done, especially with a safe non-GC language that can meaningfully guarantee it won't corrupt GC metadata or break its invariants. You only have real issues (and then only wrt. excess overhead, not unsoundness) with pervasive mutual references between the GC and non-GC parts of the program. You do need to promote GC pointers to a root anytime that non-GC code has direct access to them, and add finalizers to GC objects that may need to drop/run destructors on non-GC data.
I guess the primary reason would be running hardened code in production without compromising performance too much, same as you would run Fil-C compiled software instead of the usual way. I've no idea if it's feasible to run miri in prod.
From my understanding Fil-C is an LLVM operation, so it should be possible to build integration to have a Fil-Rust binary that is slower but gives you some of the benefits of miri. I see value in doing something like that. There are plenty of other languages that would be well served by this too!
- Give up on lock freedom of atomic pointers. This is a fun one because theoretically, it’s worse. But it comes with a net perf improvement because there’s no need to check the low bit of lowers.
As a Linux user of two decades, memory safety has never been a major issues that I would be willing to trade performance for. It doesn't magically make my application work it just panics instead of crashes, same end result for me. It just makes it so the issue can not be exploited by an attacker. Which is good but like Linux has been already safe enough to be the main choice to run on servers so meh. The whole memory safety cult is weird.
I guess Fil-C could have a place in the testing pipeline. Run some integration tests on builds made with it and see if stuff panics.
That said, Fil-C is a super cool projects. I don't mean to throw any shades at it.
Then why are all of the IO-bound low level pieces of Linux userland written in C?
Take just one example: udevd. I have a Fil-C version. There is zero observable difference in performance.
My fear is that the performance difference might add up once use it on more and more part. I imagine it uses a lot more memory. Plus once Fil-C gets adopted in the mainstream it might lower the need for devs to actually fix the code and they might start just relying on Fil-C.
To be fair, systemd itself is corporate shite to begin with and I wouldn't mind seeing it being replaced with something written in a language with memory safety.
If that argument is valid, then why hasn't it stopped adoption of slow languages?
Like, Python is waaay slower than Fil-C. And so much of Linux userland is written in shell, which is slower still.
Well, the program would still halt upon memory flaw, so there would still be a need to fix it
> That’s like saying “everyone else runs Pentium 2, why would I upgrade to Pentium 3?”
No one should blindly upgrade because bigger number is better. If I look into new hardware I research benchmarks and figure out if it would enable me to (better) run the software/games I care about it and if the improvement is worth my money.
Same with security. You need to read actual studies and figure out what the cost/benefit of certain measures is.
There are safer alternatives to Linux but apparently the situation isn't bad enough for people to switch to them.
And I am not saying you should create new projects in C or C++. Most people should not. But there is a lot of battle tested C and C++ code out there and to act as if we suddenly have this big problem with memory safety is a weird narrative to push. And if you discover a vulnerability, well fix it instead of wrapping it Fil-C and making the whole thing slower.
Here’s what Fil-C gives you that -fbounds-safety doesn’t:
- Fil-C gives you comprehensive memory safety while -fbounds-safety just covers bounds. For example, Fil-C panics on use after free and has well defined semantics on ptr-int type confusion.
- -fbounds-safety requires you to modify your code. Fil-C makes unmodified C/C++ code memory safe.
FWIW, I worked on -fbounds-safety and I still think it’s a good idea. :-)
It also doesn't have to be a complete all-at-once rewrite. Plain C can easily co-exist with other languages, and you can gradually replace it by only writing new code in another language.
For example you also get a far stronger type system (leading to fewer logic bugs) and modern tooling.
First of all, Rust's 'fearless concurrency' largely boils down to 'no concurrency' - Rust has about as much concurrency as Javascript - you can copy objects between threads, but not share memory beyond that, with certain libraries allowing some escape hatches in Rust's case.
Additionally the case for aliasing control leading to better and safer code that's easier to reason about just isn't really true in practice - it's so rare that, for example your function can accidentally can alias memory in strictly typed languages - and when it does it's kind of intentional. The concern pretty much only manifests in memcpy, as the ugly hack of 'strict aliasing' - assuming 2 pointers of incompatible types pointing to different bits of memory - works very well in practice.
It even helps with situations people complain about in Rust, like when object A and B (both mutably borrowed) take a reference to a some common service object (which is super common) - but that sort of code simply doesn't compile in Rust.
All in all, I don't dislike Rust as it is but the project's tendency to do activism trying to bully its technical skeptics into submission (which is unfortunately what all activism really is - when you lost the argument, try to louder than the other guy and paint him as reprehensible) - they focused on fixing the technical issues. There has been research into ownership schemes, some exist that are less restrictive than Rust's while offering the same safety guarantees.
In my personal opinion Rust is not done on the conceptual level - by 'done' I mean Rust serving its purpose as the language it claims to be. Maybe there will be Rust 2.0 which will overhaul the ownership system completely or maybe there'll be another language that will do what Rust does but better.
Edit: I wish I could claim I'm some sort of tin-foil conspiracy theorist, but I'm commenting under an article written by one of the key people behind Rust, and it reeks of this attitude.
...this is just blatantly false? Like it is false to the extent that I am confused as to what you could even possibly be talking about - I don't know how anyone who has actually written non-trivial programs in both languages could come to the conclusion that they have the same memory model (or much of anything in common when it comes to threads, really).
In practice Rust doesn't make concurrency easier - it forces extremely conservative architectures or basically constantly copying everything between threads (hence my example of Js workers).
Sure it's safe, but it's the nanny kind of safety, where you are basically disallowed to do something remotely challenging or interesting, instead of the bungee cord kind of safety where the framework allows you to go crazy and if you obey a few simple rules you can make sure your code will still work.
People who do not work on nuclear reactors don't seek out this kind of safety by themselves - it is forced on them.
The other funny thing about the Rust memory model is that it kind of assumes an environment where there's a malloc/free function pair which is stateless and global, and every allocation needs to be freed.
This is absolutely not the case in low-level code, with memory pools, arenas etc. In these cases Rust either enforces unnecessary restrictions or can't really keep track of the lifetime of objects.
Rust is seriously undercooked.
This is of course not disingenuous in the slightest (especially the part where you bring up "using Arc", i.e. the equivalent of e.g. C++'s shared_ptr, as an example of the sort of "extremely conservative architecture" you are forced to use when writing multithreaded Rust). Anyone who deems it so is obviously just a Rust fanatic, probably. I assume mutexes are also a Rust invention to make concurrency difficult.
I've never quite understood the complaints about "the nanny kind of safety", either. To put it very bluntly, they just come across to me as wanting to be nannied. I am supposed to entertain the concept of a developer so competent with ideas so clever that the rules of safe Rust simply cannot express them, and yet so nervous and unsure of their own brilliance that they cannot just...write out their case in unsafe Rust and move on. It's as if people want to be smarter than the compiler and yet also never actually have to demonstrate that they are, and it just doesn't make sense.
In general Rust claims to provide you with a set of safety rules that are a more comprehensive extension of C++ unique_ptr and co. so you can do write the same code, have like 10% more mental overhead because of additional safety, but in exchange you can fearlessly do anything, and know your program doesn't have memory leaks or race conditions.
However that's not how it works in practice. Rust rejects all non-trivial naive multi-threaded programs. There's a zero percent chance that you can map a non-trivial C++ program to Rust without major rewrites, even though said program is 100% safe from aliasing bugs.
How multi-threading actually works in Rust:
You copy everything between threads, or try to adapt every structure that needs to be threaded with Sync, Send, Arc etc. Not very efficient use of ones time.
Or you rely on one of the threading libraries/frameworks which makes certain styles of parallelism easier to express. These libraries usually contain giant blobs of unsafe Rust, which isn't exactly a win for the language.
The TLDR version of my post is Rust still rejects a significant percentage of non-trivial but correct programs. If we were anal about it and cared about the soundness of the language, and disallowed libraries with unsafe, that number would jump even higher.
> There's something off-putting about how Rust people communicate. Like you're talking to people selling something, or people with ulterior motives.
Which is a radically wrong description of the tone of the OP that is purportedly the reason for the comment. And for the most impressive bit of un-self-reflective projection I've seen in a long time, there's
> And the way you write is extremely rude and condescending, if you do decide to phrase your comments in such a demeaning way (which I don't ever recommend), please at least be right next time.
There are 3 kinds of ways people percieve Rust - those who see it as a cool piece of tech (which I agree with), those who see it as a practical tool, and those who see liking or not liking Rust as an ideological statement. I see it as a tool.
There's a lot of statements around Rust that are true in the strict factual sense, but clearly meant to be read differently.
'Free beer at the pub tonight' is a true statement even though the full truth would be 'Free beer at the bar tonight (entry fee $200)'.
'Fearless concurrency' is among these statements meant to be interpreted as making concurrent programming easier. The true statement would be 'fearless concurrency (but you might need to rewrite your program)'.
bit shitty of you given youre having him over his tone, no?
You have no understanding of the context of this old conversation.
Goodbye.
This is all provided we stay in safe Rust land, and don't do concurrency or call external APIs, but that's not a claim I'm holding against the language.
This isn't the point I'm making - the issue I have is that the Rust borrow checker is not powerful enough to accept most correct programs (of real-world complexity) for various reasons. This does not mean that you can't write complex programs in Rust, it only means that most complex programs which do not have the problems Rust claims to address (successfully), have basically zero chance compiling, unleast they're written the way Rust expects them to.
Rc<> and Arc<> can create true memory leaks where a cycle of Rc<> or Arc<> referencing one another can stay allocated in memory even when all outside references to the objects have disappeared, so the leak cannot even be collected. This is the one thing that is not allowed to occur with a tracing garbage collector as in Fil-C. (Rust also allows leaking memory explicitly, e.g. for the purpose of passing a reference to that memory to foreign code where Rust cannot observe its lifecycle. Then in order to properly drop that memory in Rust, you have to essentially recreate that allocation from scratch.)
Race conditions are a logical error that has nothing to do with the borrow checker. The borrow checker can only prevent properly defined data races, which are entirely unrelated to logical race conditions.
And I meant data races, I have stated in my previous post, that race conditions due to issues existing outside of the language, like external libraries or network requests are not the fault of Rust. Neither are logic mistakes a programmer makes.
The crux of what you're saying seems to be
> You copy everything between threads, or try to adapt every structure that needs to be threaded with Sync, Send, Arc etc. Not very efficient use of ones time.
You have not actually articulated any real problem with "Sync, Send, Arc, etc" beyond what just sounds like "but I don't wanna". Even leaving aside Sync and Send, which are constructs fairly unique to Rust, the fact that your other example of a construct you have to "try to adapt" your multithreaded code to use is Arc - quite literally the equivalent of std::shared_ptr - on its own casts heavy doubt on the seriousness of your complaint. Like, "oh no! I have to use an atomic reference counted smart pointer in my multithreaded program" is quite simply unserious as a complaint about a programming language.
How exactly you think pointers should be shared between threads, then? Just passing raw pointers around, and taking responsibility any use-after-free bugs that arise because you know what you're doing and definitely aren't going to write one? Well guess what, you can do that in Rust, too (and I have done that in a few non-trivial projects), but you have to mark that obviously unsafe operation with `unsafe`. For some reason, this six-letter keyword is apparently enough to get people who allegedly know what they're doing to break out in hives, and I just do not understand it.
And coming back to Send and Sync, if you understand what those traits are (versus just seeing them as something to slap on code to make the compiler stop screaming), then I'm sure you know that they are automatically derived traits. If you are having to explicitly implement them, it is because something in your structure is dubiously thread-safe - for instance, a raw pointer or a type that has been marked as thread-local. If you are that confident that you are handling whatever it is in a thread-safe manner, then what exactly is the big adaptation in the one-line implementation (each) of Send and Sync?
If anything, this whole "I have to write new, explicit syntax expressing my intentions and this is The Worst and the same thing as not being to do anything at all" business just reminds me of dynamic typing diehards' complaints about static typing; just replace "using Sync, Send, Arc, etc" with "writing out type annotations". They also complain about compilers rejecting "non-trivial but correct programs" for the minor sin of not being properly annotated.
> These libraries usually contain giant blobs of unsafe Rust, which isn't exactly a win for the language.
You simply cannot put up this cowboy coder "I don't want to be nannied" argument and then start calling for a nanny because the keyword for "I have information that the compiler doesn't, so I'm taking the reins here" is `unsafe`. It is profoundly unserious.
Same thing with this supposed overwhelming complexity of writing regular/safe multithreaded Rust, it really just sounds like wanting the language to pretend that multithreaded execution isn't complex and just handle everything about it for you as you write code as if it is single-threaded and have it just work, without having to think about the semantics. If you don't want to be nannied in that manner, then I don't understand how one can be so dissatisfied with that the language actually does (i.e. provides "certified safe - terms and conditions apply" constructs to express semantics the user is supposed to understand) that they resort to just copying everything instead and genuinely hold the opinion that the language is no different from JavaScript in that regard (a language that notoriously doesn't even have actual threads). And the comparison is plain dishonest no matter how you spin it.
Anyone who has written nontrival applications in Rust has encountered this.
This is an issue as threading a program will require a huge rewrite. And I concede the point, multithreading is difficult - but Rust doesn't make it easier, it merely disallows most correct programs and forces you to write code in a very particular way.
Thus fearless concurrency is a disingenious statement I don't fear concurrency, I fear the borrow checker.
As for the cowboy/nanny argument I think this is not a good way to represent safety. Freedom comes from safety - for example, you can make pretty much anything nowadays in HTML/JS without thinking about limitations, and the end user can still be confident that your website won't compromise his system.
Nannying is when you are constantly being told what you cannot do. It might not even ensure safety, but definitely restricts your freedom.
And the way you write is extremely rude and condescending, if you do decide to phrase your comments in such a demeaning way (which I don't ever recommend), please at least be right next time.
What actually happens in practice, as you've alluded to, is that people who simply want to continue writing software the way they always have (but in Rust, whether due to FOMO or necessity) hear "use Rc/RefCell/Cell if the compiler is yelling at you" and then start slapping those constructs everywhere to avoid having to read too many of those pesky compiler errors, without ever considering or understanding what they actually are. And then when an architectural change is needed - in this case, rewriting a single-threaded program to be multithreaded - suddenly it is also the compiler's fault because the original program was not written with a single thought towards multithreading.
Whether you are using a compiler that yells at you about it or not, properly threading any single-threaded program that was similarly written without concern for parallelism (say, a C or C++ program that also liberally used non-thread-safe reference counted pointers) would take a non-trivial rewrite. For some reason, the fact that Rust - a language that explicitly sells itself on catching such problems at compile time - actively surfaces the poor architecture as errors as opposed to leaving the user to hunt them down themselves is supposed to be considered Bad™.
The funny thing is that there actually are real examples of problems with the borrow checker which need to be fixed by the language team (for example async closures and async Fn traits are very much a work in progress). But "I used explicitly thread-unsafe constructs and now I have to refactor my code to run properly on multiple threads" is simply not one of those problems no matter how you spin it.
> Nannying is when you are constantly being told what you cannot do. It might not even ensure safety, but definitely restricts your freedom.
Nannying is also literally the job of looking after a child [that isn't yours], which is a definition that in my opinion applies with users who don't want to learn how to use a tool as prescribed (this part makes perfect sense to me and is completely fair, nobody HAS to use Rust or any other language or tool regardless of what its diehard fans think), yet also want it to protect them from their own mistakes and lack of foresight when they make a mess of using it (this part makes no sense to me).
To put it very bluntly (or rudely, or however else you want to read it): *all* languages (natural and programming alike) have constraints, that is what makes them languages. It's perfectly fine for the conversation to begin and end at "I don't like Rust's borrow checking rules". Sure, then don't use Rust! There are several languages that I simply don't use because I don't like something or other about them, even things as trivial as significant whitespace in Python. I don't want to deal with it so I simply...don't. But approaching a language in oppositional defiance and then complaining that you got the fight you were looking for makes little sense to me.
cheers, anyway